Why many triton_python_b and tritonserver processes are left after I killed the tritonserver process

[zochaoq]

I am following the readme https://github.com/triton-inference-server/tensorrtllm_backend and use pkill tritonserver to stop it. Sometimes I find I also need to do the same to pkill -9 -f triton_python_backend_stub but in general they do not block me from running the server again.

But I noticed that after above there are many triton_python_b and tritonserver processes left behind like the below. Curious what are these processes and what are the proper ways to terminate those too?

ps aux | grep triton
root        2185  0.0  0.0      0     0 ?        Z    Jul18   1:01 [tritonserver] <defunct>
root        2186  0.1  0.0      0     0 ?        Z    Jul18   1:28 [tritonserver] <defunct>
root        2187  0.1  0.0      0     0 ?        Z    Jul18   1:28 [tritonserver] <defunct>
root        2188  0.1  0.0      0     0 ?        Z    Jul18   1:29 [tritonserver] <defunct>
root        3766  0.0  0.0      0     0 ?        Z    Jul18   0:21 [triton_python_b] <defunct>
root        3825  0.0  0.0      0     0 ?        Z    Jul18   0:24 [triton_python_b] <defunct>
root        4151  0.0  0.0      0     0 ?        Z    Jul18   0:24 [triton_python_b] <defunct>
root        4162  0.0  0.0      0     0 ?        Z    Jul18   0:24 [triton_python_b] <defunct>
...

[demirciberk]

Short answer
Those (zombie) entries are not "processes still running" — they already exited. A zombie is only a leftover entry in the process table because its parent has not called wait()/waitpid() on it yet. They consume no CPU/memory/GPU (note 0.0 0.0 in your ps aux), but a large pile of them signals that reaping isn't happening properly.

Why this happens with pkill tritonserver / pkill -9 -f triton_python_backend_stub

1. Python backend stub processes are children of tritonserver. For every Python-backend model instance, Triton forks a triton_python_backend_stub process that talks to the parent over shared memory (see the Python backend README). On a normal shutdown, tritonserver sends the stubs a shutdown signal and wait()s for them before it exits.
2. pkill tritonserver sends SIGTERM (or you may be using -9) directly and asynchronously. If you kill the stub processes yourself with pkill -9 -f triton_python_backend_stub before/while the parent tritonserver is also dying, the parent may exit (or get killed) without ever reaping those children → they become zombies reparented to PID 1.
3. In a container, PID 1 is frequently the tritonserver binary itself (or a plain shell), not a real init/reaper. A correct init process (e.g. tini, or Docker's --init flag) automatically calls waitpid() on all orphaned/reparented processes, cleaning up zombies. Without that, orphaned zombies never get collected and just accumulate — this matches your output, where even [tritonserver] entries pile up (meaning the outer process that should have reaped them, e.g. docker exec'd shell or a wrapper script, isn't doing so).
4. Known related reports of stub processes going zombie/unhealthy under load or on abnormal exit: Python backend stub is becoming zombie process #8448 "Python backend stub is becoming zombie process", Python backend status zombie but Tritonserver v2/health still return 200 OK #7230 "Python backend status zombie but tritonserver v2/health still return 200 OK", and [python backend] Triton hangs when a python backend model process is killed by kernel OOM killer #5381 "Triton hangs when a python backend model process is killed by kernel OOM killer" — in all of these the root theme is the same: if the stub dies (killed, OOM, crash) without the parent properly reaping/handling it, you get zombies or a stuck parent.

Proper way to stop Triton (and avoid the buildup)

• Don't pkill -9 anything. Send a single graceful signal to the main tritonserver process only, and let it shut down its own children:

  kill -SIGINT $(pgrep -x tritonserver)
  or SIGTERM
  kill -SIGTERM $(pgrep -x tritonserver)

  If running under Docker, use docker stop (sends SIGTERM, waits, then SIGKILL after the grace period) instead of pkill inside the container.

• Never kill triton_python_backend_stub manually while the parent is still up — that races with Triton's own stub lifecycle management and is what turns stubs into unreaped zombies. Only fall back to pkill -9 -f triton_python_backend_stub as a last resort if a stub is truly hung and tritonserver itself has already been confirmed dead.

• Run the container/process tree with a real init so orphans get reaped automatically:

  • Docker: docker run --init ... (uses tini as PID 1).
  • Or explicitly install/run tini/dumb-init as PID 1 in your entrypoint.

• Existing zombies already on the box cannot be "killed" (they're already dead) — the only way to clear them is for their parent to reap them, which in practice means whatever process is PID 1 in that namespace needs to exit/restart, or you exec into the container and restart it with --init.

What would help confirm root cause further
Please share:

• How Triton is launched (bare metal vs Docker, and whether --init/tini is used).
• What is PID 1 in that environment (ps -p 1 -o comm=).
• Whether the [tritonserver] zombies' parent PID (ps -o ppid= -p ) is 1 or something else.

That will confirm whether this is the "no init/reaper" scenario above versus a Triton-side bug like #8448/#7230/#5381.