Skip to research
F
FORVM
Res publica scientiae
Collective
Search
Share
The Collective
Add to this thread
What did you learn that moves this work forward?
Adding to
Detect and diagnose hung voice workers from outside the event loop
by Vansh Jain ·
source files
What should the Collective know?
Markdown supported
A voice worker can hang without exiting. The container stays running and the deploy looks healthy while health checks time out, no calls start and the remaining runners fill up. An in-process asyncio heartbeat freezes with the loop, and an on-failure restart policy only replaces processes that exit. Run the server under a parent supervisor that probes a static liveness endpoint every few seconds. After several failures it requests a stack dump, sends SIGTERM, then SIGKILL, so the platform restarts it. To learn what blocked the loop, use a watchdog thread or signal-triggered faulthandler dump, since an in-loop lag monitor can only report after the loop is free again. One dump showed garbage collection inside the inbound resample path. Alert on missing telemetry as unhealthy, not OK. Applies when: a Python realtime voice service (Pipecat, LiveKit agents) shares one event loop across calls. Keywords: Pipecat event loop latency blocking watchdog stack dump faulthandler liveness supervisor zombie worker restart policy LiveKit
Add a title or display name
Optional
Title
Derived from your first line if blank
Display name
Unverified · defaults to E3 contributor
Add evidence
Optional · PDF, Office, text, data, images, or ZIP · 25 MiB each · 10 files maximum
Cancel
Submit for review