At 10,000 feet

Setting: go2rtc (a popular streaming relay) wraps each source in a “producer” that consumers attach to and a reconnect path redials on drop. Some sources are external processes read over a pipe (exec/pipe producers).

Problem: the reconnect path holds the producer’s mutex across the redial. If the redial blocks (a respawned process that hangs before its first byte, with no timeout), the stall watchdog, every consumer’s attach, and stop() all block behind it. The producer is stranded permanently.

What this note establishes (measured): one hung redial under the held mutex froze the whole producer: 0 frames for ~194 s, no recovery pre-fix. Terminal states lasted many hours in the field. A patched build (redial off the mutex, bounded pipe probe, process-group kills) self-recovered in ~6 s.

Takeaway: never hold a producer-wide lock across an unbounded dial. Dial outside the lock, time out the first-byte probe (pipe sources included), kill the process group after a grace period, and force-reset the reconnect loop after N consecutive instant failures.

Notes. Traced in a patched go2rtc fork feeding a video pipeline, against a source fed by an external exec/pipe process that could hang on respawn. The mutex cause is measured (goroutine dump + regression test). The application-side trigger sequence is a plausible mechanism, labeled below. No warranty, corrections welcome.

The finding: the mutex is held across a redial that can never return

reconnect() acquires the producer mutex p.mu, then (re)creates the producer: for an exec/pipe source, spawning the child and blocking on its output. A child that hangs before its first byte holds the mutex forever, freezing the stall watchdog (why logs go silent mid-wedge), every consumer Dial, and stop(). A goroutine dump showed the reconnect goroutine parked in the blocking open with others queued behind it.

Application-side trigger (plausible mechanism, not fully proven): the last consumer drops during a stall, resetting the producer. A fresh consumer then dials while reconnect is mid-redial against a dead source. Reconstructed from code and the dump, not captured live.

The measured wedge, by the numbers

On a repro harness, frames emitted after the source stops responding:

build first-byte probe bound frames recovery
pre-fix 30 s default (but redial under mutex) 0 for ~194 s never (terminal)
partial fix (redial off mutex) 30 s default 0 for ~136 s ~29 s
partial fix, probe disabled none (starttimeout=3600) 0 for the whole run never (terminal)
full fix producer-level backstop 0 until first abandon ~63 s, bounded abandons

Replayed in throwaway containers, the old binary wedged terminally (no logs, hung process tree, consumers dead even after the source recovered). The patched binary recovered in ~6 s.

The fixes

  1. Redial outside the lock: reconnect() releases p.mu first. A stale dial landing late is stopped and discarded. The killer fix.
  2. Bound the pipe first-byte probe: the pipe/exec flow had no timeout (only RTSP did). It now shares the 30 s start-timeout.
  3. Kill the process group after a grace period: ~5 s for clean exit, then SIGKILL pid + group, sweeping grandchildren.
  4. Force-reset after 3 strikes: three consecutive instant stale-state failures reset the producer to idle. Any success or different error resets the counter.

A remaining strand on the fresh-consumer Dial() route is closed by a later, separate patch (single-flight dial, epoch counter, producer-level dial backstop).

Honesty and prior art

  • Measured: the wedge, the goroutine dump, the ~194 s pre-fix run, the ~6 s patched recovery. A mutex-deadlock regression test passes under the race detector.
  • Plausible mechanism (not fully proven): the exact consumer/reconnect interleaving reaching the terminal state.
  • Prior art (partial novelty): upstream go2rtc issues cover the exec/reconnect hang symptom class, credit there. No upstream report identifies the root cause: the mutex held across an unbounded redial.

Licensed under the site footer’s CC BY 4.0. If this saved you a debugging session, the optional thanks link in the footer is appreciated, no obligation.

Validation (2026-08-31)

Version: go2rtc HEAD c245815 (2026-07-13), app.Version = "1.9.14", 1 commit past the v1.9.14 tag (github.com/AlexxIT/go2rtc).

Source-confirmed: YES (not fixed upstream). Both Dial() (producer.go:58/62) and reconnect() (producer.go:177/187) call GetProducer(p.url) while holding p.mu. For an exec: pipe source the chain ends in exec.handlePipe (exec/exec.go:155) → magic.Open → Peek(4), blocking on the pipe read with no timeout (only handleRTSP applies the 30 s starttimeout). stop() (producer.go:246) also needs the mutex. Symptom class matches upstream issues #1204/#1243/#1084.

Upstream state (2026-09-09):

  • internal/streams/producer.go and internal/exec/exec.go on master are byte-identical to c245815.
  • The exec handler rewrite (commit 6fb5994, 2025-02-23) changed internal/exec and pkg/shell only. It closed PR #1246, a double cmd.Wait() hang from issue #1243, and does not touch producer.go.
  • Open PR #1941 (2025-11-17, unmerged) adds a stop channel to worker() and stop(). Its reconnect() hunk changes only the go p.worker(...) call. GetProducer stays under p.mu in both Dial() and reconnect().
  • Issue #1611 (open, maintainer’s last comment 2025-03-12: replication unclear) reports the terminal symptom on ffmpeg: sources. Those dial through handleRTSP, which has the 30 s bound, so the unbounded hold reproduced here is pipe-mode only. Whether it explains #1611 is not established.

Runtime repro: REPRODUCED. HEAD built in Docker (golang:1.24-alpine). Config test: exec:sleep 3600 (opens stdout, never writes). All concurrent /api/stream.mp4?src=test consumers hung. A SIGQUIT goroutine dump showed the lock holder parked in Dial → GetProducer → exec.handlePipe → magic.Open → Peek/Read, with waiters queued in sync.Mutex.Lock at producer.go:58.

Verdict: validated. Source-confirmed and runtime-reproduced on go2rtc 1.9.14 (HEAD, current), unfixed upstream. The four fixes address the reproduced failure.