Document the multi-network join race and its fix

Records the strace-based root-cause investigation for the bug where
joining 2+ networks in one -r/--run left every network after the first
permanently unreachable, and the retry-on-EIO/ENETDOWN fix applied in
network_tap_relay.cpp, matching the level of detail already recorded for
the extern-connectivity investigation above it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv3s5jckJKzh6JkMoi2Akz
This commit is contained in:
2026-08-31 17:00:25 +00:00
parent 244a814e85
commit 37ab2fcb1e
2 changed files with 106 additions and 0 deletions
+34
View File
@@ -1264,6 +1264,40 @@ Source layout (all under `src/`):
`tap-relay create/attach/teardown` case (updated per `self_test.{h,cpp}`'s
own entry above) passes reliably across repeated runs.
**Real bug reported from the real target device: joining 2+ networks in
one `-r/--run` left every network after the first permanently
unreachable, regardless of extern/intern.** Root-caused via `strace -f`
on the real device (the user's own suggestion, after several
inconclusive timing-based experiments): the second network's own relay
died on its literal first frame — `write(fd_container, ..., 86) = -1
EIO`, immediately followed by `exit_group(0)`. `EIO` writing to a tap fd
means the device isn't administratively up yet, and it genuinely wasn't:
`create_tap_relay()` returns, and this relay starts polling, the instant
the container-side tap device is *created*; `join_one_network()`
(`network_join.cpp`), a *different* process, still has its own `ip addr
add`/`ip link set <if> up` steps left to run afterward for that same
device — confirmed via the trace's own timestamps, `ip addr add ... dev
eth1` ran *after* the relay's fatal write. For the first network joined
this race is narrow enough that no frame ever arrives first; for the
second (and any later) network, something reliably delivers a frame
before the interface is up, and the previous code treated any `write()`
failure as fatal — exiting for good on that single `EIO`, so the network
never worked again for the rest of the session. Fixed in the relay's
frame-forwarding loop by retrying specifically on `EIO`/`ENETDOWN` (both
mean "not up yet", a startup race, not a torn-down namespace) with a
short bounded backoff (up to 50 × 20ms = 1s) instead of exiting
immediately — generous compared to the ~14ms gap actually observed in
the trace. **Methodology note**: an earlier `strace -f` attempt, wrapped
in `timeout 30`, produced a misleadingly corrupted trace — GNU `timeout`
sends its kill signal to the whole process group by default, and
`strace`'s own tracing overhead was large enough (mount steps that
normally take seconds took 3+ minutes under trace) that the real
wall-clock timeout elapsed mid-setup, killing several traced children
prematurely; dropping the `timeout` wrapper entirely produced a clean,
complete trace. **Verified end-to-end on the real target device** with
both 2 and 3 `intern` networks joined simultaneously in one session, all
gateways reachable at 0% packet loss, clean teardown, no leftover state.
**Crash-orphan sweep**, the direct tap+relay analog of `port_forward.h`'s
own (see its own entry below): unlike a veth pair or a session's own
bridge/persistent-namespace state, a relay process is host-global state