Document the multi-network join race and its fix

Records the strace-based root-cause investigation for the bug where
joining 2+ networks in one -r/--run left every network after the first
permanently unreachable, and the retry-on-EIO/ENETDOWN fix applied in
network_tap_relay.cpp, matching the level of detail already recorded for
the extern-connectivity investigation above it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv3s5jckJKzh6JkMoi2Akz
This commit is contained in:
2026-08-31 17:00:25 +00:00
parent 244a814e85
commit 37ab2fcb1e
2 changed files with 106 additions and 0 deletions
+72
View File
@@ -577,6 +577,78 @@ minutes) on the network tested against, for both the privacy address and the
normally long-lived EUI64 address, too short-lived to build a stable
addressing scheme on top of.
## Resolved: joining 2+ networks left every network after the first unreachable
**Reported from the real target device**: joining a container to two or more
networks in a single `-r/--run` left every network after the first
permanently unreachable — `eth0`, `eth1`, `eth2` all appeared inside the
container, but only `eth0`'s gateway ever answered a ping. Reproduced
regardless of whether the networks involved were `extern` or `intern`, so
the uplink mechanism above was ruled out as a cause early on.
Several timing-based experiments (explicit delays before/after join,
varying join order) gave mixed, sometimes-contradictory results — the
device's own background load turned out to vary wildly enough (a plain
`sleep 5` observed taking anywhere from ~5s to ~60s of real wall-clock time)
that "adding a delay fixed it" couldn't be trusted as a signal at all. At
the user's own suggestion, installing `strace` on the device and tracing the
whole invocation (`strace -f -tt`, following every forked/exec'd process
with absolute timestamps) turned this from an inconclusive guessing exercise
into a five-minute, conclusive one.
**Root cause**: the trace showed the *second* network's own relay process
dying on its literal first frame:
```
write(4, "...", 86) = -1 EIO (Input/output error)
exit_group(0)
```
`EIO` writing to a tap fd means the device isn't administratively up yet —
and it genuinely wasn't. `create_tap_relay()` (`network_tap_relay.cpp`)
returns, and its relay starts polling immediately, the instant the
container-side tap device is *created*; `join_one_network()`
(`network_join.cpp`), a separate process, still has its own `ip addr add`/
`ip link set <if> up` steps left to run afterward for that same device. The
trace's own timestamps confirmed the ordering directly: `ip addr add ... dev
eth1` ran *after* the relay's fatal write, not before. For the first network
joined, this race is narrow enough in practice that no frame ever arrives
before those steps finish; for the second (and any later) network, something
reliably delivers a frame before the interface is up — and the relay's
frame-forwarding loop treated *any* `write()` failure as fatal, exiting
silently and permanently on that single `EIO`, so that network never worked
again for the rest of the session.
**Fix**: retry specifically on `EIO`/`ENETDOWN` (both mean "the device isn't
up yet," a startup race, not a torn-down namespace) with a short bounded
backoff — up to 50 × 20ms = 1s, generous compared to the ~14ms gap actually
observed in the trace — instead of exiting immediately. No change was needed
to `join_one_network()`'s own ordering; the relay simply waits out the gap.
**A tempting shortcut ruled out**: reordering `join_one_network()` to bring
the interface up *before* creating the container-side tap device isn't
possible — the device has to exist before it can be addressed or brought
up, so some gap between "device exists" (when the relay starts polling) and
"device is up" (when the relay can actually forward into it) is unavoidable
by construction. Retrying in the relay is the only fix that doesn't require
the relay to somehow block until an unrelated process finishes its own,
separate setup steps.
**Methodology note**: an earlier `strace -f` attempt, wrapped in `timeout
30`, produced a misleadingly corrupted trace instead of a clean one — GNU
`timeout` sends its kill signal to the whole process group by default, and
`strace`'s own tracing overhead was large enough (mount steps that normally
take seconds took 3+ minutes under trace) that the real wall-clock timeout
elapsed while still in early setup, silently killing several traced children
mid-flight and producing a trace that looked like a crash but was really
just the test harness cutting it off. Dropping the `timeout` wrapper
entirely and letting the traced command finish naturally produced a clean,
complete trace that correctly captured the real bug.
**Verified end-to-end on the real target device**: both 2 and 3 `intern`
networks joined simultaneously in one session, all gateways reachable at 0%
packet loss, clean teardown, no leftover state.
## IPv6: no NAT (MASQUERADE), by design
**Trigger**: the real target device's `ip6tables` build doesn't support a