Document the multi-network join race and its fix
Records the strace-based root-cause investigation for the bug where joining 2+ networks in one -r/--run left every network after the first permanently unreachable, and the retry-on-EIO/ENETDOWN fix applied in network_tap_relay.cpp, matching the level of detail already recorded for the extern-connectivity investigation above it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gv3s5jckJKzh6JkMoi2Akz
This commit is contained in:
@@ -577,6 +577,78 @@ minutes) on the network tested against, for both the privacy address and the
|
||||
normally long-lived EUI64 address, too short-lived to build a stable
|
||||
addressing scheme on top of.
|
||||
|
||||
## Resolved: joining 2+ networks left every network after the first unreachable
|
||||
|
||||
**Reported from the real target device**: joining a container to two or more
|
||||
networks in a single `-r/--run` left every network after the first
|
||||
permanently unreachable — `eth0`, `eth1`, `eth2` all appeared inside the
|
||||
container, but only `eth0`'s gateway ever answered a ping. Reproduced
|
||||
regardless of whether the networks involved were `extern` or `intern`, so
|
||||
the uplink mechanism above was ruled out as a cause early on.
|
||||
|
||||
Several timing-based experiments (explicit delays before/after join,
|
||||
varying join order) gave mixed, sometimes-contradictory results — the
|
||||
device's own background load turned out to vary wildly enough (a plain
|
||||
`sleep 5` observed taking anywhere from ~5s to ~60s of real wall-clock time)
|
||||
that "adding a delay fixed it" couldn't be trusted as a signal at all. At
|
||||
the user's own suggestion, installing `strace` on the device and tracing the
|
||||
whole invocation (`strace -f -tt`, following every forked/exec'd process
|
||||
with absolute timestamps) turned this from an inconclusive guessing exercise
|
||||
into a five-minute, conclusive one.
|
||||
|
||||
**Root cause**: the trace showed the *second* network's own relay process
|
||||
dying on its literal first frame:
|
||||
|
||||
```
|
||||
write(4, "...", 86) = -1 EIO (Input/output error)
|
||||
exit_group(0)
|
||||
```
|
||||
|
||||
`EIO` writing to a tap fd means the device isn't administratively up yet —
|
||||
and it genuinely wasn't. `create_tap_relay()` (`network_tap_relay.cpp`)
|
||||
returns, and its relay starts polling immediately, the instant the
|
||||
container-side tap device is *created*; `join_one_network()`
|
||||
(`network_join.cpp`), a separate process, still has its own `ip addr add`/
|
||||
`ip link set <if> up` steps left to run afterward for that same device. The
|
||||
trace's own timestamps confirmed the ordering directly: `ip addr add ... dev
|
||||
eth1` ran *after* the relay's fatal write, not before. For the first network
|
||||
joined, this race is narrow enough in practice that no frame ever arrives
|
||||
before those steps finish; for the second (and any later) network, something
|
||||
reliably delivers a frame before the interface is up — and the relay's
|
||||
frame-forwarding loop treated *any* `write()` failure as fatal, exiting
|
||||
silently and permanently on that single `EIO`, so that network never worked
|
||||
again for the rest of the session.
|
||||
|
||||
**Fix**: retry specifically on `EIO`/`ENETDOWN` (both mean "the device isn't
|
||||
up yet," a startup race, not a torn-down namespace) with a short bounded
|
||||
backoff — up to 50 × 20ms = 1s, generous compared to the ~14ms gap actually
|
||||
observed in the trace — instead of exiting immediately. No change was needed
|
||||
to `join_one_network()`'s own ordering; the relay simply waits out the gap.
|
||||
|
||||
**A tempting shortcut ruled out**: reordering `join_one_network()` to bring
|
||||
the interface up *before* creating the container-side tap device isn't
|
||||
possible — the device has to exist before it can be addressed or brought
|
||||
up, so some gap between "device exists" (when the relay starts polling) and
|
||||
"device is up" (when the relay can actually forward into it) is unavoidable
|
||||
by construction. Retrying in the relay is the only fix that doesn't require
|
||||
the relay to somehow block until an unrelated process finishes its own,
|
||||
separate setup steps.
|
||||
|
||||
**Methodology note**: an earlier `strace -f` attempt, wrapped in `timeout
|
||||
30`, produced a misleadingly corrupted trace instead of a clean one — GNU
|
||||
`timeout` sends its kill signal to the whole process group by default, and
|
||||
`strace`'s own tracing overhead was large enough (mount steps that normally
|
||||
take seconds took 3+ minutes under trace) that the real wall-clock timeout
|
||||
elapsed while still in early setup, silently killing several traced children
|
||||
mid-flight and producing a trace that looked like a crash but was really
|
||||
just the test harness cutting it off. Dropping the `timeout` wrapper
|
||||
entirely and letting the traced command finish naturally produced a clean,
|
||||
complete trace that correctly captured the real bug.
|
||||
|
||||
**Verified end-to-end on the real target device**: both 2 and 3 `intern`
|
||||
networks joined simultaneously in one session, all gateways reachable at 0%
|
||||
packet loss, clean teardown, no leftover state.
|
||||
|
||||
## IPv6: no NAT (MASQUERADE), by design
|
||||
|
||||
**Trigger**: the real target device's `ip6tables` build doesn't support a
|
||||
|
||||
Reference in New Issue
Block a user