The tap+relay intern-ping test and the DNS hostname-ping test both
flaked intermittently (roughly 1 in 13-20 runs), always in a "found_success
== false" shape with no assertion-level clue as to why.
Root cause, confirmed via a temporary production-code diagnostic (since
reverted) and cross-referenced against network_join.cpp's own code: the
readiness poll only waited for eth0 to *exist* (`ip link show eth0`), but
an interface can become visible before join_one_network()'s own later
`ip addr add`/`ip link set ... up` steps for it have actually run. A
script that used the interface as soon as it merely existed could ping,
get "Network unreachable" (no address yet), and exit almost immediately
-- and since a session's own sandboxed process is the sole occupant of
its pid/net namespace (--unshare-pid/--unshare-net), its exit destroys
that namespace outright. That, in turn, made whichever other nsenter
call was still in flight against the same namespace -- join_one_network()'s
own remaining steps, or the entirely separate per-session DNS resolver
(network_dns.cpp's start_dns_resolver(), which enters every networked
session's namespace regardless of whether --hostname was given) -- fail
with "No such file or directory" against a namespace that had already
collapsed underneath it.
This is the same general class of race network_join.{h,cpp}'s own
CLAUDE.md entry already documents (a very-short-lived sandboxed command
can outrun its own concurrent network setup), just one step further than
the eth0-existence race already fixed earlier in this file -- a test-code
issue, not a production bug. Fixed by polling for an actually assigned
address on eth0 instead of mere existence, in both wait_for_eth0_then()
and BackgroundPeer's own inline readiness script.
Verified with the diagnostic in place that the DNS resolver's own
namespace lookup was never itself stale, isolating the cause to the
script's own premature exit. Re-verified extensively after the fix:
8/8 isolated repeats of the previously-flaky tap+relay test, and 6
consecutive full [integration][root][net] suite runs (78 test-case
executions total) with no failures.
Two peers on the same intern network, one started with --hostname
peer-a, resolve and ping each other by name via the per-session dnsmasq
resolver (network_dns.cpp). Skips cleanly when dnsmasq isn't available.
wait_for_hostname_then() retries the whole ping-by-name probe (not just
an eth0 existence check) until it succeeds or the bound is hit, since
this races against two independent things starting concurrently with
the sandboxed command: the interface coming up, and the per-session
resolver picking up the peer's own hosts record -- same underlying
join_networks() timing limitation the earlier wait_for_eth0_then() fix
was for. Verified end-to-end as root, both variants.
The positive counterpart to the intern-network isolation tests: a
container joined to an extern network gets a default route through its
uplink and can reach a real outside address. Verified end-to-end as
root, both variants.
Same shape as the intern-network peer-ping pair, on an extern network
instead: confirms same-bridge peer reachability still works once the
extern uplink (network_bridge.cpp's ensure_uplink_provisioned()) is also
provisioned alongside the bridge. Verified end-to-end as root, both
veth and tap+relay variants.
A container joined only to an intern network gets no default route
(network_join.cpp), so pinging a real outside address (8.8.8.8) must
fail outright, not merely succeed slower than an extern join would.
Verified end-to-end as root: both variants correctly report
"Network unreachable" and the tests assert a RESULT= line was seen
(the command actually ran) that isn't RESULT=0.
First of a planned series of end-to-end -n/--network join tests
(tests/integration/test_network_join_scenarios.cpp, [integration][root][net]):
two containers joined to the same intern network ping each other by IP,
run once with a real veth pair and once forced onto the tap+relay
fallback, since the two are genuinely different implementations. IPv6 is
deliberately excluded pending a known device-specific peculiarity.
split_lines_trimmed()/extract_marked_lines() moved from
test_rootless_run.cpp into tests/support/fixtures.{h,cpp} for reuse here.
wait_for_eth0_then() wraps a sandboxed command's own network-touching
script in a poll for eth0 to exist first: join_networks() runs
concurrently with, not before, the sandboxed command starting, so a
near-instant command can otherwise exit before its own join finishes --
the exact limitation already documented in network_join.{h,cpp}'s own
CLAUDE.md entry. Confirmed by testing (not assumed): without this, the
container's own immediate ping-and-exit sometimes raced ahead of the
veth-move step, which then failed outright ("Invalid netns value")
against an already-exited pid.
ceamac