ceamac/slocker-lite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Create tap devices persistently via ip tuntap add, drop retry logic

Browse Source 
Real-device testing showed the previous retry-based fix (ad30f93) was
insufficient: a bare ioctl(TUNSETIFF)-created tap device (no IFF_PERSIST)
could work for one command and then vanish for the next on that kernel,
not just be slow to appear. Both tap devices are now created ahead of
time via an external `ip tuntap add dev <name> mode tap` before being
attached to via open_tap(), making them genuine persistent netdevices
with no tie to any fd/process lifetime -- the same technique QEMU/libvirt
use. stop_tap_relay() now explicitly `ip link del`s the host-side device
since it no longer disappears on its own; the crash-orphan sweep records
each relay's network kind/name too so it can do the same for orphans.
All retry logic (network_join.cpp's run_with_retry(), self_test.cpp's
wait_for_container_device_visible()) is removed as no longer needed.

self_test.cpp's post-teardown assertions updated to match: the host-side
device is now expected gone after stop_tap_relay(), while the
container-side device is expected to persist (it only goes away once its
own namespace is torn down, not merely because the relay stopped).

Verified end-to-end on the dev machine with --no-veth forcing the
fallback: eth0 stayed visible and usable across repeated commands with
no disappearance, and both gateway and outside (8.8.8.8) ping succeeded
at 0% loss.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv3s5jckJKzh6JkMoi2Akz
This commit is contained in:
Viorel Munteanu
2026-08-30 19:59:14 +00:00
parent ad30f93f77
commit bebf559039
6 changed files with 363 additions and 181 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+118 -47
View File
@@ -333,21 +333,25 @@ Source layout (all under `src/`):
trusting its pid. Confirms the host-side tap gets created and attached to
the bridge (`ip link show` output contains `master <bridge>`), the
container-side tap gets created with the requested name inside the target
namespace (checked via `nsenter --net=/proc/<pid>/ns/net -- ip link show`,
wrapped in `wait_for_container_device_visible()` — a `.cpp`-local bounded
retry, same shape as `network_join.cpp`'s own `run_with_retry()`, added
for the same real device-reported reason — see that function's own doc
comment for the full story, including the tempting-but-harmful internal-
self-check "fix" that was tried and ruled out first, confirmed by this
exact self-test: it started reliably failing, 100% of the time, the moment
that harmful check was added — despite having passed reliably many times
earlier the same session before that change — and immediately went back to
passing reliably once the harmful check was removed again), and — the
biggest previously-unverified assumption from `docs/networking-design.md`'s
tap+relay addendum — **both devices actually disappear on their own once
`stop_tap_relay()` stops the relay process, with no explicit `ip link del`
needed** (neither was created with `IFF_PERSIST`) — confirmed directly,
many times over, on this dev machine (root, via the scoped `doas` rule).
namespace (checked via a plain, single-shot
`nsenter --net=/proc/<pid>/ns/net -- ip link show`; an earlier version
wrapped this in a bounded retry, added when tap devices were still created
via bare `ioctl(TUNSETIFF)` and intermittently weren't immediately
visible — see `network_tap_relay.{h,cpp}`'s own entry below for why that
retry, and the whole class of symptom it was compensating for, is gone
now that devices are created persistently instead), and — **updated once
the original "both devices disappear on their own once `stop_tap_relay()`
stops the relay, no explicit `ip link del` needed" assumption turned out
to be wrong on the real target device** (`network_tap_relay.{h,cpp}`'s own
entry below has the full story) — now confirms the *host*-side device is
explicitly gone after `stop_tap_relay()` (the new `ip link del` step)
while the *container*-side device deliberately still exists (correctly
persistent — only the relay stopped, not the container's own network
namespace); the container-side device's actual disappearance, once that
namespace itself is torn down, isn't separately re-checked (`nsenter` has
nothing left to target once the namespace's only holding process has
already exited) — it's destroyed moments later anyway, at the very end of
this test, when its throwaway namespace-holder process is killed.
Deliberately its own small file since more real tests are expected here as
more of the networking feature lands.
- `env_spec.{h,cpp}` — `resolve_env_specs()` turns an ordered list of
@@ -869,12 +873,10 @@ Source layout (all under `src/`):
/ns/net -- ip addr add 10.168.0.2/24 dev eth0` failing with `"Cannot find
device \"eth0\""` right after `network_tap_relay.h`'s relay had already
created it — and confirmed by hand that simply retrying the whole session a
few times eventually worked. `run_with_retry()` (`.cpp`-local, same
shape/spirit as `wait_for_isolated_net_namespace()` below) replaces `run()`
for the three steps that touch the just-created `container_if` (IPv4
address, IPv6 address, bringing it up) — retries quietly (no per-attempt
log spam) for up to 500ms before giving up loudly, same as `run()`'s own
single-attempt error.
few times eventually worked. A first fix added a bounded (~500ms) retry
around the steps that touch the just-created `container_if` — later found
insufficient (see below) and removed again; `join_one_network()` now uses a
plain, single-attempt `run()` for every step, same as before any of this.
**A tempting "fix" investigated and ruled out by direct A/B testing on this
dev machine, not just reasoned about**: the obvious first instinct — have
@@ -892,17 +894,27 @@ Source layout (all under `src/`):
still holding it open, immediately after device creation, appears to
corrupt the device's *external* visibility specifically on this kernel;
the *same* process's own view of the device it just created stayed correct
throughout). Confirmed via direct A/B testing, not just correlation:
removing that internal check immediately restored 100%-reliable, zero-retry
first-try success on three separate real sessions; re-adding it reliably
broke it again, including for a from-scratch self-test reproduction (see
`self_test.{h,cpp}` below) that had passed reliably many times earlier in
the same session before this specific change. The fix that shipped is
unambiguous: never add an internal, same-process/fd-holding self-check to
the relay; only the *external*, separate-process retry above
(`run_with_retry()`) is safe, and is sufficient on its own — every retried
real-device call this session succeeded on the very first attempt once the
harmful internal check was gone.
throughout). The lesson that survived into the final fix: never add an
internal, same-process/fd-holding self-check to the relay.
**The retry fix above turned out to be insufficient**: a further round of
real-device testing showed a *different* failure — `ip addr add` against
the container-side device would sometimes succeed, only for the very next
command against that same device (`ip link set eth0 up`) to fail with
"Cannot find device", exhausting every retry. The device wasn't merely
slow to become visible after creation; it was **disappearing on its own**,
consistent with the underlying `ioctl(TUNSETIFF)`-created device (no
`IFF_PERSIST`) having a more fragile lifetime on that kernel than "stays
alive as long as the one fd that created it stays open." Per the user's
own suggested direction, the fix was structural, not another retry: both
tap devices are now created ahead of time via an external `ip tuntap add
dev <name> mode tap` (`create_persistent_tap()`, `network_tap_relay.cpp` —
see that file's own entry below for the full detail), which sidesteps the
whole class of symptom by making the device a genuinely persistent
netdevice with no tie to any fd or process. All retry logic (`run()` is
used unconditionally, everywhere) was removed as part of this — the
earlier retry was compensating for a problem this fix removes outright,
not one it makes more likely to need retrying.
- `port_forward.{h,cpp}` — implements `-p`. `parse_port_forward_spec()`
splits `"[<network>:]<host-port>:<container-port>"` on `':'` (2 or 3
fields; the network name is deliberately restricted to excluding `':'` --
@@ -1021,13 +1033,21 @@ Source layout (all under `src/`):
caller-provided (not derived here) specifically so a future caller
(`join_one_network()`, once this is wired in) can reuse its own existing
fnv1a-based veth-naming scheme rather than this file growing a second,
drifting copy of that six-line hash. `open_tap()` (`.cpp`-local) creates
each device via a direct `open("/dev/net/tun")` + `ioctl(TUNSETIFF,
IFF_TAP | IFF_NO_PI)` — `IFF_NO_PI` so both ends agree on raw-frame
framing with no extra header, deliberately **not** `IFF_PERSIST`, so each
device is expected to disappear on its own once its one-and-only fd
closes, the same "no explicit teardown" property veth already has (see
`self_test.{h,cpp}` above for where this got verified). The relay child's
drifting copy of that six-line hash. Each device is now created two-step:
`create_persistent_tap(name)` (`.cpp`-local) first runs an external `ip
tuntap add dev <name> mode tap`, then `open_tap()` (`.cpp`-local, mostly
unchanged) `open("/dev/net/tun")` + `ioctl(TUNSETIFF, IFF_TAP |
IFF_NO_PI)`s onto that already-existing device (`IFF_NO_PI` so both ends
agree on raw-frame framing with no extra header) — `open_tap()` now only
*attaches* an fd to a device, it no longer *creates* one. **This split
replaced an earlier, simpler design** where `open_tap()` alone both
created (via the same `ioctl`, with no `IFF_PERSIST`) and attached, on the
assumption the device would then simply disappear on its own once its
one-and-only fd closed, the same "no explicit teardown" property veth
already has — see this entry's own "tap devices need to be created
persistently" paragraph further down for why that assumption turned out
to be wrong on the real target device, and `docs/networking-design.md`'s
matching section for the full incident writeup. The relay child's
entire setup sequence — enter the network's own namespace first if
`intern` (`persistent_netns_path()`, `persistent_netns.h`), create+attach
the host-side tap, `setns()` into the container's namespace (the
@@ -1041,9 +1061,17 @@ Source layout (all under `src/`):
between the two fds — this loop *is* the actual "veth wire," just
implemented once in userspace instead of by the kernel. No `SIGTERM`
handler is installed in the relay: default disposition (terminate) already
closes both fds on the way out, which is all `stop_tap_relay()`'s
"no explicit teardown" contract needs. `stop_tap_relay()` sends `SIGTERM`
and reaps the process. Once `create_tap_relay()` returns successfully,
closes both fds on the way out. `stop_tap_relay()` sends `SIGTERM`, reaps
the process, then explicitly `ip link del`s the host-side device
(`wrap_for_network(handle.network, ...)`, reaching wherever it lives —
host root for `extern`, the network's own persistent namespace for
`intern` — hence `TapRelayHandle` carrying its own `NetworkEntry`) — now
required since the device is persistent and no longer disappears just
because the relay's fd closed (see below). The container-side device
needs no matching step: it lives inside the container's own network
namespace, which the kernel already tears down (every interface inside
it, persistent or not, along with it) once the session itself ends.
Once `create_tap_relay()` returns successfully,
`container_if_name` is a completely ordinary interface from the
container's own point of view — `join_one_network()`'s existing IP
assignment/route/DNAT-target-address code (unchanged, not yet wired to
@@ -1087,6 +1115,27 @@ Source layout (all under `src/`):
process, not just bwrap, is a separate, already-scoped concern — see the
crash-orphan sweep below), the added complexity wasn't worth it.
**Real bug reported from the real target device: tap devices need to be
created persistently, not tied to the relay's own fd lifetime.** Two
rounds of real-device testing (`network_join.cpp`'s own entry above has
the full incident writeup) found the container-side device intermittently
either not immediately visible after creation, or — worse, found on the
second round — visible and usable for one command (e.g. `ip addr add`
succeeding) and then gone for the very next one (`ip link set ... up`
failing with "Cannot find device"), on a kernel where the bare
`ioctl(TUNSETIFF)`-created (no `IFF_PERSIST`) device evidently has a more
fragile lifetime than "stays alive as long as its one creating fd stays
open." Per the user's own suggested direction, the fix (see
`create_persistent_tap()` above) creates both tap devices ahead of time
via an external `ip tuntap add dev <name> mode tap` — the same technique
QEMU/libvirt use to let an unprivileged process attach to a tap device set
up ahead of time — turning each into a genuinely persistent netdevice with
no tie to any fd or process at all, the same as a veth pair already is.
All retry logic from the first round's fix (`network_join.cpp`'s
`run_with_retry()`, `self_test.cpp`'s `wait_for_container_device_visible()`)
was removed once this structural fix made it unnecessary — see both
files' own entries.
**Verified end-to-end on this dev machine (root, via the scoped `doas`
rule), using a `--no-veth` `extern` network specifically to exercise this
path**: two real containers joined the same network, each getting a
@@ -1115,6 +1164,18 @@ Source layout (all under `src/`):
`--delete-network-full` exists specifically so this class of
stale-state-masking-as-a-bug can't recur.
**Re-verified end-to-end on this dev machine after the persistent-device
redesign above**, again with `--no-veth` forcing the fallback: a single
container repeatedly used its tap-relay-backed `eth0` across several
commands in a row (`ip link show`, `ip addr show`, two rounds of `ping`)
with no disappearance between commands — the exact symptom the real
device hit — and both gateway ping and outside/internet ping (`8.8.8.8`)
succeeded at 0% loss. Session cleanup left no leftover host-side tap
device behind (only the bridge itself, deliberately left standing per
this project's reboot-reconciliation design); `-t/--test`'s own
`tap-relay create/attach/teardown` case (updated per `self_test.{h,cpp}`'s
own entry above) passes reliably across repeated runs.
**Crash-orphan sweep**, the direct tap+relay analog of `port_forward.h`'s
own (see its own entry below): unlike a veth pair or a session's own
bridge/persistent-namespace state, a relay process is host-global state
@@ -1127,8 +1188,14 @@ Source layout (all under `src/`):
`session_pid_file_path()`/`port_forward_state_path()` already use, so
`clean_stale_tap_relays()` can cross-reference filenames directly against
`list_sessions()`'s own `SessionInfo::path`. `record_tap_relays()` writes
one line per relay (`"<relay_pid> <host_tap_name>"`) to that path — a
no-op if there's nothing to record. `clean_stale_tap_relays()`
one line per relay (`"<relay_pid> <host_tap_name> <kind> <network_name>"`,
`<kind>` = `"extern"`/`"intern"`, `<network_name>` last since it's the one
field that can contain whitespace) to that path — a no-op if there's
nothing to record; the `<kind>`/`<network_name>` fields were added
alongside the persistent-tap-device redesign above, so a later sweep can
reconstruct a `NetworkEntry` and reach the right namespace to remove the
now-persistent host-side device too, not just kill the relay process.
`clean_stale_tap_relays()`
(`commands.cpp`'s `clean_processes_command()`, alongside
`clean_stale_sessions()`/`clean_stale_port_forwards()`) scans that
directory: a record whose filename doesn't match any currently-*running*
@@ -1136,9 +1203,13 @@ Source layout (all under `src/`):
already-dead pid, or one this process was never the parent of, isn't
treated as an error, since this sweep runs from a *separate* later
invocation that can't `waitpid()` an orphan it didn't fork — its true
parent's own exit, or `init` after reparenting, reaps it) before the
record file itself is deleted; a record whose session is still running is
left completely untouched. **Verified via a controlled scratch test**,
parent's own exit, or `init` after reparenting, reaps it), the host-side
tap device it named is removed (`ip link del`, via
`wrap_for_network()` against a `NetworkEntry` reconstructed from the
record's own `<kind>`/`<network_name>` fields — best-effort, same as
`stop_tap_relay()`'s own removal) before the record file itself is
deleted; a record whose session is still running is left completely
untouched. **Verified via a controlled scratch test**,
the same shape `port_forward.h`'s own sweep test used: root wasn't needed
for the sweep *logic* itself (only real tap/bridge creation needs it),
so this ran as a plain rootless daemonized session (`-D`, no `-n`) to get