Create tap devices persistently via ip tuntap add, drop retry logic

Real-device testing showed the previous retry-based fix (ad30f93) was
insufficient: a bare ioctl(TUNSETIFF)-created tap device (no IFF_PERSIST)
could work for one command and then vanish for the next on that kernel,
not just be slow to appear. Both tap devices are now created ahead of
time via an external `ip tuntap add dev <name> mode tap` before being
attached to via open_tap(), making them genuine persistent netdevices
with no tie to any fd/process lifetime -- the same technique QEMU/libvirt
use. stop_tap_relay() now explicitly `ip link del`s the host-side device
since it no longer disappears on its own; the crash-orphan sweep records
each relay's network kind/name too so it can do the same for orphans.
All retry logic (network_join.cpp's run_with_retry(), self_test.cpp's
wait_for_container_device_visible()) is removed as no longer needed.

self_test.cpp's post-teardown assertions updated to match: the host-side
device is now expected gone after stop_tap_relay(), while the
container-side device is expected to persist (it only goes away once its
own namespace is torn down, not merely because the relay stopped).

Verified end-to-end on the dev machine with --no-veth forcing the
fallback: eth0 stayed visible and usable across repeated commands with
no disappearance, and both gateway and outside (8.8.8.8) ping succeeded
at 0% loss.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv3s5jckJKzh6JkMoi2Akz

docs/networking-design.md

@@ -407,9 +407,13 @@ Landed as four commits (a fifth, this doc update, closes it out) — see
    Verified via a new `-t/--test` case: a throwaway bridge + throwaway
    network namespace, confirming the host-side tap attaches to the bridge,
    the container-side tap appears inside the target namespace with the
-   requested name, and — the biggest assumption going in — both devices
-   disappear on their own once the relay is stopped, no explicit
-   `ip link del` needed (neither is created with `IFF_PERSIST`).
+   requested name. **The original assumption here — that both devices
+   disappear on their own once the relay is stopped, since neither was
+   created with `IFF_PERSIST` — turned out to be wrong on the real target
+   device** (see "Resolved: tap devices need to be created persistently,
+   not tied to the relay's own fd lifetime" below); devices are now created
+   persistently via an external `ip tuntap add`, and the host-side one is
+   explicitly removed (`ip link del`) when the relay stops.
 3. **Wire into `join_one_network()`/`join_networks()`.** `JoinedNetwork`
    gains an optional `relay` handle; `run_container()` collects and stops
    them after `run_bwrap()` returns, mirroring `-p`'s own
@@ -532,11 +536,11 @@ failing with `"Cannot find device \"eth0\""` immediately after the relay had
 already created it — confirmed by hand that retrying the whole session a few
 times eventually let it succeed.
 
-**Fix**: `network_join.cpp`'s `join_one_network()` now retries (bounded,
-~500ms, quiet until final success/give-up) the three steps that touch the
-just-created container interface — IPv4 address, IPv6 address, bringing it
-up — via a new `run_with_retry()` instead of the plain `run()` used
-elsewhere.
+**First fix tried, later superseded (see next section)**: `network_join.cpp`'s
+`join_one_network()` retried (bounded, ~500ms, quiet until final
+success/give-up) the three steps that touch the just-created container
+interface — IPv4 address, IPv6 address, bringing it up — via a
+`run_with_retry()` instead of the plain `run()` used elsewhere.
 
 **A tempting "fix" investigated and ruled out by direct A/B testing on the
 dev machine, not just reasoned about**: the obvious first instinct — have
@@ -554,9 +558,86 @@ change was reverted. Root cause not fully understood (something about
 forking a subprocess that inherits the tap fd — deliberately not
 `O_CLOEXEC` — while still holding it open, immediately after device
 creation, appears to corrupt the device's *external* visibility on this
-kernel specifically), but the fix that shipped is unambiguous: never add an
-internal, same-process/fd-holding self-check to the relay; the external,
-separate-process retry above is safe and sufficient on its own.
+kernel specifically), but the lesson that survived into the final fix is
+unambiguous: never add an internal, same-process/fd-holding self-check to
+the relay.
+
+## Resolved: tap devices need to be created persistently, not tied to the relay's own fd lifetime
+
+**Trigger**: the retry fix above turned out to be insufficient — a further
+round of real-device testing (a second `run.log`) showed a *different*
+failure pattern: `ip addr add` against the container-side device would
+sometimes *succeed*, only for the very next command against that same
+device (`ip link set eth0 up`) to fail with "Cannot find device", exhausting
+every retry. The user's own diagnosis, confirmed correct: the device wasn't
+merely slow to become visible after creation (the earlier theory) — it was
+actually **disappearing on its own**, on this kernel, independent of
+anything this project's own code was doing to it. This is consistent with
+the underlying device having been created via a plain `ioctl(fd,
+TUNSETIFF, &ifr)` with no `IFF_PERSIST` flag: on that kernel, its lifetime
+seems to be tied to something more fragile than "the one fd that created
+it stays open" (the relay process never closes or re-opens its own fds
+around any of this) — never fully root-caused, and not worth chasing
+further once a structurally different approach removed the whole class of
+symptom.
+
+**Fix, per the user's own suggestion**: stop relying on `ioctl(TUNSETIFF)`
+alone to *create* the device at all. Both the host-side and container-side
+tap devices are now created ahead of time by an external `ip tuntap add dev
+<name> mode tap` command (`create_persistent_tap()`,
+`network_tap_relay.cpp`) — the same technique QEMU/libvirt use to let an
+unprivileged process attach to a tap device someone else set up — and only
+*attached to* afterward via the existing `open_tap()`'s `open("/dev/net/tun")`
++ `ioctl(TUNSETIFF)` call (unchanged; it no longer creates, only opens an
+fd onto an already-existing device). A device created this way is a first-
+class, persistent netdevice from the kernel's point of view, with no tie to
+any single fd or process at all — the same reason `ip tuntap add`/`ip
+link add ... type veth` never need an owning process to stay alive either.
+
+**Consequence: teardown is no longer automatic.** Since neither device
+disappears on its own once the relay stops, `stop_tap_relay()` now
+explicitly `ip link del`s the host-side device after reaping the relay
+process (wrapped via `wrap_for_network()` to reach wherever it lives — host
+root for `extern`, the network's own persistent namespace for `intern`).
+The container-side device needs no equivalent step: it lives inside the
+container's own network namespace, which the kernel already tears down
+(taking every interface inside it along, persistent or not) once the
+session itself ends — nothing new required there. The crash-orphan sweep
+(`record_tap_relays()`/`clean_stale_tap_relays()`) was extended the same
+way: its state file now also records each relay's network kind/name (not
+just its pid and host-side device name), so a sweep for a crashed session
+can reconstruct a `NetworkEntry` and reach the right namespace to remove
+the orphaned host-side device, not just kill the orphaned relay process.
+
+**All retry logic from the previous fix was dropped**, per the user's own
+explicit instruction, once persistent creation removed the underlying
+disappearing-device problem it was compensating for: `network_join.cpp`'s
+`run_with_retry()` is gone (reverted to the plain `run()` used everywhere
+else), and `self_test.cpp`'s matching `wait_for_container_device_visible()`
+retry helper is gone too.
+
+**`self_test.cpp`'s own expectations updated accordingly**: the test used to
+assert that *both* the host-side and container-side devices vanish on their
+own once `stop_tap_relay()` stops the relay — exactly the assumption this
+fix disproves. It now asserts the host-side device is gone (the new
+explicit `ip link del` step) while the container-side device is still
+present (correctly persistent, since only the relay stopped, not the
+container's own namespace) — the namespace itself is destroyed moments
+later, at the very end of the test, when its throwaway holder process is
+killed.
+
+**Verified end-to-end on this dev machine, `--no-veth` forcing the fallback
+path** (root, via the scoped `doas` rule): a fresh `extern` network's
+container repeatedly used its tap-relay-backed `eth0` across several
+commands in a row (`ip link show`, `ip addr show`, ping) with no
+disappearance; gateway ping (0% loss) and outside/internet ping to
+`8.8.8.8` (0% loss) both worked; a second round of `ip link
+show`/`ip addr show` after the pings still saw the same device correctly.
+Session cleanup left no leftover host-side tap/veth devices behind (only
+the bridge itself, which is deliberately left standing per this project's
+reboot-reconciliation design). `-t/--test`'s own `tap-relay
+create/attach/teardown` case, updated as above, passes reliably across
+repeated runs.
 
 ## Explicitly out of scope for now