Document the tap+relay veth fallback in the design doc
Adds a dedicated section to docs/networking-design.md covering the tap+relay fallback for veth-less kernels: the trigger (real target device supports tun/tap but not veth), why tap can't 1:1 replace veth, the confirmed design (two tap devices + a relay reusing the existing bridge, replacing veth's earlier N-way-switch-daemon sketch once bridge support was confirmed available), the four-commit implementation sequence with what testing actually found (the fd-leak deadlock, the reverted cgroup fix), and an honest writeup of the unresolved gateway/outside-reachability gap. README.md's -n/--network row now also flags that gap directly, next to the existing NAT-hairpinning limitation note for -p. This closes out the tap+relay fallback work for now: peer-to-peer connectivity through it is solid and dev-verified; gateway/outside reachability needs re-verification on the actual veth-less target device before being relied on. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gv3s5jckJKzh6JkMoi2Akz
This commit is contained in:
@@ -92,7 +92,7 @@ slocker-lite -V|--version
|
||||
| `--list-volumes` | List all named volumes (see `-v/--volume`) with their host directory. |
|
||||
| `--delete-volume <name>` | Remove a named volume from the config. The host directory is left untouched. |
|
||||
| `--delete-volume-full <name>` | Like `--delete-volume`, but also recursively deletes the volume's host directory. |
|
||||
| `-n, --network <name>` | Create/manage a persistent named network: requires exactly one of `--extern` (a real Linux bridge in the host's own namespace, with NAT/forwarding set up so containers on it reach the host's real network) or `--intern` (a bridge inside its own dedicated, routeless namespace, only reachable by other containers on the same network). `--subnet <cidr>` overrides the auto-allocated IPv4 range (`10.168.0.0/24`, incrementing per network); `--no-ipv6` disables (and `--subnet6 <cidr>` overrides) the auto-allocated IPv6 range, on by default. `--no-veth` forces the tap+relay join fallback even on a kernel that supports veth (useful for testing that path; it's otherwise chosen automatically whenever the running kernel lacks veth support). With `--run`, instead joins `<name>` to the container as its own `eth<N>` interface with an address from the network's subnet; repeatable, no membership limit. Root-only for now. See [`docs/networking-design.md`](docs/networking-design.md). |
|
||||
| `-n, --network <name>` | Create/manage a persistent named network: requires exactly one of `--extern` (a real Linux bridge in the host's own namespace, with NAT/forwarding set up so containers on it reach the host's real network) or `--intern` (a bridge inside its own dedicated, routeless namespace, only reachable by other containers on the same network). `--subnet <cidr>` overrides the auto-allocated IPv4 range (`10.168.0.0/24`, incrementing per network); `--no-ipv6` disables (and `--subnet6 <cidr>` overrides) the auto-allocated IPv6 range, on by default. `--no-veth` forces the tap+relay join fallback even on a kernel that supports veth (useful for testing that path; it's otherwise chosen automatically whenever the running kernel lacks veth support). With `--run`, instead joins `<name>` to the container as its own `eth<N>` interface with an address from the network's subnet; repeatable, no membership limit. Root-only for now. **Known gap**: on a network joined via the tap+relay fallback, peer-to-peer connectivity works, but reaching the network's own gateway (and thus the outside, for `extern`) does not yet — root cause unconfirmed, see [`docs/networking-design.md`](docs/networking-design.md). |
|
||||
| `--list-networks` | List all named networks (see `-n/--network`) with their kind, IPv4 subnet, and IPv6 subnet (or `(no ipv6)`). |
|
||||
| `--delete-network <name>` | Remove a named network from the config. |
|
||||
| `-p, --port-forward [<network>:]<host-port>:<container-port>` | With `--run`, forward a TCP port from the host into the container. `<network>` is optional, defaulting to the container's sole `--extern` network (an error if it joined more than one without specifying). Repeatable. Reachable via the host's real, externally-facing IP; `localhost`/loopback access has a known NAT-hairpinning limitation (see [`docs/networking-design.md`](docs/networking-design.md)). |
|
||||
|
||||
@@ -313,6 +313,118 @@ pass.
|
||||
attempted, and its file cleaned up, while a record matching a real
|
||||
running session was left untouched.
|
||||
|
||||
## TUN/TAP fallback for veth-less kernels
|
||||
|
||||
**Trigger**: the real target device's kernel supports `tun`/`tap`
|
||||
(`CONFIG_TUN` — Android needs this for `VpnService`-based VPN apps) but not
|
||||
`veth` (`CONFIG_VETH`, commonly stripped from mobile kernels), so `-n
|
||||
--extern`/`--intern` as designed above (a veth pair per join) simply can't
|
||||
work there at all — `ip link add ... type veth ...` fails outright. Bridge
|
||||
support was separately confirmed working on this same device, which rules
|
||||
out the more complex fallback this section originally considered (see
|
||||
`git log` on this file for the superseded sketch: a per-network userspace
|
||||
Ethernet switch with no bridge dependency at all) in favor of a much
|
||||
smaller design.
|
||||
|
||||
**Why tap can't just replace veth 1:1**: a veth pair is two real kernel
|
||||
netdevices, switched between (or into a bridge) entirely by the kernel with
|
||||
zero userspace involvement. A tap device only has *one* kernel-side
|
||||
netdevice — the other "end" is a raw-Ethernet-frame file descriptor that
|
||||
only a userspace process can read/write, so there's no second kernel
|
||||
endpoint to attach to a bridge. This is exactly why `slirp4netns`/QEMU's own
|
||||
tap networking need a userspace process on the fd side at all.
|
||||
|
||||
**Design, confirmed and implemented**: per network-join, two tap devices +
|
||||
one small relay process that copies bytes 1:1 between them — a direct
|
||||
functional substitute for one veth pair, **reusing the existing bridge as
|
||||
the switching fabric** so `provision_bridge()` needs no changes at all:
|
||||
|
||||
- A **host-side tap device**, created wherever the network's bridge lives
|
||||
and enslaved to it — exactly veth's host-side role.
|
||||
- A **container-side tap device**, created directly inside the container's
|
||||
own namespace, **named `eth<N>` from the start** — no peer-name-then-
|
||||
rename dance needed, unlike veth.
|
||||
- A **relay process** holding both fds open, copying raw Ethernet frames
|
||||
bidirectionally between them for as long as it runs. This *is* the "veth
|
||||
wire," just implemented once in userspace instead of by the kernel.
|
||||
|
||||
Once the container-side tap exists as `eth<N>`, everything downstream —
|
||||
IP assignment, routes, the address handed to `-p` — is completely
|
||||
unchanged; only the interface-creation step is swapped. Strategy selection
|
||||
is per-network, per-join: `should_use_veth(network) = network.veth &&
|
||||
probe_veth_support()`, mirroring the existing kernel-capability-vs-policy
|
||||
split `namespace_policy_enabled()` (`bwrap.cpp`) already uses for
|
||||
`--unshare-xxx`. `--no-veth` at network-creation time forces the fallback
|
||||
even on a veth-capable kernel — how this was actually tested, since the
|
||||
real target device wasn't available during development.
|
||||
|
||||
### Implementation plan: commit sequence
|
||||
|
||||
Landed as four commits (a fifth, this doc update, closes it out) — see
|
||||
`CLAUDE.md`'s own entries (`network_bridge.{h,cpp}`, `network_tap_relay.{h,cpp}`,
|
||||
`network_join.{h,cpp}`, `self_test.{h,cpp}`) for full file-by-file detail:
|
||||
|
||||
1. **Veth capability probe + `--no-veth` flag, no relay yet.**
|
||||
`probe_veth_support()` (fork, `unshare(CLONE_NEWNET)` into a throwaway
|
||||
namespace, try `ip link add ... type veth ...` there — the same
|
||||
kernel-capability-probing shape `bwrap.cpp`'s own
|
||||
`kernel_supports_namespace()` already uses); `NetworkEntry::veth` +
|
||||
YAML round-trip; `--no-veth` CLI flag. Verified: `probe_veth_support()`
|
||||
returns `true` on this dev machine (a real veth pair is created
|
||||
successfully); `--no-veth` persists `veth: false`.
|
||||
2. **`network_tap_relay.{h,cpp}`: relay creation/loop/teardown, standalone.**
|
||||
Verified via a new `-t/--test` case: a throwaway bridge + throwaway
|
||||
network namespace, confirming the host-side tap attaches to the bridge,
|
||||
the container-side tap appears inside the target namespace with the
|
||||
requested name, and — the biggest assumption going in — both devices
|
||||
disappear on their own once the relay is stopped, no explicit
|
||||
`ip link del` needed (neither is created with `IFF_PERSIST`).
|
||||
3. **Wire into `join_one_network()`/`join_networks()`.** `JoinedNetwork`
|
||||
gains an optional `relay` handle; `run_container()` collects and stops
|
||||
them after `run_bwrap()` returns, mirroring `-p`'s own
|
||||
`active_port_forwards` handling exactly. **Two real bugs found here, not
|
||||
assumed**: the relay child, unlike every other forked child in this
|
||||
project, never `exec()`s, so it inherited (and never closed) a live copy
|
||||
of `daemonize.cpp`'s own report-pipe write end, hanging `-D` combined
|
||||
with `-n` indefinitely until fixed with an explicit
|
||||
`close_inherited_fds()`; and a first-attempt fix making `--kill` reach
|
||||
the relay directly (adding its pid to the session's own cgroup) was
|
||||
reverted after it caused a *different* bug (the cgroup's own removal,
|
||||
which happens before `run_container()` gets to stop the relay, started
|
||||
failing with `EBUSY`) — the ordinary flow already stops the relay
|
||||
correctly on its own, so the added complexity wasn't worth it. Verified
|
||||
end-to-end: two containers on a `--no-veth extern` network got distinct
|
||||
addresses via two tap+relay pairs (no veth at all) and pinged each other
|
||||
with 0% packet loss, repeatably.
|
||||
4. **Crash-orphan sweep.** `record_tap_relays()`/`clean_stale_tap_relays()`,
|
||||
the direct structural analog of `-p`'s own sweep, wired into
|
||||
`--clean-processes`. Verified the same way the port-forward sweep was: a
|
||||
real rootless session's pid alongside a hand-written matching record
|
||||
(left untouched) and a fabricated stale one (correctly swept).
|
||||
|
||||
### Known gap: gateway/outside reachability unconfirmed
|
||||
|
||||
**Confirmed by testing, not yet root-caused.** Peer-to-peer connectivity
|
||||
through the tap+relay fallback is solid (verified above). Reaching the
|
||||
network's own gateway IP — and, in turn, the real outside through NAT — is
|
||||
**not**: neither ICMP nor a TCP `wget` ever got a response, despite ARP
|
||||
resolving correctly (the container's own `ip neigh` shows a `REACHABLE`
|
||||
entry with the gateway's real MAC, ruling out an L2/relay-framing problem).
|
||||
The identical bridge/subnet/host reached via veth instead of this fallback
|
||||
works perfectly — ruling out every environment-level explanation (host
|
||||
firewall, `rp_filter`, this dev sandbox's own networking) that would
|
||||
otherwise affect both paths equally, since those apply regardless of which
|
||||
mechanism connects the container. `rp_filter=0` (tried at the host-tap,
|
||||
bridge, and global `all` scope) did not fix it. Diagnosing further needs
|
||||
host-level tools (`tcpdump`, direct `iptables`/`sysctl` inspection) this
|
||||
project's `doas`-scoped root access during development didn't permit
|
||||
(restricted to running `slocker-lite` itself, no other commands). This
|
||||
needs re-verification — ideally on the actual veth-less target device,
|
||||
where the environment differs and this dev sandbox's own unidentified cause
|
||||
may simply not apply — before extern/outside connectivity through this
|
||||
fallback is relied on. `intern` networks (peer-to-peer only, no gateway
|
||||
involved at all) are unaffected by this gap.
|
||||
|
||||
## Explicitly out of scope for now
|
||||
|
||||
- **Rootless networking.** An earlier draft of this design considered a
|
||||
|
||||
Reference in New Issue
Block a user