Wire tap+relay fallback into join_one_network()

join_one_network() now branches on should_use_veth(network)
(network_bridge.h): the existing veth-pair path when the kernel
supports veth and the network wasn't created with --no-veth, or
create_tap_relay() (network_tap_relay.h) otherwise -- both produce the
same postcondition (a ready eth<N> in the container's namespace)
before the unchanged IP-assignment/route code runs. JoinedNetwork
gains an optional `relay` field; run_container() (commands.cpp)
collects these into active_relays (mirroring active_port_forwards) and
calls stop_tap_relay() for each after run_bwrap() returns.

Two real bugs caught by testing while verifying this end-to-end:

1. The relay child, unlike every other forked child in this project,
   never exec()s, so O_CLOEXEC on fds created before its fork (like
   daemonize.cpp's own report pipe) never takes effect -- it only
   closes fds across exec(), not across a fork that never execs. The
   relay inherited a live copy of that pipe's write end and kept it
   open forever, so `-D` combined with `-n <no-veth network>` hung
   indefinitely (the daemonize handshake's read-until-EOF never saw
   EOF). Fixed with close_inherited_fds(), scanning /proc/self/fd and
   closing everything except stdio and the relay's own report pipe, as
   the first thing relay_child_main() does.

2. A first-attempt companion fix -- adding the relay's pid to the
   session's own cgroup so --kill would reach it directly -- was tried
   and reverted: remove_session_cgroup() runs inside run_bwrap(),
   before run_container() gets to call stop_tap_relay(), so the cgroup
   was still non-empty at removal time and every such session left a
   stray cgroup directory behind (EBUSY, confirmed by testing). The
   ordinary flow already stops the relay correctly (killing bwrap lets
   run_container() reach its own cleanup), so this wasn't worth the
   added complexity.

Verified end-to-end as root via the doas rule, using a --no-veth
extern network on this dev machine specifically to exercise the
fallback: two containers joined the same network, got distinct
addresses via two tap devices + relays (no veth at all), and pinged
each other with 0% packet loss, repeatably.

Known gap, confirmed by testing, not yet root-caused: neither
container could reach the network's own gateway IP (ICMP or TCP),
despite ARP resolving correctly -- ruling out an L2/relay framing
problem. The identical bridge/subnet/host reached via veth instead
works perfectly, ruling out every environment-level explanation that
would affect both paths equally. rp_filter=0 (host-tap, bridge, and
`all` scope) was tried and confirmed not to fix it. Diagnosing further
needs host tools (tcpdump, direct iptables/sysctl inspection) this
session's doas access doesn't permit. Peer-to-peer connectivity (an
intern network's whole purpose) is solid; gateway/outside reachability
through this fallback needs re-verification, ideally on the actual
veth-less target device, before being relied on.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gv3s5jckJKzh6JkMoi2Akz
This commit is contained in:
2026-08-30 16:31:27 +00:00
parent 96efcf37e1
commit f323dab3da
5 changed files with 251 additions and 57 deletions
+113 -23
View File
@@ -271,7 +271,14 @@ Source layout (all under `src/`):
`remove_port_forward_record(container_name, bwrap_pid)` (`bwrap_pid` `remove_port_forward_record(container_name, bwrap_pid)` (`bwrap_pid`
captured from the same callback, in a variable declared in captured from the same callback, in a variable declared in
`run_container()`'s own scope) so a cleanly-exiting session's own record `run_container()`'s own scope) so a cleanly-exiting session's own record
doesn't linger for `--clean-processes` to find later. doesn't linger for `--clean-processes` to find later. The same callback
also collects each `JoinedNetwork::relay` (`network_join.h`) returned by
`join_networks()` into a `std::vector<TapRelayHandle> active_relays`
declared in `run_container()`'s own scope — the direct tap+relay
(`network_tap_relay.h`) analog of `active_port_forwards` above, since a
relay process is likewise independent host-global state (unlike a veth
pair) that needs an explicit `stop_tap_relay()` call for each, made right
alongside the `remove_port_forward()` loop after `run_bwrap()` returns.
- `self_test.{h,cpp}``run_self_tests()` implements `-t/--test`, this - `self_test.{h,cpp}``run_self_tests()` implements `-t/--test`, this
project's own built-in self-test mode (distinct from the Meson-driven project's own built-in self-test mode (distinct from the Meson-driven
fixture smoke test under `tests/`, described in "Build & test commands" fixture smoke test under `tests/`, described in "Build & test commands"
@@ -690,12 +697,29 @@ Source layout (all under `src/`):
networked services are unaffected — confirmed by testing (see below). networked services are unaffected — confirmed by testing (see below).
For each named network: looked up in `config.networks` (an unknown name is For each named network: looked up in `config.networks` (an unknown name is
a per-network error, not fatal to the others); `ensure_network_provisioned()` a per-network error, not fatal to the others); `ensure_network_provisioned()`
(`network_bridge.h`) covers post-reboot recreation; a veth pair is created (`network_bridge.h`) covers post-reboot recreation; then, per
wherever that network's bridge lives (`wrap_for_network()`, reused from `should_use_veth(network)` (`network_bridge.h` — combines the network's own
`network_bridge.h`), the bridge-side end attached and brought up, the `veth` policy flag with a kernel-capability probe, see that file's own
container-side end moved into the session's own namespace (`ip link set entry above), either a veth pair is created wherever that network's bridge
... netns <ns_pid>`) and renamed `eth<N>` (`N` = the network's position in lives (`wrap_for_network()`, reused from `network_bridge.h`), the
the `-n` list, so multiple joins each get a distinct interface). **Address bridge-side end attached and brought up, the container-side end moved into
the session's own namespace (`ip link set ... netns <ns_pid>`) and renamed
`eth<N>`, **or**, when veth isn't available or the network was created
with `--no-veth`, `create_tap_relay()` (`network_tap_relay.h`, see below)
is used instead, producing the same end state (a ready `eth<N>` in the
container's namespace) via two tap devices and a relay process rather than
a kernel veth pair. `eth<N>`'s `N` is the network's position in the `-n`
list, so multiple joins each get a distinct interface, regardless of which
strategy created it — everything downstream (IP assignment, routes, the
address returned to `port_forward.h`) is identical either way, since it
only ever operates on `eth<N>` by name. Unlike a veth pair (torn down by
the kernel automatically once the session's namespace goes away, whatever
else fails), a tap relay is an independent process with no such automatic
cleanup — if any step after `create_tap_relay()` succeeds fails later in
`join_one_network()` (address exhaustion, a failed `ip addr add`/route
command), a `fail()` helper (`.cpp`-local, only present when a relay was
actually created) calls `stop_tap_relay()` before returning `nullopt`, so
a partial failure doesn't leak the relay process. **Address
allocation, `pick_free_address()`, needed a real fix during testing, not allocation, `pick_free_address()`, needed a real fix during testing, not
just design**: an interface's actual assigned IP lives inside its own just design**: an interface's actual assigned IP lives inside its own
private per-container namespace, invisible from the bridge's own namespace private per-container namespace, invisible from the bridge's own namespace
@@ -721,23 +745,29 @@ Source layout (all under `src/`):
route command needed for same-bridge reachability regardless of kind). route command needed for same-bridge reachability regardless of kind).
Every step failure is logged specifically (which command, which network) Every step failure is logged specifically (which command, which network)
and best-effort: `join_networks()` returns one `JoinedNetwork {network, and best-effort: `join_networks()` returns one `JoinedNetwork {network,
container_ip}` per network that actually joined (in `-n` order, so shorter container_ip, relay}` per network that actually joined (in `-n` order, so
than the request list on any partial failure), never fatal to the shorter than the request list on any partial failure), never fatal to the
already-running session (network setup can only happen after `bwrap`'s own already-running session (network setup can only happen after `bwrap`'s own
namespace exists, i.e. potentially after the sandboxed command is already namespace exists, i.e. potentially after the sandboxed command is already
running) — this return value exists specifically for `port_forward.h` running) — this return value exists specifically for `port_forward.h`
(see below) to resolve a `-p` spec against which networks/IPs are actually (see below) to resolve a `-p` spec against which networks/IPs are actually
usable, not as a pass/fail signal on its own. An empty `network_names` usable, not as a pass/fail signal on its own; `relay` (`nullopt` for a
returns immediately (no namespace wait at all), so callers that always veth-joined network) is what `run_container()` (`commands.cpp`) collects
invoke this once `on_bwrap_pid_known` fires for any reason (`commands.cpp` to call `stop_tap_relay()` on after `run_bwrap()` returns, mirroring how it
also fires it for `-D/--daemonize` alone, with no `-n`) don't pay for a already collects `active_port_forwards` for `-p`'s own cleanup. An empty
wait that has nothing to do. Veth teardown `network_names` returns immediately (no namespace wait at all), so callers
that always invoke this once `on_bwrap_pid_known` fires for any reason
(`commands.cpp` also fires it for `-D/--daemonize` alone, with no `-n`)
don't pay for a wait that has nothing to do. Veth teardown
needs no explicit code: the kernel destroys an entire veth pair (both needs no explicit code: the kernel destroys an entire veth pair (both
ends, including the one still attached to the bridge) the instant *either* ends, including the one still attached to the bridge) the instant *either*
end's owning namespace is destroyed, so a session's veths disappear on end's owning namespace is destroyed, so a session's veths disappear on
their own once its namespace does — only the bridge/iptables/persistent- their own once its namespace does — only the bridge/iptables/persistent-
namespace state is deliberately left behind (`network_bridge.h`'s namespace state is deliberately left behind (`network_bridge.h`'s
reboot-reconciliation design). **Verified end-to-end on this dev machine reboot-reconciliation design); a tap relay instead needs the explicit
`stop_tap_relay()` call described above, since it's an independent process
with no namespace of its own to be torn down by. **Verified end-to-end on
this dev machine
(root, via a scoped `doas` rule)**: two concurrently-running containers on (root, via a scoped `doas` rule)**: two concurrently-running containers on
the same `intern` network got distinct addresses and could ping each the same `intern` network got distinct addresses and could ping each
other; an `intern`-joined container could not reach the outside other; an `intern`-joined container could not reach the outside
@@ -907,14 +937,74 @@ Source layout (all under `src/`):
`container_if_name` is a completely ordinary interface from the `container_if_name` is a completely ordinary interface from the
container's own point of view — `join_one_network()`'s existing IP container's own point of view — `join_one_network()`'s existing IP
assignment/route/DNAT-target-address code (unchanged, not yet wired to assignment/route/DNAT-target-address code (unchanged, not yet wired to
call this) needs no changes at all. **Not yet wired into call this) needs no changes at all. See `self_test.{h,cpp}` above for how
`join_one_network()`/`run_container()`** — see `self_test.{h,cpp}` above this file's own create/attach/teardown cycle was verified end-to-end in
for how this file's own create/attach/teardown cycle was verified isolation first, before being wired into `join_one_network()`.
end-to-end in isolation first; full traffic-plane verification (does
`-p`/inter-container connectivity actually work end-to-end through this **Real fd-leak bug caught by direct testing, not assumed**: the relay
path) is deferred to the commit that does that wiring, where two real child, unlike every other forked child elsewhere in this project, never
`--no-veth`-joined containers pinging/curling each other is a more natural `exec()`s — so `O_CLOEXEC` on fds created *before* this fork (e.g.
and simpler proof than hand-rolled raw-socket test traffic. `daemonize.cpp`'s own report-pipe write end, still open in the forking
process at this point since `report_daemon_started()` — which closes it —
hasn't run yet when `join_networks()` is called) never takes effect,
since it only closes fds *across `exec()`*, not across a fork that never
execs. Without a fix, the relay child inherited a live copy of that pipe's
write end and never closed it, so `daemonize()`'s read-until-EOF in the
*original, pre-fork* process blocked forever, even after
`report_daemon_started()` closed its own copy — a pipe only reports EOF
once *every* copy of its write end, across every process, is closed.
Confirmed directly: `-r -D -n <no-veth network> -- sleep 600` hung
indefinitely; killing the session (which reaches `stop_tap_relay()` via
`run_container()`'s own post-`run_bwrap()` cleanup, closing the leaked
copy) immediately unblocked the original process. Fixed by
`close_inherited_fds()` (`.cpp`-local): scans `/proc/self/fd` and closes
everything except stdin/stdout/stderr and the report pipe's own write
end, called as the very first thing in `relay_child_main()`. A
first-attempt companion fix — adding the relay's pid to the session's own
cgroup (`session_cgroup.h`) so `--kill` would reach it directly, since a
relay is a *sibling* of bwrap rather than a descendant and so would never
inherit cgroup membership on its own — was tried and then **reverted**:
`remove_session_cgroup()` runs *inside* `run_bwrap()`, before
`run_container()` ever gets to call `stop_tap_relay()`, so the cgroup was
still non-empty (the relay still in it) at removal time, and every
session using this fallback left a stray, never-removed cgroup directory
behind (`rmdir` failing with `EBUSY`, confirmed by testing). Since the
ordinary flow already stops the relay correctly on its own (killing bwrap
unblocks `run_bwrap()`'s own `waitpid()`, letting `run_container()`
finish its normal cleanup, `stop_tap_relay()` included) and the only gap
left by *not* doing this is a benign, self-resolving race in `--kill`'s
own "has it fully stopped" check (a genuine crash of the whole session
process, not just bwrap, is a separate, already-scoped concern — see the
crash-orphan sweep below), the added complexity wasn't worth it.
**Verified end-to-end on this dev machine (root, via the scoped `doas`
rule), using a `--no-veth` `extern` network specifically to exercise this
path**: two real containers joined the same network, each getting a
distinct address (`10.168.0.2`/`10.168.0.3`) via the tap+relay path with
no veth involved at all, and pinged each other successfully (0% packet
loss, confirmed repeatably). **Known gap, confirmed by testing, root
cause not yet identified**: neither container could reach the network's
own gateway IP (the address `provision_bridge()` assigns directly to the
bridge itself) — not with ICMP, not with a TCP `wget` — despite ARP
resolving correctly (confirmed via the container's own `ip neigh`/
`/proc/net/arp` showing a `REACHABLE` entry for the gateway's real MAC),
meaning L2 delivery through the relay is not at fault. The *exact same*
bridge, same subnet, same host, same MASQUERADE/`ip_forward` setup, but
joined via veth instead of this fallback, reaches the same gateway IP
perfectly (0% loss) — ruling out every environment-level explanation
(host firewall, `rp_filter`, this dev sandbox's own networking) that would
otherwise affect both paths identically. `net.ipv4.conf.<host-tap>
.rp_filter=0` (and broader `all`/bridge-scoped variants) was tried and
confirmed *not* to fix it. Root cause undetermined — diagnosing further
needs host-level tools (`tcpdump`, direct `iptables`/`sysctl` inspection)
this session's `doas` access doesn't permit (restricted to running
`slocker-lite` itself only). **This means gateway/outside reachability
through the tap+relay fallback is unconfirmed** — peer-to-peer
connectivity (an `intern` network's whole purpose, and half of what an
`extern` network offers) is solid; reaching the real internet through one
specifically needs re-verification, ideally on the actual veth-less
target device (a different kernel/environment where this dev sandbox's
own unidentified cause may not even apply) before being relied on.
- `session_cgroup.{h,cpp}` — gives `--kill` (`kill_session.{h,cpp}`, see - `session_cgroup.{h,cpp}` — gives `--kill` (`kill_session.{h,cpp}`, see
below) a reliable way to find every process a session ever started, however below) a reliable way to find every process a session ever started, however
deeply forked/daemonized/reparented, by putting it in a dedicated cgroup v2 deeply forked/daemonized/reparented, by putting it in a dedicated cgroup v2
+18
View File
@@ -41,6 +41,7 @@
#include "network_bridge.h" #include "network_bridge.h"
#include "network_join.h" #include "network_join.h"
#include "network_subnet.h" #include "network_subnet.h"
#include "network_tap_relay.h"
#include "oci_image.h" #include "oci_image.h"
#include "pid_file.h" #include "pid_file.h"
#include "port_forward.h" #include "port_forward.h"
@@ -692,6 +693,14 @@ int run_container(const std::filesystem::path& image_tar,
// the session's namespace goes away. // the session's namespace goes away.
std::vector<ActivePortForward> active_port_forwards; std::vector<ActivePortForward> active_port_forwards;
// Populated inside on_bwrap_pid_known (below) from join_networks()'s own
// JoinedNetwork::relay, read again after run_bwrap() returns to stop
// each one -- unlike a veth-joined network (torn down by the kernel on
// its own once the session's namespace goes away), a tap relay
// (network_tap_relay.h, used when should_use_veth() is false) is an
// independent process with no such automatic cleanup.
std::vector<TapRelayHandle> active_relays;
// Set inside on_bwrap_pid_known below, read again after run_bwrap() // Set inside on_bwrap_pid_known below, read again after run_bwrap()
// returns to remove this session's own port-forward state record. // returns to remove this session's own port-forward state record.
pid_t bwrap_pid = -1; pid_t bwrap_pid = -1;
@@ -706,6 +715,11 @@ int run_container(const std::filesystem::path& image_tar,
// have already had its chance to run by then rather than racing // have already had its chance to run by then rather than racing
// an already-returned parent. // an already-returned parent.
auto joined = join_networks(pid, network_specs, app_config); auto joined = join_networks(pid, network_specs, app_config);
for (const auto& joined_network : joined) {
if (joined_network.relay) {
active_relays.push_back(*joined_network.relay);
}
}
for (const auto& spec : parsed_port_forwards) { for (const auto& spec : parsed_port_forwards) {
if (auto active = add_port_forward(spec, joined)) { if (auto active = add_port_forward(spec, joined)) {
active_port_forwards.push_back(*active); active_port_forwards.push_back(*active);
@@ -740,6 +754,10 @@ int run_container(const std::filesystem::path& image_tar,
remove_port_forward_record(container_name, bwrap_pid); remove_port_forward_record(container_name, bwrap_pid);
} }
for (const auto& relay : active_relays) {
stop_tap_relay(relay);
}
if (!unmount_layer(mounted->top_layer_id)) { if (!unmount_layer(mounted->top_layer_id)) {
spdlog::error("failed to unmount layer {}", mounted->top_layer_id); spdlog::error("failed to unmount layer {}", mounted->top_layer_id);
} }
+68 -33
View File
@@ -36,6 +36,7 @@
#include "network_bridge.h" #include "network_bridge.h"
#include "network_subnet.h" #include "network_subnet.h"
#include "network_tap_relay.h"
#include "pid_file.h" #include "pid_file.h"
#include "process.h" #include "process.h"
#include "sandbox_process.h" #include "sandbox_process.h"
@@ -145,65 +146,99 @@ std::optional<std::string> pick_free_address(const NetworkEntry& network, bool i
return std::nullopt; return std::nullopt;
} }
std::optional<std::string> join_one_network(pid_t ns_pid, const NetworkEntry& network, int if_index) { std::optional<JoinedNetwork> join_one_network(pid_t ns_pid, const NetworkEntry& network, int if_index) {
if (!ensure_network_provisioned(network)) { if (!ensure_network_provisioned(network)) {
return std::nullopt; return std::nullopt;
} }
std::string bridge = bridge_name(network.name); std::string bridge = bridge_name(network.name);
// Distinct prefixes ("vh"/"vp") so the two ends' names can't collide with // Distinct prefixes ("vh"/"vp"/"th") so names can't collide with each
// each other while both still live in the same (bridge's) namespace, // other while a veth pair's two ends still both live in the same
// right after creation and before the peer end is moved away. // (bridge's) namespace, right after creation and before the peer end is
// moved away.
std::string suffix = fmt::format("{:08x}", fnv1a(fmt::format("{}-{}", network.name, ns_pid))); std::string suffix = fmt::format("{:08x}", fnv1a(fmt::format("{}-{}", network.name, ns_pid)));
std::string host_veth = "vh" + suffix;
std::string peer_veth = "vp" + suffix;
if (!run(wrap_for_network(network, {"ip", "link", "add", host_veth, "type", "veth", "peer", "name", peer_veth}),
"create veth pair", network.name)) {
return std::nullopt;
}
if (!run(wrap_for_network(network, {"ip", "link", "set", host_veth, "master", bridge}), "attach veth to bridge",
network.name) ||
!run(wrap_for_network(network, {"ip", "link", "set", host_veth, "up"}), "bring host veth up",
network.name)) {
return std::nullopt;
}
if (!run(wrap_for_network(network, {"ip", "link", "set", peer_veth, "netns", fmt::to_string(ns_pid)}),
"move veth into the container's namespace", network.name)) {
return std::nullopt;
}
std::string container_if = fmt::format("eth{}", if_index); std::string container_if = fmt::format("eth{}", if_index);
if (!run(wrap_in_container(ns_pid, {"ip", "link", "set", peer_veth, "name", container_if}),
"rename the container's interface", network.name)) { // should_use_veth() (network_bridge.h) combines the network's own `veth`
return std::nullopt; // policy flag with a kernel-capability probe -- see
// docs/networking-design.md's tap+relay addendum for why a kernel might
// lack veth support at all (the real target device supports tun/tap but
// not veth), and network_tap_relay.h for what the fallback below does.
// Both branches leave `container_if` a completely ordinary interface
// from the container's own point of view -- everything after this if/else
// (IP assignment, routes, the address returned to port_forward.h) is
// unchanged either way.
std::optional<TapRelayHandle> relay;
if (should_use_veth(network)) {
std::string host_veth = "vh" + suffix;
std::string peer_veth = "vp" + suffix;
if (!run(wrap_for_network(network,
{"ip", "link", "add", host_veth, "type", "veth", "peer", "name", peer_veth}),
"create veth pair", network.name)) {
return std::nullopt;
}
if (!run(wrap_for_network(network, {"ip", "link", "set", host_veth, "master", bridge}),
"attach veth to bridge", network.name) ||
!run(wrap_for_network(network, {"ip", "link", "set", host_veth, "up"}), "bring host veth up",
network.name)) {
return std::nullopt;
}
if (!run(wrap_for_network(network, {"ip", "link", "set", peer_veth, "netns", fmt::to_string(ns_pid)}),
"move veth into the container's namespace", network.name)) {
return std::nullopt;
}
if (!run(wrap_in_container(ns_pid, {"ip", "link", "set", peer_veth, "name", container_if}),
"rename the container's interface", network.name)) {
return std::nullopt;
}
} else {
std::string host_tap = "th" + suffix;
auto created = create_tap_relay(network, bridge, host_tap, ns_pid, container_if);
if (!created) {
spdlog::error("failed to create tap relay for network '{}'", network.name);
return std::nullopt;
}
relay = created;
} }
// Unlike a veth pair (torn down by the kernel on its own once the
// session's namespace goes away, whatever else fails below), a tap
// relay is an independent process with no such automatic cleanup -- if
// it was already created, any later failure in this function must stop
// it explicitly, or it leaks forever.
auto fail = [&]() -> std::optional<JoinedNetwork> {
if (relay) {
stop_tap_relay(*relay);
}
return std::nullopt;
};
auto container_ip = pick_free_address(network, false); auto container_ip = pick_free_address(network, false);
if (!container_ip) { if (!container_ip) {
spdlog::error("no free IPv4 address available on network '{}'", network.name); spdlog::error("no free IPv4 address available on network '{}'", network.name);
return std::nullopt; return fail();
} }
if (!run(wrap_in_container(ns_pid, {"ip", "addr", "add", *container_ip, "dev", container_if}), if (!run(wrap_in_container(ns_pid, {"ip", "addr", "add", *container_ip, "dev", container_if}),
"assign the container's IPv4 address", network.name)) { "assign the container's IPv4 address", network.name)) {
return std::nullopt; return fail();
} }
if (network.ipv6) { if (network.ipv6) {
auto container_ip6 = pick_free_address(network, true); auto container_ip6 = pick_free_address(network, true);
if (!container_ip6) { if (!container_ip6) {
spdlog::error("no free IPv6 address available on network '{}'", network.name); spdlog::error("no free IPv6 address available on network '{}'", network.name);
return std::nullopt; return fail();
} }
if (!run(wrap_in_container(ns_pid, {"ip", "-6", "addr", "add", *container_ip6, "dev", container_if}), if (!run(wrap_in_container(ns_pid, {"ip", "-6", "addr", "add", *container_ip6, "dev", container_if}),
"assign the container's IPv6 address", network.name)) { "assign the container's IPv6 address", network.name)) {
return std::nullopt; return fail();
} }
} }
if (!run(wrap_in_container(ns_pid, {"ip", "link", "set", container_if, "up"}), "bring the container's interface up", if (!run(wrap_in_container(ns_pid, {"ip", "link", "set", container_if, "up"}), "bring the container's interface up",
network.name)) { network.name)) {
return std::nullopt; return fail();
} }
if (network.kind == NetworkKind::extern_) { if (network.kind == NetworkKind::extern_) {
@@ -231,7 +266,7 @@ std::optional<std::string> join_one_network(pid_t ns_pid, const NetworkEntry& ne
// Bare IP, no prefix -- what port_forward.h needs as a DNAT target; // Bare IP, no prefix -- what port_forward.h needs as a DNAT target;
// pick_free_address() returns the CIDR form since that's what `ip addr // pick_free_address() returns the CIDR form since that's what `ip addr
// add` needs above. // add` needs above.
return container_ip->substr(0, container_ip->find('/')); return JoinedNetwork{network, container_ip->substr(0, container_ip->find('/')), relay};
} }
} // namespace } // namespace
@@ -258,8 +293,8 @@ std::vector<JoinedNetwork> join_networks(pid_t bwrap_outer_pid, const std::vecto
++if_index; ++if_index;
continue; continue;
} }
if (auto container_ip = join_one_network(*ns_pid, *it, if_index)) { if (auto joined_network = join_one_network(*ns_pid, *it, if_index)) {
joined.push_back({*it, *container_ip}); joined.push_back(std::move(*joined_network));
} else { } else {
spdlog::error("failed to join network '{}'", name); spdlog::error("failed to join network '{}'", name);
} }
+7 -1
View File
@@ -16,20 +16,26 @@
#pragma once #pragma once
#include <optional>
#include <string> #include <string>
#include <vector> #include <vector>
#include <sys/types.h> #include <sys/types.h>
#include "config_file.h" #include "config_file.h"
#include "network_tap_relay.h"
// One network a session successfully joined, and the IPv4 address it was // One network a session successfully joined, and the IPv4 address it was
// assigned there -- what port_forward.h needs to resolve a -p spec (by name, // assigned there -- what port_forward.h needs to resolve a -p spec (by name,
// or by "the container's sole extern network" when none is given) to an // or by "the container's sole extern network" when none is given) to an
// actual DNAT target. // actual DNAT target. `relay` is set only when the join used the tap+relay
// fallback (network_tap_relay.h) instead of a veth pair -- nullopt for a
// veth-joined network, which needs no explicit teardown (the kernel tears
// the veth pair down on its own once the session's namespace goes away).
struct JoinedNetwork { struct JoinedNetwork {
NetworkEntry network; NetworkEntry network;
std::string container_ip; std::string container_ip;
std::optional<TapRelayHandle> relay;
}; };
// Joins a just-started -r/--run session (identified by bwrap_outer_pid, the // Joins a just-started -r/--run session (identified by bwrap_outer_pid, the
+45
View File
@@ -16,6 +16,7 @@
#include "network_tap_relay.h" #include "network_tap_relay.h"
#include <dirent.h>
#include <fcntl.h> #include <fcntl.h>
#include <linux/if_tun.h> #include <linux/if_tun.h>
#include <net/if.h> #include <net/if.h>
@@ -27,6 +28,7 @@
#include <unistd.h> #include <unistd.h>
#include <cerrno> #include <cerrno>
#include <cstdlib>
#include <cstring> #include <cstring>
#include <vector> #include <vector>
@@ -39,6 +41,47 @@
namespace { namespace {
// Closes every open fd except stdin/stdout/stderr and `keep_fd` (the report
// pipe's write end). Necessary specifically because this relay child, unlike
// every other forked child elsewhere in this project, never exec()s -- it
// keeps running as a plain continuation of this same process image, so
// O_CLOEXEC (already used throughout this codebase for exactly this kind of
// fd hygiene -- e.g. pid_file.h's SessionLock, daemonize.cpp's own report
// pipe) never actually takes effect here: it only closes fds *across
// exec()*, not across a fork() that never execs. **Real bug caught by
// direct testing, not assumed**: without this, the relay child inherited a
// live copy of daemonize.cpp's own report-pipe write end (still open in the
// forking process at this point, since report_daemon_started() -- which
// closes it -- hasn't run yet when join_networks() is called) and never
// closed it, so daemonize()'s read-until-EOF in the *original, pre-fork*
// process blocked forever even after report_daemon_started() closed its own
// copy -- a pipe only reports EOF once *every* copy of its write end, across
// every process, is closed. Confirmed directly: `-r -D -n <no-veth network>`
// hung indefinitely; killing the session (which incidentally reaches
// stop_tap_relay() via run_container()'s own post-run_bwrap() cleanup)
// closed the leaked copy and immediately unblocked the original process.
void close_inherited_fds(int keep_fd) {
DIR* dir = opendir("/proc/self/fd");
if (!dir) {
return;
}
struct dirent* entry;
while ((entry = readdir(dir)) != nullptr) {
char* end = nullptr;
long fd = strtol(entry->d_name, &end, 10);
if (!end || *end != '\0') {
continue; // "." / ".."
}
int fd_int = static_cast<int>(fd);
if (fd_int == STDIN_FILENO || fd_int == STDOUT_FILENO || fd_int == STDERR_FILENO || fd_int == keep_fd ||
fd_int == dirfd(dir)) {
continue;
}
close(fd_int);
}
closedir(dir);
}
// Opens /dev/net/tun and creates a tap device named `name` in whatever // Opens /dev/net/tun and creates a tap device named `name` in whatever
// network namespace this process is currently in -- IFF_NO_PI so both ends // network namespace this process is currently in -- IFF_NO_PI so both ends
// of a relay agree on raw-frame framing with no extra header, IFF_TAP (not // of a relay agree on raw-frame framing with no extra header, IFF_TAP (not
@@ -105,6 +148,8 @@ void report_line(int fd, const std::string& line) {
[[noreturn]] void relay_child_main(int report_fd, const NetworkEntry& network, const std::string& bridge, [[noreturn]] void relay_child_main(int report_fd, const NetworkEntry& network, const std::string& bridge,
const std::string& host_tap_name, pid_t container_ns_pid, const std::string& host_tap_name, pid_t container_ns_pid,
const std::string& container_if_name) { const std::string& container_if_name) {
close_inherited_fds(report_fd);
if (network.kind == NetworkKind::intern) { if (network.kind == NetworkKind::intern) {
if (!enter_namespace(persistent_netns_path(network.name).string())) { if (!enter_namespace(persistent_netns_path(network.name).string())) {
report_line(report_fd, "ERROR failed to enter network's persistent namespace\n"); report_line(report_fd, "ERROR failed to enter network's persistent namespace\n");