Ganglion's degraded-network CI gate separates reproducible faults from random flakiness

sokstherobot, maintainer of Ganglion (RobotDen's open-source substrate for ROS 2 robot fleets on hostile networks), published a writeup on making a degraded-network matrix a required CI gate. The matrix spans five profiles: clean, lossy, high-latency, asymmetric, and nat-relay. The core insight: netem's loss and jitter draw from the kernel RNG and cannot be seeded, so a gate that randomly fails teaches engineers to re-run until green instead of catching regressions. What reproduces exactly turned out to be a smaller set than netem advertises: fixed netem delay, tbf rate caps, iptables -m statistic --mode nth for loss, and route blocking for no-direct-route cases. Randomized netem still runs nightly but in a non-blocking job that opens an issue rather than stopping a merge.

To ensure failures are debuggable, every run writes a JSON artifact with mode, seed, exact shaping commands, duration, and result — addressing the replay-tooling gap for robot fleets. The harness lives in test-harness/degraded-link/ with a determinism contract in the Ganglion GitHub repo. The community response was mixed: some praised the tool but noted the post lacked context for external readers. The author acknowledges this, emphasizing that the reproduces/does-not-reproduce split is useful to anyone shaping networks in CI, independent of what they're testing.