A Real Loss Rate Finally Caught With My Own Packet-Capture Tool
Four more mlvpn-rs releases in one night, a fix that honestly didn't work, a rollback that turned out to be a false alarm caught just in time, and — after ruling out nearly everything else I could think of — finally proving with hard numbers that real packets are disappearing between my two routers in a way none of the tunnel's own instrumentation had ever caught. Plus a first day back at the hardware store.
Two more real fixes, then one that honestly didn't help
Picked back up right where the last post left off: the tunnel's shared internal lock was still costing real throughput under load. Split it so that reading the connection's security keys (which happens on every single packet sent) no longer has to wait in line behind the much rarer moments that actually change those keys — a design borrowed directly from looking at how WireGuard itself handles the same problem. Followed it with a smaller fix to a background housekeeping task that was grabbing the same lock a second time, unnecessarily, on every packet that arrived out of order.
Then tried something that seemed promising on paper: making the tunnel's own "wait briefly for out-of-order packets" window automatically shrink once it's confident only one connection is actually being used, since there's nothing left to reorder. Shipped it, deployed it to my real home-to-cloud routers, and it genuinely did shrink the window the way it was supposed to. It just didn't move the actual problem at all — same slow speeds, if anything slightly more retransmitting than before. Rather than call that a win because the code worked as designed, said so plainly and moved on to the next idea, which is a better use of an evening than pretending a fix helped when the numbers say otherwise.
Borrowing WireGuard's whole approach — and a real bug caught before it shipped
The next idea was bigger: WireGuard doesn't bother holding packets back to fix ordering at all on a single connection — it just delivers them the instant they arrive and lets the receiving computer's own networking stack sort out anything that came in out of order, which normal networking already does just fine. mlvpn-rs's own "wait and reorder" logic only exists because it can spread traffic across multiple connections at once, where genuine reordering is possible — once it's settled on using just one, that whole mechanism is pure overhead. Built exactly that: skip the reorder step entirely once the tunnel has settled on a single active connection.
Sent it out for a second opinion before shipping, same as always for anything touching how packets actually get delivered — and it was the right call. The reviewer traced through the reasoning I'd written for why one particular safety check could be skipped, and found the reasoning itself was wrong: it assumed a certain cleanup step would always run at least once per connection, when in fact it might never run again once this new fast path kicks in. Left alone, that could have caused the tunnel to get permanently stuck favoring the wrong connection after a routine security key rotation, with no way back short of a restart. Fixed it, re-reviewed clean, shipped it as its own release.
A rollback that turned out to be a false alarm — caught in time
Ran the usual real-world speed test right after deploying that fix and got a genuinely bad number — worse than before, by every measure. Rolled straight back to the previous version to be safe. Then, before concluding anything, ran the exact same test again on the version I'd just rolled back to, under the same conditions, a minute later — and it was just as bad. The new version was never the problem; whatever's actually wrong with the connection tonight had simply gotten worse in the few minutes between tests, and the first comparison was against an old, stale number from earlier in the night rather than a fair one taken right then. Put the new version back. Small thing, but a good reminder: never trust a "this got worse" conclusion without a fresh, same-moment comparison against whatever you're rolling back to.
Proving it's the software, not the internet connection
With four releases now hunting the same mystery and none of them moving the needle, it was time to stop guessing and start eliminating things methodically. Logged into the cable modem directly and pulled its own error history — completely clean, no sign of a bad physical connection. Renewed the internet lease on the router just in case — no change, same address, same result.
Then the real turning point: I already had a plain WireGuard tunnel configured between the same two routers, over the exact same internet connection, set up previously for exactly this kind of comparison. Brought it up and ran the identical speed test through it, back to back with mlvpn-rs, on the same connection, at the same moment. WireGuard: fast, clean, barely any retransmitting. mlvpn-rs, seconds later, same connection: slow, and heavily retransmitting. That's about as close to a controlled experiment as you get outside a lab — same wire, same two computers, same instant, wildly different result. Whatever's wrong, it's mine to fix, not the cable company's.
Ruling things out one at a time, then building a tool to actually prove it
From there it became a real process-of-elimination hunt. A theory that the tunnel's own periodic bandwidth-measurement feature was flooding the connection and starving real traffic — checked directly by turning it off entirely and testing again; still slow, so not that (though it did turn up a real, separate bug in how long that measurement is allowed to run, worth fixing on its own later). A theory that mlvpn-rs sends data in uneven bursts while WireGuard sends smoothly — checked by capturing both tunnels' actual traffic and measuring the gaps between packets directly; turned out WireGuard is actually the burstier of the two, so that wasn't it either. A theory that the computer's own network buffer was overflowing invisibly at the operating-system level — checked against the exact kernel counter that would catch that; it hadn't moved a single count during an active bad run.
Every one of those came back clean, which was frustrating but also narrowing things down fast. So I wrote a small tool from scratch: capture the tunnel's real traffic simultaneously on both ends at once — the sending router and the receiving one — and match up every single packet by its own internal ID number to see whether packets that genuinely left one side ever actually showed up at the other. First pass looked like it had caught something alarming — the same packet appearing to be sent twice — but a closer look at the actual bytes showed that was a mistake in my own tool, not a real bug: it hadn't accounted for the fact that both directions of traffic share the same wire and the same ID numbering scheme, so it was occasionally matching up two completely unrelated packets, one from each direction, that just happened to land on the same number by coincidence. Fixed the tool, verified the fix made the false match disappear entirely, and re-ran it for real.
The corrected result: of everything the sending router genuinely transmitted, about 23 percent of it never showed up on the receiving end at all — real, measurable loss, on a connection the tunnel's own built-in health checks report as essentially perfect. That's the real number that's been hiding behind every one of tonight's failed fixes. As of writing this, mid-investigation, the next test in progress is changing which network port the tunnel uses, to check whether something along the path is specifically picking out this one connection rather than treating all traffic the same way. Don't have an answer yet — but for the first time tonight, there's a real, solid number to chase instead of a guess.
Personal note
First day back at the Ace Hardware in Zion landed on a delivery day, so most of the shift was spent getting new stock out onto the shelves, with a couple of customers helped along the way. Good, simple contrast to the rest of tonight.