A 900-Mile Detour Found and Fixed in My Own Network
Closed the book on the mlvpn-rs mystery for good — every single layer it actually controls, proven clean — which pointed the finger somewhere I hadn't looked yet: my own BGP transit provider. Traced a real routing detour worth hundreds of extra miles and fixed it live, on a real production router, while it kept carrying real traffic the whole time. Two smaller, unrelated infrastructure bugs turned up and got fixed along the way. Plus a volunteer day at church with some real decisions on the table.
Closing the mlvpn-rs mystery — for real this time
Picked the throughput hunt back up and went all the way to the bottom of it. Watched the actual low-level system calls the tunnel makes to hand data to the network — every single one succeeded, zero failures, during a real slowdown. Checked the network card's own error counters directly — zero. Checked the operating system's own traffic-shaping layer — zero drops there too. Every layer this software actually touches, on both ends of the connection, came back completely clean.
Then tried the most direct test yet: deliberately made the tunnel's own traffic pattern look as much like a completely different, unrelated VPN tool as I could — same packet size, chatter cut down to nearly nothing — to see if something in between was treating it differently because of how it looked. No change at all. That ruled out the last theory I had. The honest conclusion: this specific piece of software isn't the problem, full stop, and hasn't been the problem this whole time. Whatever's actually wrong is happening out on the wire itself, somewhere between my two locations — not something more code changes were ever going to fix.
The real culprit: a 900-mile detour on my own internet route
That's what finally pointed me at my BGP transit provider instead — the free service that lets me announce my own chunk of internet address space to the wider internet, the same way a real ISP does. Ran a side-by-side test: my server's own regular internet connection versus that same server sending traffic sourced from my own address space specifically. Regular connection: excellent, as expected. My own address space: terrible — a small fraction of the speed, and a very strange, very high first hop before the traffic even left the building.
Traced that first hop and found it: my provider's peering point for my address space was sitting in Kansas City, while my server physically lives in Salt Lake City. Every single packet using my own address space was taking a real, unnecessary 900-mile round trip before going anywhere else on the internet at all — regardless of where it was actually headed.
Got a new connection to my provider's Fremont, California location instead — a much shorter, more sensible path from Salt Lake City — and did the actual cutover live, on the real production router, without taking the connection down. Verified real improvement immediately: that same strange first hop dropped by roughly half, and a real round-trip test to a nearby server confirmed the path now correctly matches Fremont's actual geography instead of Kansas City's. Upload speed on that same path improved about five times over. Download is still capped lower than it should be — flagged that specifically to my provider's own network team with real before/after numbers, and I'm waiting to hear back before pushing further on that piece.
Two smaller, unrelated bugs found along the way
While chasing all of the above, found a real conflict on my home router: two different pieces of software were both trying to manage the same two internet connections independently, and the one that wasn't supposed to be involved at all was quietly winning, sending everything out the slower of the two connections instead of the faster one. Fixed that for good by telling the extra software to leave those two connections alone entirely, matching how everything else on that router was already set up.
Also tested a laptop's own connection back into my network that had never successfully connected before, watching it live this time — it connected cleanly, which is real, useful progress on a longer-running mystery, even though a companion device's own connection issue is still unresolved.
Volunteer day at church
Spent the day volunteering at Great Lakes Church in Kenosha — edited the video programming for both the Elementary and Preschool rooms, and set up the month's programming in Playlister for Preschool and ProPresenter for Elementary. There was real discussion about bringing me on part-time (about 8 hours a week) as actual paid staff, to help compensate for the work that already goes into keeping Sunday services running smoothly behind the scenes. We also talked through a harder decision: whether to stop offering childcare during our third service specifically, since volunteering in that age range has fallen off the last couple of years and my wife has often been covering the birth-through-5 rooms alone. My wife and I have been volunteering there since the church was still in the planning stages, back in 2008 — good, honest conversations to be having after that long.