A Real Weekend Outage, a No-Signal Slate, Two OBS Bugs, and a Rebuilt Wiki
A stretch of days that started with real live-production firefighting on SRT Matrix — a startup bug that killed every source for a whole weekend without anyone noticing, plus a batch of failures that were failing completely silently — grew into a proper "no signal" slate feature, then turned into actually starting OBS SRT Toolkit: a design I'd sat on since late July became a real, working OBS plugin in a single day, including two genuine bugs found and fixed along the way. Finished by rebuilding jpps/wiki with cleaner web addresses and updating every doc to point people at the plugin.
A weekend outage nobody noticed, and three bugs that were failing silently
Found a real bug the hard way: a video source only started listening for its stream again if someone told it to by hand. Destinations already came back on their own after any restart, but nothing did the same for sources. A crash, an update, or a plain restart silently killed every source's connection, with no way back short of manually re-triggering each one. Fixed it so sources come back the same automatic way destinations already did, plus a test to keep it that way.
That fix uncovered a second, uglier one: if the server reboots before its network connection is fully up, it fails to start listening at all, and it only ever tried once. One real reboot left every source's connection permanently dead — for an entire weekend, since nothing was actively trying to connect to notice. Added a retry (30 attempts, 2 seconds apart) for that specific startup failure, while genuine problems like a port already in use still fail right away like before.
Chasing a separate live incident — a return feed over a VPN that kept retrying forever with nothing showing up in the logs, even though I could prove real data was flowing both directions — turned up a pattern repeated in three different places: three different kinds of failed connection were being silently thrown away, counted internally but never actually written to the log. Fixed all three, so the next time this happens it shows up in the logs right away instead of needing a deep network trace to even start narrowing it down.
Smaller but real: saved presets could already remember which source feeds which destination, but had no way to remember a source's own return feed. Applying a preset silently left return routing untouched no matter what the preset said. Added that support so presets now cover both directions.
Building a real "no signal" slate, live, during another incident
A field decoder receiving nothing at all — a station that's down, or a route nobody's set up yet — just shows a bare connection error, which looks identical to a real network problem even when the actual cause is as mundane as "the other end hasn't started streaming yet." Built a proper fix: any route with nothing coming in now automatically falls back, after 5 seconds, to a "technical difficulties" video loop instead of sending silence, and switches back to the real picture the instant it resumes.
A few real refinements followed once it was actually in use: return feeds now show the slate even when nobody's routed anything to them yet, since a field operator's monitor should never just be dark because of that. Clearing an already-idle return used to leave the decoder showing an error for the full 5-second grace period before the slate kicked in; now it jumps straight to the slate right away, since there's nothing to gain by waiting on a feed that can't produce anything real anyway. The slate image itself can be changed per install from the dashboard, with a live preview so you're never uploading blind, and each site gets its own thumbnail so an operator can tell at a glance whether a site is showing real content or the slate without checking the field decoder directly.
Then, cleaning up SRT Matrix's own builder
Small housekeeping before turning to OBS: SRT Matrix's Windows installer was still showing the old lowercase product name in Add/Remove Programs and its own title bar instead of the real one. Fixed that. Also sped up one of the automated build jobs: it was wiping its cached build files on every single run, which meant every build started completely from scratch even when nothing had changed. Stopped it from wiping that cache, and the job dropped from several minutes down to under 20 seconds.
Also fixed a small but real wording bug on the dashboard: the license panel was still calling capacity limits "servers," left over from before a naming pass that had already renamed everything else to "destinations." Fixed both spots.
Actually starting OBS SRT Toolkit — a design turned into a working plugin in one day
OBS SRT Toolkit had existed as an idea since late July: a plan for an OBS Studio plugin that would give OBS proper SRT support, instead of relying on an old trick of repurposing a different OBS feature to fake it. Nothing beyond the plan had actually been built. Today that changed. I built a real output that sends video out over SRT, with its own settings screen, a real input that receives an SRT stream and hands it to OBS to decode, a small on-screen control panel, and a Windows installer that finds an existing OBS install automatically. Along the way I fixed a string of real Windows-only build problems that only showed up on that platform.
With both directions built, the obvious next step was proving they actually worked against something real instead of just compiling cleanly — SRT Matrix, already running live, was right there. Sending a feed into SRT Matrix worked cleanly on the first try. Getting OBS's own program feed back out to SRT Matrix did not: SRT Matrix showed a connected link, but no video ever came through.
First bug: OBS's own log was complaining that the plugin wasn't correctly marking its output as encoded video. I patched that setting directly after creating the output, since the building-block code I was using for this never set it.
Fixing that got the connection further, but still no video came through. I dropped a temporary log line into the part of the plugin that's supposed to receive each video frame, and confirmed OBS was calling it exactly zero times despite a fully connected link. That sent me digging into OBS's own source code rather than guessing: it turns out a real OBS output plugin has to explicitly tell OBS "I'm ready, start sending me frames now," and the building-block code never did that on its own. One line fixed it. After both fixes, SRT Matrix showed a live picture of OBS's actual program feed with real, sustained video — not just a connected-but-idle link. A real plugin, built and proven working end to end, the same day it went from a plan to code.
Rebuilding jpps/wiki to work like GitLab does
Wanted a wiki page documenting all of this for real users, which meant first fixing a design
gap in jpps/wiki,
the self-hosted wiki I built
to replace GitLab's own. It only supported flat web addresses like
wiki.jpps.us/<project>, fine while everything belonged to one group, but not
going forward, since I eventually want a personal section alongside the company one. Rebuilt
it to work the way GitLab itself does, with a group name ahead of the project name, and a
shortcut so the old flat links still work and figure out the right group on their own.
Added a real database migration, a sidebar listing every page in the current project, and fixed the top of the page to show the actual project's title instead of a generic label. Tested the whole migration against a copy of the real database before ever touching the live one, then deployed it with a backup taken first. Verified live afterward with zero errors in the server's own log.
Publishing the wiki page, updating the Software page, and making the docs agree
With both the plugin and the wiki actually working, wrote a real user-facing wiki page for OBS SRT Toolkit and added it to jpps.us's Software page — a new card plus a full detail page, following the same structure as SRT Matrix's own page but without a pricing section, since this one's GPL-2.0 and free.
Then went back through every place that still described the plugin as a "planned, not yet implemented" idea and fixed it, since that was true a week ago and isn't anymore: SRT Matrix's own OBS Setup Guide got restructured to lead with "install OBS SRT Toolkit" as the recommended path, with the old manual OBS workarounds (hand-crafted SRT URLs in a Media Source, repurposing the Recording tab to fake an SRT output) kept further down as a documented fallback for anyone who can't install a plugin. The roadmap page moved the toolkit from "on the roadmap" to "shipped." And the plugin's own README had two dead links pointing at GitLab's old built-in wiki, which hasn't been the real wiki in a while — fixed those too.