Support your Local [Sherrif] Sysadmin

The plot of the 1969 Western calls for James Garner to produce exasperated and amused facial expressions as his character attempts to emigrate to Australia but is frustrated by a lack of funds. The character he portrays, Jason McCollough, is stuck in a nineteenth century frontier town put on the map due to an accidental discovery of gold. Because the cost of living in town inflates hourly and the pay is good, McCollough, a brilliant gunslinger, reluctantly accepts the job as the local sheriff.  

This also happens to be a solid description of most IT and sysadmins I’ve worked with over the years. Quick on the draw when it comes to technical challenges, sysadmins are a jack of all trades who can resolve issues with both ingenuity and a refreshing lack of perfectionism when stabilizing a situation. Also: many sysadmins are not in the job they envisioned when they graduated, must work with the equivalent of the town ‘character’, and have gleeful management who are thrilled someone else deals with the local mess of bad horses, bad men and bad decisions.

One challenge for a sysadmin is that like the sheriff, when things are going smoothly, neither of them is front of mind for day-to-day operations. You only notice them when unwelcome changes occur, such as a core software platform provider gets bought out by an aggressive organization operating like a PE who then squeezes the clientele for more juice. As a sysadmin you have no good choices: costs might suddenly triple, but you built your disaster recovery plan on this technology. You’re “encouraged” to move to new products if you know what's good for you – can you really afford to walk away from the new owner who decided to weaponize their newly held asset against you, their customer?  

Even the low-grade annoyances can flare up out of the blue. Accepting a service providers’ half-hearted developer experiences, inscrutable services catalogs, or byzantine documentation because they offer stability in reliability, identity and service boundaries seems like reasonable tradeoff. Until the service provider starts surprising you with issues in authorization checks and access controls across your VMs, then responds to legitimate customer sysadmin concerns with a statement: “No one has really found that exploit in a real world situation.”

Notice that even mature, decades-old technologies from suppliers who had been stable, if not accommodating, business partners can be a new wrinkle for sysadmins. A new singularity in the form of AI springs new leaks in what had been a relatively steady-state environment. Today, sysadmins have a trifecta of fleet-wide technology changes that are necessary to get the most from AI, bad actors who use AI to launch attacks at a scale human teams can’t match, and a core technology that no one has yet mastered the art of setting or meeting budget projections for. A recent KPMG report states that nearly 1/3 of corporate leaders are finding it difficult to understand the operating costs, much less control them when it comes to AI implemented at enterprise scale. It’s not just financial operations either: there are new potential legal exposures related to copyright ownership, new supplier contract terms and/or unintended regulatory missteps.

For the AI-plagued sysadmin, here are a few shortlist areas for your triage efforts.  

First, protect the areas where the entire company could be put at risk:

• For product development, consider a private AI implementation or open weights model, small language models or industry-specific models you fine tune. This can help ensure your use of AI in product development isn’t a two-way channel that trains your supplier’s AI on how you develop your products as the basis for future competitors. This is especially important if your product happens to be software and is currently protected through copyright.  
• Ensure that any new tools or technology suppliers can meet your insurability requirements through contract negotiations or vendor selection processes
• Employ AI for threat detection including fraud detection while being very deliberate and prescriptive about where AI is allowed to take action to remediate.

Next, use “multi-cloud” principles to ensure company operations can run independent of specific models to avoid vendor API lock-in and build in finOps modeling to anticipate budget “tops and tails."

No CFO has ever been excited to hear sysadmins tell them “I have no idea what budget I need, or whether I can hit the budget within 5% of my estimates”. Like cloud services as they matured, AI tooling prices have experienced wild swings in the last three years, and model retirement has already caused headaches for what were once stable enterprise workflows.  

Finally, look for real-time operational governance capabilities in your AI solutions.

The regulatory landscape for AI is broad, varies greatly between states and nations, and can carry significant penalties for missteps. In the old west, the town charter and the town council were necessary but not sufficient to tackle the opportunities and challenges that came from a sudden influx of gold prospectors. Similarly, AI governance committees are necessary, but not sufficient in a playing field that changes at the speed of compute.

In the end, the role of the sysadmin hasn’t changed nearly as much as the environment around them. Like the reluctant sheriff, they step in not because the job is easy or glamorous, but because someone has to keep order when complexity and chaos collide. As AI reshapes the frontier yet again, the tools may evolve and the stakes may rise, but the mission remains the same: protect the town, steady the systems, and make sure everyone gets through the day without noticing just how close things came to going off the rails. So, this Sysadmin Day, remember—the quiet stability you rely on is no accident; it’s the work of someone who’s always on watch, even when you don’t see them.