The Case of the Ghost Tickets: A DevOps Mystery
It was a Tuesday morning when I first noticed it. 8:03 AM, and the ticket queue had a new entry that made no sense:
Subject: Still broken.
Description: Seriously. Fix it.
No user information. No asset tag. No system reference. And strangest of all—no audit trail showing how it got there.
It was as if the ticket had materialized from thin air, bypassing every input validation rule we had carefully set up over the years.
"Another ghost ticket," sighed Marcus from L1. "Sending it your way. Not something we can handle."
Not the First Apparition
After some digging, I discovered this wasn't an isolated incident.
Six similar tickets had appeared in the past three weeks—all with minimal information, all mysteriously appearing around our team's shift change between 8:00 and 8:30 AM.
Our ticketing system is supposed to be airtight. Every ticket needs:
- A requesting user
- An affected service or asset
- Proper categorization
- Detailed description
So how were these blank tickets slipping through?
The Investigation
I pulled the server logs and started tracking API calls.
Nothing unusual at first, but then I noticed something: each ghost ticket coincided with an email hitting our system from the same domain—our own.
After cornering our infrastructure team and buying them enough coffee to loosen their lips, I finally got access to the email flow logs.
The Culprit Revealed
It turned out that Frank, our well-meaning but technically overambitious department manager, had set up an Outlook rule:
IF subject = "Update?" THEN forward to: servicedesk@domain.com
Frank had been sending "Update?" emails to various project teams. When team members replied with frustrations like "Still broken" or "Seriously, fix it," those replies were automatically forwarded to our service desk email—which fed directly into our ticketing API.
The API was doing exactly what it was designed to do: create tickets from emails sent to that address. But since these were forwarded messages with mangled headers, all the user metadata was getting stripped out.
The Fix
The solution wasn't technical—it was procedural. We:
- Disabled Frank's Outlook rule (after a delicate conversation about unauthorized automation)
- Added validation to our email-to-ticket gateway to reject messages with incomplete headers
- Created a proper project status reporting workflow that didn't involve email-based hacks
Lessons Learned
Ghost tickets taught us something valuable: sometimes the most mysterious technical problems have the most human causes. Our ticketing system hadn't failed—it was working exactly as designed, processing the inputs it received however malformed they were.
In all our focus on coding, scripting, and infrastructure, we'd forgotten that humans will always find creative ways to route around process—especially when they're trying to be helpful.
Now whenever we see unexplained behavior in our systems, we don't just look at the logs. We ask: "Which well-meaning human might have created an efficiency that's breaking our workflow?"
Because in IT, the ghosts in the machine are usually just people trying to get things done.
