Community stories
programmingby JC · 515 words · 711 views

03:14

On-call week. Phone on the pillow, face up, volume all the way. One page, one graph shaped like a cliff, one guy called Wen who sounds how I sound.

The page comes at 3:14 in the morning, which is at least a nice number. I'm on call this week. On call means the phone lives on the pillow next to me, face up, volume all the way, and I sleep like a dog outside a butcher's. My partner has learned to sleep through it. I have not. The alert says: p99 latency, checkout service, above threshold for five minutes. I read it three times because at 3:14 words don't arrive in order. Laptop. Kitchen table. The screen is the only light in the apartment and it's too much. I turn the brightness down and open the dashboard and there it is, a spike shaped like a cliff. Checkout requests that normally take two hundred milliseconds are taking six seconds. Some are timing out. Somewhere in the world, someone is watching a spinner and deciding whether they actually needed the thing in their cart. First rule: don't be clever. Look at what changed. Nothing deployed since 6pm. No config changes. No infra alerts. The database looks fine. CPU fine. Memory fine. This is the worst kind, the kind where everything is green except the one graph that matters. I open the runbook, which I wrote, which says "check the usual suspects," which was very helpful of me. 3:31. I've been staring at traces for fifteen minutes and I finally see it. One downstream call, to the tax calculation service, is taking five seconds. Every single time. Not failing. Just slow. Slow enough to not trip its own alert and fast enough to ruin ours. I page the team that owns tax. I feel bad about it for exactly as long as it takes to remember they paged me twice last month. 3:40. A guy called Wen answers. He sounds like I sound. "It's the vendor," he says. "The rate lookup. They do maintenance Sunday nights and they didn't email us this time." "Can we cache it?" "We do cache it. For an hour. The cache expired at three." "Can we make it a day?" Pause. "Tax rates don't change at three in the morning."
"That's what I'm saying." He changes it. I watch the graph. It takes a few minutes for the cliff to become a hill and the hill to become the boring flat line I love more than most people. 3:52. I write the incident notes while it's fresh, because if I don't, morning me will write "vendor issue, resolved" and evening me will hate morning me. I write down the actual timeline. I write down that the tax service's alert threshold should be lower. I write down that the runbook should say something more useful than "usual suspects," and I put my own name next to that one. Then I sit in the kitchen for a bit because I'm too awake to go back to bed. The fridge hums. Outside, a bin lorry is doing its thing. I think about the person with the spinner and hope they got their thing. 4:10. Phone on the pillow. Face up. Volume all the way.

How hot is it?

Not rated yet
#on-call#incident#sre#first-person#present-tense

Themes: programming on call · incident response story · sre first person · present tense story · software engineering · on call incident · tech support story · software engineer · programming story · on call experience

Comments

Loading comments…