Managed services / blocking incidentMedical device manufacturer, anonymized × red9CS-0590
Monthly failed jobs went from 10 the month before to 1 on a production instance through a bad February
The problem. A medical device manufacturer's production instance had a rough February. Alert volume climbed 83.5% to 167 for the month, one of them a P1 blocking-query incident, deadlocks stayed elevated at 1,133, SQL errors rose 52% to 16,575 with 7,276 of those a new data-truncation fault in one table, and storage latency began drifting from 15-20 ms toward 25-30 ms. On top of that, VPN and directory permission changes were locking our own DBAs out of the server they were supposed to be watching.
What we did. Access came first: we worked through several VPN and zero-trust configurations with the client's MSP until administrative access was stable again. Then we handled the 167 alerts, contained the P1 blocking incident and 25 P2s, cleared the failed-job backlog, opened a storage-latency investigation on the rising numbers, and took the truncation errors to the application team as a code fix rather than a database patch.
Red9 · Managed Services Month
Failed jobs, in the month
10 → 1
Cleared while alert volume was at its highest.
Alerts handled
167
Including one P1 blocking incident and 25 P2s.
Transaction volume
736.3M, steady
Throughput held flat through the whole stability push.
Failed jobs
10 → 1
90% fewer
P1 blocking incident
Contained
same day
Storage latency
15-20 ms → 25-30 ms
flagged, investigating
Where the numbers come from. All of it is out of the client's monthly managed-services report: job history for the 10 to 1, the alert log for the 167 and its P1 and P2 split, and the instance counters for the 736.3 million transactions and the SQL usage figure of 10.29%. The latency drift is reported as found rather than as fixed, because the storage investigation was still open at month end.
The result. The instance ended the month with one failed job instead of 10, the blocking incident closed, and the truncation errors owned by the team that could actually fix them. Throughput held flat while all of this was cleaned up: 736.3 million transactions went through the instance.
The technical detail
What the month looked like. Of the 16,575 SQL errors that month, 7,276 were data-truncation failures in one table, and that is an application contract problem for the app team to own. The P1 was a single long-running statement holding locks across a busy window; killing it restored service, and the query itself went onto the tuning list.
How the month was worked (findings summarized, identifiers withheld):
-- Managed services month, production instance
-- P1 blocking : one long-running statement holding locks; contained, then tuned
-- P2 x25 : IO and read-latency alerts, deadlock events, failed jobs
-- errors : 16,575 total; 7,276 = truncation in one table -> app team
-- jobs : 10 failing -> 1
-- storage : latency 15-20ms -> 25-30ms, investigation opened
Each item carried an owner and a next step, so the client could see which faults were ours to fix and which belonged to the application or the storage vendor.