Maintenance build and storage RCAIdentity-verification SaaS company, anonymized × red9CS-0420
577 non-clustered indexes moved to their own filegroup, and a SAN fault proved
The problem. An identity-verification SaaS company had a production estate with no complete backup or maintenance regime, no integrity checks, and no alerting. All 577 of its non-clustered indexes sat mixed into the primary filegroup with the data, which left no control over how either was distributed. On top of that, tier-1 disks on both development and production kept logging the warning that an I/O request had taken longer than 15 seconds.
What we did. Fundamentals first. We deployed a full, differential and log backup plan, index and statistics maintenance, integrity checks, 14 SQL Agent alerts, and a fail-safe operator, and set trace flags 1117, 1118, 2371 and 3226. Then we created a dedicated INDEXES filegroup and migrated all 577 non-clustered indexes into it. Finally we ran disk-latency tests, isolated the poor random 4K figures to SAN multipathing rather than to SQL Server, handed the storage team the evidence, and re-tested after their fix to confirm the disks behaved as tier-1 storage should.
Red9 · Foundations and Root Cause
Indexes relocated
577
All non-clustered indexes, off the primary filegroup.
Agent alerts deployed
14
Plus a fail-safe operator, where there had been none.
Disk warnings before the fix
I/O over 15 s
Traced to SAN multipathing, evidenced, and confirmed resolved on retest.
Backup regime
Full, diff, log
built from nothing
Non-clustered indexes
577 moved
own filegroup
Random 4K performance
Bad → tier-1 expected
after the SAN fix
Non-clustered indexes
moved to the INDEXES filegroup
An honest note on these numbers. The 577 index count, the 14 alerts, the four trace flags and the backup regime are all countable items from the engagement record. The disk result is the exception: the before-and-after throughput charts sat in images inside the client's own report, so what is on record here is the qualitative confirmation that post-fix results matched what a tier-1 disk should deliver. We would rather say that than invent a ratio.
The result. The estate ended the engagement with scheduled full, differential and log backups, integrity checks, maintenance on a schedule, and alerts with somebody on the other end. Separating the indexes gave the client control over placement and growth, and the storage problem stopped being blamed on the database once the multipathing evidence was in front of the right team.
The technical detail
Why the filegroup split matters. With indexes and data sharing the primary filegroup, you cannot place them on different storage, size their growth separately, or rebuild them without touching the data filegroup. Moving 577 non-clustered indexes into their own filegroup restores all three options without touching a single query.
What was deployed (identifiers withheld):
-- Foundations built where none existed
-- backups : full + differential + log, scheduled and verified
-- integrity : DBCC CHECKDB on schedule
-- maintenance: index and statistics jobs
-- alerting : 14 SQL Agent alerts + fail-safe operator
-- trace flags: 1117, 1118, 2371, 3226
-- filegroup : new INDEXES filegroup; all 577 nonclustered indexes migrated
-- storage : RND 4K Q32T16 tests isolated SAN multipathing; retest confirmed fix
The disk finding is the part worth copying: test the storage yourself, in a way the storage team can reproduce, before accepting that the database is at fault.