The importance of verifying your backup

This isn’t going to be one of those posts about how “I didn’t check the backup and I lost everything”. It’s a cautionary tale of the perils of green check marks, the woes of complexity abstraction and the importance of making a list and checking it twice, with a Happy Ending.

Every month I do what I call the Monthly Maintenance for my client, checking event logs, disk space, AV updates, UPS functionality and backups. This particular client runs a Dell SBS 2008 box which is now just over two years old and 10 client PCs. We use the built in backup program in SBS 2008 with two external HDDs for backup (swapped daily). As part of the backup check I restore a few random files of the backup to an alternate location and open them up to verify that the backup is indeed working. Same result this time and since the detailed SBS email status report I receive every morning had pretty green ticks next to Backup I felt quite secure in the knowledge that I had reliable backups.

I found however in the Application event log some worrying database page corruption errors from Exchange 2007, further research found some database logfile problems as well. Checking in the folder where the email database is located I found just over 24 000 logfiles (equal to about 24 GB), these should be truncated by the nightly backup and although they’re heavy email users there shouldn’t be that many logfiles. Further research in the event log showed me that the Exchange part of the backup had stopped about three weeks ago.

I posted in Microsoft’s partner forum, created a new database on another volume on the server, moved all mailboxes to the new database. One mailbox refused to move with the “tolerate corrupt messages” set to 0, but did move OK when set to 2. OK, so out of an “archive” type mailbox, storing about 40 000 messages, one or two messages had been lost. Not too bad. Contacted Dell and upgraded the firmware on the RAID controller, driver was already up to date. Moved the mailboxes back to a new database on the original drive and checked to see how the backup would go, again file backup worked but not the Exchange part. Further research in the event log showed that the Exchange “plug in” part wasn’t working correctly and Microsoft had a script and step by step instruction for how to remedy:

1. Remove Windows Server Backup feature, including command line from Server Manager
2. Reboot the server
3. Run the script on the box.
4. Add Windows Server Backup Feature, including command line from Server Manager.

The script:

Note: The script is a Powershell script and requires to be run from an Admin Powershell Console.

Note: Copy/Paste the script into a notepad file and save the file as Jadoo.ps1

Note: If you are running the script from the working directory, it should be run as.\Jadoo.ps1. For example if the script is on the desktop, proper way to run is:

PS C:\Users\UserName\Desktop> .\Jadoo.Ps1

Here is the script:

Script Starts

$WsbBinPath=”c:\program files\Windows Small Business Server\bin\wsbexchange.exe”

if ((get-service wsbexchange).Status -eq “Running”)
stop-service wsbexchange
sc.exe delete wsbexchange

reg add “HKCR\CLSID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /t REG_SZ /d “CExchangeHelper Class” /f
reg add “HKCR\CLSID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /v AppId /t REG_SZ /d “{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /f
reg add “HKCR\CLSID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}\LocalServer32” /t REG_SZ /d “$WsbBinPath” /f
reg add “HKCR\APPID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /t REG_SZ /d “CExchangeHelper Class” /f
reg add “HKCR\APPID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /v LocalService /t REG_SZ /d “wsbexchange” /f
reg add “HKCR\APPID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /v LaunchPermission /t REG_BINARY /d “010004806000000070000000000000001400000002004c0003000000000014001f000000010100000000000512000000000018001f000000010200000000000520000000200200000000180003000000010200000000000520000000270200000102000000000005200000002002000001020000000000052000000020020000” /f
reg add “HKCR\APPID\wsbexchange.exe” /v AppId /t REG_SZ /d “{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /f
reg add “HKLM\Software\Microsoft\windows nt\currentversion\WindowsServerBackup\Application Support\{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}” /v “Application Identifier” /t REG_SZ /d Exchange /f
reg add “HKLM\Software\Microsoft\windows nt\currentversion\WindowsServerBackup\Application Support\{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}” /v CLSID /t REG_SZ /d “{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /f
reg add “HKLM\Software\Microsoft\windows nt\currentversion\WSBAppExchangeHelper” /v AutoMarkDbRecoverable /t REG_DWORD /d 1 /f
reg add “HKLM\Software\Microsoft\windows nt\currentversion\WSBAppExchangeHelper” /v AutoMountOnPITRecovery /t REG_DWORD /d 1 /f
sc.exe create wsbexchange binpath= $WsbBinPath type= own start= demand error= ignore obj= LocalSystem DisplayName= “Microsoft Exchange Server Extension for Windows Server Backup”
sc.exe description wsbexchange “Enables Windows Server Backup users to back up and recover application data for Microsoft Exchange Server.”
Script Ends

I followed these instructions and still the Exchange backup didn’t complete properly. The number of logfiles had now grown to close to 100 000, over 100 GB of disk space. The other way (apart from running a backup) of truncating Exchange logfiles is stopping the database and ensure that it’s in a clean shutdown state using Eseutil /mh which I did. All the databases where cleanly shutting down so I moved all the logfiles to another volume, starting each database then creates a fresh set of logs and backup now ran fine.

The moral of the story, don’t trust green tick marks, always check for yourself and verify that not only the file part of a backup is working, check Exchange (and SharePoint / SQL) as well.

For the full story with all the gory details check the MS partner forums under Exchange / Messaging for “SBS 2008/Exchange 2007 unhappy database – please recommend next steps”.

Thanks for reading – and remember to verify your backups!

Office 365 rebuttal published

Some you of might subscribe (it’s free) to the CRN Magazine here in OZ, in the August issue Craig Deveson of Devnet had a one page article in the “threads” section on the benefits of Google Apps/Docs over Office 365.

Since there were some points in that article that weren’t strictly true I offered to write a counterpoint, outlining why Office 365 is the better choice. My article was just published in CRN Magazine, it’s on page 27 but it’s not available online at this time so I can’t link to it. If you go to you can subscribe to the magazine there.

Enjoy and thanks for reading.