The importance of verifying your backup

This isn’t going to be one of those posts about how “I didn’t check the backup and I lost everything”. It’s a cautionary tale of the perils of green check marks, the woes of complexity abstraction and the importance of making a list and checking it twice, with a Happy Ending.

Every month I do what I call the Monthly Maintenance for my client, checking event logs, disk space, AV updates, UPS functionality and backups. This particular client runs a Dell SBS 2008 box which is now just over two years old and 10 client PCs. We use the built in backup program in SBS 2008 with two external HDDs for backup (swapped daily). As part of the backup check I restore a few random files of the backup to an alternate location and open them up to verify that the backup is indeed working. Same result this time and since the detailed SBS email status report I receive every morning had pretty green ticks next to Backup I felt quite secure in the knowledge that I had reliable backups.

I found however in the Application event log some worrying database page corruption errors from Exchange 2007, further research found some database logfile problems as well. Checking in the folder where the email database is located I found just over 24 000 logfiles (equal to about 24 GB), these should be truncated by the nightly backup and although they’re heavy email users there shouldn’t be that many logfiles. Further research in the event log showed me that the Exchange part of the backup had stopped about three weeks ago.

I posted in Microsoft’s partner forum, created a new database on another volume on the server, moved all mailboxes to the new database. One mailbox refused to move with the “tolerate corrupt messages” set to 0, but did move OK when set to 2. OK, so out of an “archive” type mailbox, storing about 40 000 messages, one or two messages had been lost. Not too bad. Contacted Dell and upgraded the firmware on the RAID controller, driver was already up to date. Moved the mailboxes back to a new database on the original drive and checked to see how the backup would go, again file backup worked but not the Exchange part. Further research in the event log showed that the Exchange “plug in” part wasn’t working correctly and Microsoft had a script and step by step instruction for how to remedy:

1. Remove Windows Server Backup feature, including command line from Server Manager
2. Reboot the server
3. Run the script on the box.
4. Add Windows Server Backup Feature, including command line from Server Manager.

The script:

Note: The script is a Powershell script and requires to be run from an Admin Powershell Console.

Note: Copy/Paste the script into a notepad file and save the file as Jadoo.ps1

Note: If you are running the script from the working directory, it should be run as.\Jadoo.ps1. For example if the script is on the desktop, proper way to run is:

PS C:\Users\UserName\Desktop> .\Jadoo.Ps1

Here is the script:

Script Starts
——————

$WsbBinPath=”c:\program files\Windows Small Business Server\bin\wsbexchange.exe”

if ((get-service wsbexchange).Status -eq “Running”)
{
stop-service wsbexchange
}
sc.exe delete wsbexchange

reg add “HKCR\CLSID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /t REG_SZ /d “CExchangeHelper Class” /f
reg add “HKCR\CLSID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /v AppId /t REG_SZ /d “{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /f
reg add “HKCR\CLSID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}\LocalServer32” /t REG_SZ /d “$WsbBinPath” /f
reg add “HKCR\APPID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /t REG_SZ /d “CExchangeHelper Class” /f
reg add “HKCR\APPID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /v LocalService /t REG_SZ /d “wsbexchange” /f
reg add “HKCR\APPID\{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /v LaunchPermission /t REG_BINARY /d “010004806000000070000000000000001400000002004c0003000000000014001f000000010100000000000512000000000018001f000000010200000000000520000000200200000000180003000000010200000000000520000000270200000102000000000005200000002002000001020000000000052000000020020000” /f
reg add “HKCR\APPID\wsbexchange.exe” /v AppId /t REG_SZ /d “{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /f
reg add “HKLM\Software\Microsoft\windows nt\currentversion\WindowsServerBackup\Application Support\{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}” /v “Application Identifier” /t REG_SZ /d Exchange /f
reg add “HKLM\Software\Microsoft\windows nt\currentversion\WindowsServerBackup\Application Support\{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}” /v CLSID /t REG_SZ /d “{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}” /f
reg add “HKLM\Software\Microsoft\windows nt\currentversion\WSBAppExchangeHelper” /v AutoMarkDbRecoverable /t REG_DWORD /d 1 /f
reg add “HKLM\Software\Microsoft\windows nt\currentversion\WSBAppExchangeHelper” /v AutoMountOnPITRecovery /t REG_DWORD /d 1 /f
sc.exe create wsbexchange binpath= $WsbBinPath type= own start= demand error= ignore obj= LocalSystem DisplayName= “Microsoft Exchange Server Extension for Windows Server Backup”
sc.exe description wsbexchange “Enables Windows Server Backup users to back up and recover application data for Microsoft Exchange Server.”
——————
Script Ends

I followed these instructions and still the Exchange backup didn’t complete properly. The number of logfiles had now grown to close to 100 000, over 100 GB of disk space. The other way (apart from running a backup) of truncating Exchange logfiles is stopping the database and ensure that it’s in a clean shutdown state using Eseutil /mh which I did. All the databases where cleanly shutting down so I moved all the logfiles to another volume, starting each database then creates a fresh set of logs and backup now ran fine.

The moral of the story, don’t trust green tick marks, always check for yourself and verify that not only the file part of a backup is working, check Exchange (and SharePoint / SQL) as well.

For the full story with all the gory details check the MS partner forums under Exchange / Messaging for “SBS 2008/Exchange 2007 unhappy database – please recommend next steps”.

Thanks for reading – and remember to verify your backups!

4 thoughts on “The importance of verifying your backup”

  1. Great post. I was checking continuously this blog and I’m impressed! Very useful info specially the last part 🙂 I care for such information much. I was looking for this particular information for a long time. Thank you and best of luck.

  2. I have been browsing online greater than three hours nowadays, but I never found any fascinating article
    like yours. It is beautiful value enough for me. In my view, if all site owners and bloggers made excellent content as you did, the net might be a lot more helpful than ever before.

  3. I tend not to create a comment, but after reading some of the remarks on The importance of verifying your backup Tell IT
    as IT is. I do have a couple of questions for you if you don’t mind. Could it be simply me or do some of the responses look like they are left by brain dead individuals? 😛 And, if you are writing at additional online social sites, I would like to keep up with anything new you have to post. Would you make a list of every one of all your social sites like your Facebook page, twitter feed, or linkedin profile?

    1. Hi,
      I put it down to unfamiliarity with English rather than brain deadness, but I’ll keep watching it 🙂

      I write a fair bit for 4Sysops.com and my twitter feed is paulschnack.

      /Paul

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s