Our Thinking / Field Notes / The Guide

STEP-BY-STEP GUIDANCE / BACKUP AND RECOVERY

How to Check an Immutable Backup Can Actually Be Restored

Immutability stops backups being deleted or altered. It does not prove they restore. Here is a safe way to restore from an immutable copy into isolation, check the data and application, and time the result.

Start here

Do not rely on the job's green tick. Confirm which copies are genuinely immutable and for how long, then restore a known test set from that copy into an isolated location and compare it with what you backed up. Open a real application from the restored data and record how long it all took. Immutability tells you a copy cannot be deleted or changed. Only a restore tells you the copy is usable.

What a green tick does and does not tell you

A green tick means the backup job reported success. It does not mean anyone has opened what it wrote. Immutability adds a second, separate guarantee: during its protection period, a recovery point cannot be deleted or changed, even by someone with stolen administrator credentials. That is valuable, and it is the right control against ransomware that goes looking for your backups.

But an immutable copy can still be unusable. It may have captured data that was already encrypted or corrupted. It may be protected for a shorter window than you assumed. Perhaps nobody can reach the encryption password, or the restore takes three days when the business can manage one. The NCSC's ransomware guidance makes two relevant points: make sure you know how to restore from the backup and test regularly that it works, and scan backups for malware before restoring, because ransomware may have reached them before it was detected.

This article gives you a restore test that proves recovery rather than configuration. You can follow it with almost any backup product. Where Veeam Backup & Replication or Azure Backup have a specific behaviour that changes the test, we say so.

Before you begin

You will need read access to your backup console and permission to run a restore. You will also need somewhere safe to restore to. Nothing here needs you to delete, edit or unlock a production backup.

  • A spare machine, virtual machine or isolated virtual network where a restored copy cannot talk to your live systems. A restored server can carry the same name, address and identity as the live one, and it may carry malware too.
  • Enough free disk space for the restored test data. A single file share or one small server is plenty for a first run.
  • Someone who knows the application well enough to say whether the restored data looks right. A passing checksum is good. A person recognising their own invoices is better.

Confirm which copies are immutable, and for how long

Start with a read-only look at the setting. You want to know three things: which repository or vault holds the copy you would rely on in a ransomware incident, how long its recovery points are protected, and what is not covered.

Protection length matters more than people expect. Ransomware can sit quietly for some time before it detonates, and the NCSC notes that infections can replicate into backups before they are discovered. If recovery points are immutable for 7 days but you might not notice a compromise for three weeks, your older, clean points may exist but be deletable. That is a design judgement, not a fault, but you should make it knowingly.

In Azure Backup, open the vault, go to Properties, then Immutable vault, and select Settings. The vault will be in one of three states: Disabled, Enabled, or Enabled and locked. Enabled can still be switched off by someone with Backup Contributor permissions who also passes Resource Guard verification. Only Enabled and locked cannot be reversed, and Microsoft describes locking as irreversible. There is also a choice between Enable based on backup policy and Enable for specific duration. With a specific duration, recovery points stop being immutable once that period passes, even though they remain until policy retention expires. Microsoft's concept page also says immutability does not apply to operational backups for blobs, files and disks, so check which tier of copy you are actually looking at.

In Veeam Backup & Replication, a hardened repository applies an immutability period set in days to its backup files. Compare that period with your job retention, and confirm the job you are testing really writes to that repository rather than to a copy that sits beside it.

Please do not try to prove immutability by deleting a production recovery point, or by cutting a live policy's retention to see what happens. If immutability is not on, you will have performed the very deletion you were guarding against. If you want to watch the block in action, use a disposable test vault in the unlocked state. Microsoft's own example is that reducing a policy's retention in an immutable vault is rejected, while increasing it is accepted. Never lock a test vault, because locking cannot be undone.

Plant canary files and take a backup

A restore test is only convincing if you know exactly what should come back. Rather than comparing against live data that has changed since the backup ran, create canary files: a handful of small, deliberately created files whose contents you record in advance. You then compare what returns against that record, using a SHA-256 hash, which is a fingerprint that changes if even one byte differs.

This will create three tiny text files and a manifest on a file server. The files will then be backed up like anything else and will stay in the backup until retention and immutability expire. They are harmless, but do not expect to delete them early.

  1. Pick a file share or folder that your backup job already covers. On that server, run the commands below in PowerShell with an account that can write to the folder. Change the D:\Shares path to a folder inside your protected data. The commands create the canary files and save their hashes to a manifest outside the protected folder.
    powershell
    $folder = 'D:\Shares\RecoveryCanary'
    New-Item -ItemType Directory -Path $folder -Force | Out-Null
    New-Item -ItemType Directory -Path 'C:\Admin' -Force | Out-Null
    1..3 | ForEach-Object { Set-Content -Path "$folder\canary$_.txt" -Value "Canary file $_ created $(Get-Date -Format s)" }
    Get-ChildItem -Path $folder -File | Get-FileHash -Algorithm SHA256 | Select-Object Hash, @{n='Name';e={Split-Path $_.Path -Leaf}} | Export-Csv -Path 'C:\Admin\canary-manifest.csv' -NoTypeInformation
  2. Run the backup job that covers that folder, or wait for its next scheduled run, and note the time the new restore point completes. You are about to test that exact point.
  3. Copy the manifest to somewhere you can reach from your isolated test machine, such as an encrypted USB drive or a restricted admin share. Do not leave the only copy on the server you are testing.

Restore from the immutable copy into isolation

If you run Veeam Backup & Replication and your licence includes SureBackup, it automates much of this for virtual machine backups. Its full recoverability mode starts machines from the backup in an isolated virtual lab and runs heartbeat, ping and application tests. Its lighter backup verification and content scan mode only checks file integrity with a CRC check and scans content, and it does not boot anything. Both are useful. Only the first shows that a machine starts. Check your edition before relying on it, and still do the manual restore below at least occasionally, because a human will notice things a scripted test does not.

The manual route works with any product. The aim is to restore from the copy that would survive an attack, not from the convenient local copy that is usually quicker.

  1. Prepare the isolated destination: a spare machine, or a virtual machine attached to a virtual switch with no route to your production network or the internet.
  2. Scan the recovery point for malware before restoring, using your backup product's scanning option or security software on the isolated machine. The NCSC advises scanning backups before restoring files, since an infection may already be in them.
  3. Start the restore from the immutable repository or vault, selecting the restore point you noted earlier. Restore to a new location on the isolated machine. Never choose an option that overwrites the original, and never connect a restored server to the live network to see what happens. Note the start time.
  4. Wait for it to finish. A small file restore may be quick. A full server restore depends on data size, repository speed and, for cloud copies, your download bandwidth, so it may take a long time. Do not guess it. The elapsed time is one of your results.

Check the restored data, then the application

Integrity of files and usability of a system are different questions, so check both. The canary comparison is read-only and quick. The application check is what tells you a business system would come back.

  1. On the test machine, run the comparison below, changing the two paths to your manifest and the restored canary folder. Every line should begin MATCH. MISSING means the file did not come back. DIFFERS means it came back altered, which you should treat as a failed restore.
    powershell
    $restored = 'E:\RestoreTest\RecoveryCanary'
    $manifest = Import-Csv -Path 'E:\RestoreTest\canary-manifest.csv'
    foreach ($item in $manifest) {
        $path = Join-Path $restored $item.Name
        if (-not (Test-Path $path)) { "MISSING  $($item.Name)"; continue }
        $hash = (Get-FileHash -Path $path -Algorithm SHA256).Hash
        if ($hash -eq $item.Hash) { "MATCH    $($item.Name)" } else { "DIFFERS  $($item.Name)" }
    }
  2. Open a few real restored files or records with someone who uses them. For a restored virtual machine, boot it on the isolated network, sign in with a local administrator account and confirm that the services the business depends on start. If the only way in is a domain account, you have just found a recovery dependency.
  3. If you restored a SQL Server database to a test instance, run a consistency check against the restored copy only. It reads the whole database, so expect heavy disk activity and do not run it on the production server. A clean run reports no errors. Replace YOUR_RESTORED_DATABASE with the name you restored it as.
    sql
    DBCC CHECKDB (YOUR_RESTORED_DATABASE) WITH NO_INFOMSGS;
  4. Write down the date, the restore point used, the elapsed time, what passed and what did not. Then tear down the isolated machine or virtual machine and delete the restored copy. It holds a full copy of your data, so do not leave it lying around.

Would this work on a bad day?

A restore that works on a calm Tuesday with everything online can still fail when your directory, your email and your main server are gone. Use the same test to ask what you would be missing. These are our engineering judgements rather than vendor requirements:

As for how often, the NCSC says to test regularly but does not set an interval. A sensible pattern is a quick file restore like this one every month or so, and a full system restore at least once a year or after any significant change to your backup set-up. Pick a rhythm you will actually keep.

  • Can you sign in to the backup console if your normal identity provider or domain is unavailable? A documented local or emergency account, with its multi-factor recovery stored offline, is the usual answer.
  • Are encryption passwords and keys stored somewhere that does not depend on the systems being backed up? A password held only inside the encrypted backups cannot help you.
  • Could you reconnect a freshly installed backup server to the repository, and do you hold a copy of its configuration?
  • Does the elapsed time you recorded fit what the business can tolerate? If not, that is a finding. It is not a reason to skip the test.

If the restore does not work

Most failures fall into a few patterns. Treat each as a result worth fixing, which is the point of testing.

  • The restore point is missing or the older point you expected has gone. Compare job retention with the immutability period. In Azure Backup with a specific duration, points can lapse out of protection and then expire under normal policy.
  • Access is denied during the restore. Check which account and role the restore needs, and whether a Resource Guard or approval step is involved. Sort this out before an emergency, not during one.
  • Files match but the application will not start. The backup may be crash-consistent rather than application-consistent, meaning it captured disk state without the application flushing its data. Check whether application-aware processing is switched on for that job.
  • The restore completes but is far slower than expected. Note the repository speed, the network path and the size of the data. Cloud copies may take much longer to pull back than local ones.
  • Hashes show DIFFERS or the malware scan reports a finding. Stop. Do not connect that restored copy to anything. Try an earlier restore point, and if live systems may be affected, follow your incident response plan and seek specialist help.
Technical terms explained
application-consistent
A backup taken with the application's cooperation, so its data is in a clean, usable state. A crash-consistent backup only captures the disk as if the power had been cut.
canary files
Small test files you create on purpose, with contents you have recorded, so that you can tell afterwards whether a restore brought back exactly what was backed up.
immutability
A protection that stops a backup copy being changed or deleted for a set period, even by an administrator account. It guards the copy, but says nothing about whether the copy restores.
restore point
A single saved version of your data at a particular time, created by one backup run. You choose a restore point when you recover.
SHA-256
A method of producing a fixed-length fingerprint, called a hash, from a file's contents. If a single byte changes, the fingerprint changes completely.
virtual lab
In Veeam's SureBackup, an isolated environment where machines are started from backup for testing without touching production.

Sources and further reading

KEEP THE CONVERSATION GOING

Want a second pair of eyes on a recovery test?

If you would like help planning a restore test around the systems you actually depend on, tell us what you run and we will talk it through.

Talk to us