Backup Succeeded, But Can You Recover? How to Test Your Recovery

Backed up, recoverable and still operating are three different things

Most backup dashboards answer one question: did the job finish? A green tick is a statement about the job, not about your Monday morning. Recovery needs two more things the dashboard rarely measures, and they fail in different ways.

• Backup success: the software read the source, wrote a restore point and reported no error. It says nothing about whether that copy is usable.
• Recoverability: you can turn that data back into a working, correct service, using access and dependencies you will actually have on a bad day, within a time the business can tolerate.
• Business continuity: the organisation keeps functioning while IT recovers, and afterwards. That covers people, suppliers, manual workarounds and how data created during an outage is merged back into restored systems. The NCSC recommends planning that reconciliation in advance so a single correct dataset results.

What a green tick does not tell you

Most failed recoveries are not caused by a corrupt backup file. They come from something the job never checked. These are the usual suspects.

• Consistency. A crash-consistent copy is what you would get if the power was pulled: the disk is intact but a database may need to replay or repair its logs, and sometimes cannot. An application-consistent copy has had the application paused or flushed to a clean state first, usually through Microsoft's Volume Shadow Copy Service (VSS) on Windows. Backup software can quietly fall back from one to the other. Veeam, for example, documents an option to carry on if application processing fails, which lets the job finish but gives a crash-consistent result rather than a transactionally consistent one.
• Integrity of the file is not the same as a working service. Veeam's recovery verification checks the backup file with a CRC check, confirms the VM is sending a heartbeat, pings it and runs any test scripts you supply. Only the scripts tell you anything about the application, so a verification job is exactly as meaningful as the scripts you give it.
• Missing dependencies. A restored application server is little use without DNS, a domain controller, certificates, licence servers or the database it talks to. The NCSC says backups should cover more than data: applications, identity systems, trust infrastructure and configuration.
• Access. If the backup console signs in with domain accounts, or the recovery notes sit on the file server you have just lost, the backups exist but nobody can use them.
• Backups are a target. In a 2024 Sophos survey of 5,000 IT and security leaders, 94% of organisations hit by ransomware said attackers tried to compromise their backups, and 57% of those attempts succeeded. It is a vendor survey, so treat the numbers as indicative, but the direction matches CISA's warning that many ransomware variants try to find and delete or encrypt accessible backups.

Checking how consistent your Hyper-V backups are likely to be

Hyper-V's own settings do not tell you what your backup product actually did. The job log does. They are still worth a look, because they show whether the host is set up to produce clean point-in-time images in the first place.

Microsoft describes two checkpoint types. Production checkpoints use VSS, or a file system freeze on a Linux guest, to make a data-consistent image and take no memory state. Standard checkpoints capture memory state too, and Microsoft notes they are not a full backup and can cause data consistency problems with systems that replicate between nodes, such as Active Directory. If a VM is set to Production and the production checkpoint fails, Hyper-V creates a standard one instead. ProductionOnly stops that happening.

The two checks below are read-only. Run them in an elevated PowerShell session on the Hyper-V host, as a local administrator or a member of Hyper-V Administrators. Nothing is changed.

1. List each VM with its checkpoint type.
   powershellCopy

   Get-VM | Select-Object Name, State, CheckpointType

2. Look at the CheckpointType column. Production or ProductionOnly is what you would normally expect for server workloads. Standard on a domain controller or database server deserves a question, because it may not be the clean image you assume.
3. Check that Integration Services, the guest components that give Hyper-V its heartbeat and VSS coordination, are healthy in a running VM. Replace YOUR_VM_NAME with the VM's name. A healthy VM normally shows the service enabled with a status of OK. Veeam's application-aware processing needs these services in the guest, and its heartbeat test is skipped if they are missing.
   powershellCopy

   Get-VMIntegrationService -VMName YOUR_VM_NAME -Name Heartbeat

4. Open the most recent backup job session in your backup tool and read the detail for warnings about application processing, rather than trusting the summary icon.

Agree what recovered means before you test it

A test needs a pass mark. Two numbers provide it, and both are business decisions rather than technical settings.

The recovery point objective (RPO) is, in NIST's wording, the point in time to which data must be recovered after an outage. In plain terms: how much recent work can you afford to lose? If the answer is an hour, a nightly backup cannot meet it however reliable it is. The recovery time objective (RTO) is the length of time a system can be in recovery before it harms the business. How long can this service be down before it hurts?

Two points catch people out. First, RTO is not the backup window or the copy time. It runs from the moment of failure to the moment the business can use the service, so it includes finding the right restore point, getting credentials, restoring dependencies in order, waiting for large VMs to copy back and checking the result. Second, RPO and RTO belong to a business service, not a server. Ask finance or operations how long they can manage without a system, then check whether your design can honestly deliver it. When it cannot, the test has done its job: it has shown that the stated objective and the real capability disagree.

Test levels and what an isolated environment is for

Restoring something over the top of production to see if it works is a poor test and a risky one. A restored copy has the same machine name, network identity and often the same IP address as the live system, so putting it on the production network can cause conflicts, or in a ransomware case reintroduce the problem. Test in an isolated recovery environment instead: a network with no route to production, where restored machines can talk to each other but nothing else.

Veeam calls its automated version a virtual lab. Backed-up VMs boot in the lab, a proxy appliance acts as the gateway into it, and the verification tests run against them. Veeam also documents a manual route that suits smaller sites: create an isolated network, run Instant VM Recovery for the VM you are verifying, and at the network step connect it to the isolated network rather than production. If the application depends on other servers, restore those into the same isolated network and start them in order, for example DNS, then the domain controller, then the application server. In plain Hyper-V terms, the isolated network is a virtual switch that is not bound to a physical adapter, such as a Private switch. Which Veeam editions and licences include the automated verification has varied between versions, so check your own entitlement rather than assuming. The principle applies to any backup product.

Not every system needs the heaviest test. Our view, and it is judgement rather than a published standard, is to match the effort to how much the business depends on the system.

• Automated verification after backups: cheap, frequent and good for every critical VM. It proves the operating system boots and the file is readable. It proves the application works only if your scripts do.
• Application check: a person or script signs in, opens a recent record, runs a report or posts a test transaction, and confirms the newest data is there. This is the first level that tests recoverability in the sense the business cares about.
• Service restore drill: restore a whole chain of systems in dependency order into the isolated network, timed, by the person who would do it under pressure, using only the written procedure. Having someone other than the usual engineer follow the notes is a very effective way to find what lives only in someone's head.
• Ransomware scenario: a drill or tabletop that starts from the assumption that identity and the backup server are compromised. The next section covers it.

Testing for ransomware, not just for a failed disk

Recovering from hardware failure and recovering from ransomware are different exercises. In the second, you cannot trust the accounts, the network or possibly the oldest recent restore points. CISA's guidance is to keep offline, encrypted backups of critical data, regularly test their availability and integrity in a disaster recovery scenario, restore based on a prioritisation of critical services and take care not to re-infect clean systems. The NCSC adds that backups should be immutable where possible, meaning they cannot be altered or deleted for a set period, and kept separate from primary systems. It also advises prioritising business-critical services in restoration testing because restoring can take significant time.

Whether you have immutable or offline copies is a design question for another day. These are the things a ransomware-flavoured test should check.

• Can you sign in to the backup console and the repository without domain accounts that an attacker may now control?
• Is there a restore point that predates the intrusion, and how would you decide which one? Try restoring an older point, not just the latest.
• Can identity come back first? Practise restoring or rebuilding the domain controllers and DNS in the isolated network before anything that depends on them.
• Does restored data get scanned and the machines checked before they join a production network? Veeam's current guide lists malware detection and Secure Restore features, but availability and behaviour depend on version and licence, so verify before you rely on them.
• Where is the runbook? NCSC advises keeping recovery documentation secure and separate from production systems. A copy only on the encrypted file server does not count.
• How many systems can you recover within the RTO if you have to rebuild rather than restore? The answer is often fewer than expected and is better learned in a test.

Run a recovery test: the sequence

This works for a first test and scales up. Plan for the restore of large machines to take real time. Measure it, because you cannot reliably infer it from how long the backup took. If the test is a long copy, you can step away, but stay reachable: restore wizards sometimes pause for a decision.

1. Pick one business service rather than one server, such as the finance application, its database, the DNS and domain controller it relies on and the file shares it uses. Write down the RPO and RTO the business owner would expect for it.
2. Choose restore points deliberately. Use the latest one and also an older one, for example from before the last month-end. Testing only the newest point misses problems with retention and backup chains.
3. Prepare the isolated network and confirm it has no route to production before you power on anything. This is the one step where a test can harm live systems: if a restored VM lands on the production network with its original name and address, it can clash with the real one. Check the network selection in the restore wizard before you start, not after.
4. Start the clock. Restore dependencies first, then the application, following only the written procedure, and note every point where you had to guess, look something up or ask someone.
5. Check the application, not the VM. Confirm that users can sign in, that a recent record is present, that the newest data is as recent as the RPO promises, and that someone from the business agrees the service is usable.
6. Stop the clock when that person agrees. Compare elapsed time with the RTO and the age of the newest data with the RPO.
7. Clean up through your backup tool by ending the recovery session and removing the test machines, so nothing is left mounted from the repository or running on a host. Do not delete virtual disk files by hand. Confirm afterwards that no test VMs remain on the host and the isolated network is no longer in use.

Document the result and turn gaps into work

A test nobody wrote up might as well not have happened. Keep the record short enough that it will be completed, and store it somewhere that does not depend on the systems you are recovering. Never put passwords or recovery keys in it.

• Record: date, service tested, restore points used, who ran it, the written procedure followed and the environment used.
• Record the results: elapsed time against RTO, data age against RPO, and whether the business owner accepted the service as usable.
• Record the friction: every guess, missing credential, absent licence, surprising dependency, undocumented step and failed verification script.
• For each gap, add an owner, a fix, a date and a retest. Typical gaps are documentation, access that depends on the failed domain, dependency order, a backup missing a component, an application consistency setting and plain slowness.
• If a gap is simply that recovery takes longer than the RTO, take it back to the business owner. Either invest to shorten recovery or agree a longer objective, and write down which. Leaving the old number in place and hoping is the only option that is certainly wrong.
• Retest after meaningful changes such as a new backup platform, a replaced domain controller or a major application upgrade, as well as on a regular schedule. The NCSC and CISA both say regularly without naming an interval, so pick one your business will sustain and keep to it.