Fixing Veeam Hyper-V Replication Failure Caused By VSS Writer

I just had an issue with the Veeam Backup and Replication Software where a replication job was failing.  I have Veeam setup to email error logs to me and I started getting error messages with a replication job where it was not reconciling the differences in the disk on the fail-over server.

Here is a sample of the email error.

After getting a few of these emails I checked the server and discovered on of the VM replication jobs appeared to be stuck on in progress.

A Sample of the log on the Veeam Server

---

VM {VMNAME} task has finished with 'InProgress' state.
Task details: Failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.

The writer experienced a transient error.  If the backup process is retried,
the error may not reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.

Make sure VM does not have 'iSCSI Software Target Storage Provider' feature installed.
Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.

The writer experienced a transient error.  If the backup process is retried,
the error may not reoccur.

--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)

Task has been rescheduled

---

Googling the issue I only found this blog http://www.insidetechnologies.eu/en/blog/veeam-br-resolve-hyper-v-vss-writer-error/  and after double checking that his solution wouldn't bring down my Hyper-V Server by checking Microsoft Technet https://social.technet.microsoft.com/Forums/en-US/69e6dcbd-fbe3-43e0-a7b3-65111fbdf00a/will-the-running-vms-be-impacted-if-i-restart-the-hyperv-virtual-machine-management-service?forum=winserverhyperv

I disabled the replication jobs and stopped the Hyper-V Virtual Management Management Service on the primary server; while it took a while for the management service to stop but once it did and I restarted it everything came back no problem and the replication jobs started, reconciled and finished without an issue.

Setting Up Veeam Backup and Replication in a Hyper-V Enviroment

This is a modification of a Veeam backup and Replication setup I have done that is in production.  This system is setup in a WORKGROUP enviroment.   This was done for 2 reasons, all the virtual machines are in a non-domain setup, and I didn't have the resources to setup a full domain for the Veeam setup.  I also have some recommended reading that I would suggest you do before you start to implement the Veeam Backup and Replication System.  This post is going to be a variation of the Simple Deployment Model based on the Veeam user guide.

Veeam Hyper-V Simple Deployment Model

I do have some recommendations that I would suggest putting in place if your organizations budget can handle it.

1. Two physical active directory controllers and 1 virtual read only controller if your using a clustered hyper-v system (Mostly used for the Cluster System)
2. A cluster system for the Veeam Backup and Proxy Server which are virtualized.  Having a separate server for the Guest interaction proxy and backup proxy will take load off the source host.
3. Freenas or other high storage and performance NAS System with dual power supplies and battery backup with at minimum 2 disk redundancy (for freenas known as RAIDZ2).
4. Smart switches with LACP LAGG and vlan support

For the purposes of this post we are going to assume a few things.  You have all the necessary licences you require for Microsoft Windows Server 2012R2 or better including all CALS, SQL Server, SQL Server Cals, Veeam licences etc.  There is have a working infrastructure in place and for our purposes we are going to use 192.168.42.x network.  This is a setup for a SMB with no available offsite hosting capabilities, this is just an onsite redundant setup for when the virtualized servers needs to be put into a maintenance mode or  have had some sort of failure either in the VM or the Hyper-V Host.

The setup

We are going to use a workgroup setup and the VM's don't need to be on the same network as the host or the storage essentially what you really need to talk to each other is the NAS, Veeam Server, the Primary and Fail-Over servers and the gateway/router.   For my sanity on this post everything is just on the same network.  If you use a domain you can simplify a lot of this due to having a domain level admin on all systems attached to the domain.

Gateway/Router Primary Host Failover Host Veeam Server VM 1 (SQL Server) VM 2 (Apache Server on internet) VM 3 (3rd party connection server) VM 4 (report server) FreeNAS SMB

192.168.42.1 192.168.42.2 192.168.42.3 192.168.42.4 192.168.42.5 192.168.42.6 192.168.42.7 192.168.42.8 192.168.42.9

In Veeam when you setup a VM for replication the wizard gives you a variety of questions to answer.

1. Create a new replication job



















2. Select your failover host and specify the save location.

3.  Verify The Job Settings.  Next I will be going though Advanced Settings

4.  This Veeam setup is on the same LAN, so optimizing for low bandwidth is not as important but you should always optimize your replication job for which ever environment your in.  In this case a LAN Network Target Server for Failover and Production.

5.  I've setup my Veeam instance for Application Aware Image Processing.  Guest Processing, which will allow you to create a transitionally consistent backup, configure transaction log handling settings, and enable guest file system indexing.

6.  Veeam also has the ability to excite PowerShell scripts when running fail-over/replication jobs.  I haven't set anything up yet but it is possible to do.

7.  If I had setup an off-host proxy server that would be handling the majority of the backup and not the Primary host server which does create quite a bit of additional load.  Currently as per the diagram in the beginning of this post there is the Veeam Server, The Primary Production Server and the Fail-Over/Backup Server.  I will be adding a off-host proxy server to take most of the load off the production server in an effort to speed up the backups and the replication time.

8.  If you have a copy of the VM on the backup server you can do what's called Replica Seeding.  This will significantly reduce the amount of bandwidth used.

9.  Here is where you define the guest processing for the Veeam Backup and Replication Server.  It is recommend that you set this up.

10.  Configure you backup/replication schedule

11.  This is the summary screen, it will tell you everything about your backup/replication job and it is pretty easy to find configuration errors.  One done your Veeam replication job is ready and running.

I would definitely recommend Veeam for any company that is looking for a easy fail-over solution especially if they need to be up 24/7.  The backup jobs work much the same way and when I have done testing of the fail-over I've setup it appears as just a blip in the process, and is not really noticed.  A few of my personal goals for this is to get an off-site proxy server set up, and move the backup server offsite and setup the reconfiguration ip addresses which Veeam is capable of.

I definitely recommend going though Veeam University https://www.veeam.com/university.html