Wednesday, August 24, 2016

FreeNAS Disaster Recovery

FreeNAS is a fantastic NAS solution it is really robust when it comes to management, backup and restoring from a dead USB Key, Hard Drive failure are fantastic.  However what if your motherboard dies?  What if your using a RAID controller in JOBD and it dies?  

I had the RAID controller die on me and the consensus on the internet was I'm F*cked.  

I had setup a FreeNAS system on a Quadcore AMD Phenom System from 2007, because I needed a system with lots of storage space, for backup purposes, which this chassy allowed.  The Processor is 2.3 Ghz, and has 40 GB of RAM along with dual power supplies.  I had put in an Areca 1120 because the SATA Bus is faster then the onboard SATA (SATA2 Vs SATA1) on the motherboard.  The system was configured with 8 3TB WD RED NAS drives in a Z2RAID, and an 80GB Intel 520 SSD for a arc cache.  The system was working as an iSCSI target and SMB server and performed extremely well, with the slowest part of the system being the network connection.  Then when I came in to work this morning the system was completely unresponsive.  I had 2 virtual machines running off the iscsi target which were in production, a Linux Print Server and a mailforwarder / Scheduling server.  I had a backup up to a few days before which I recovered to but I had to get the the data for the the scheduling software.

After trying a few reboots hoping the ARC RAID controller might magically work I disconnected it and tried to plug the drives into the mainboard and the system booted but FreeNAS couldn't find the Storage Pool.  I was not surprised with this result and no backup raid controller, it was time to get creative.  I had access to 2 different PCI Silicon Image Software RAID cards a 4 port and a 2 port so I thought I would try put them in and hook up 6 of the drives that were on the failed raid controller to these software based controllers I might have a chance to get the data; after all I had configured the 1120 as a JOBD setup and let FreeNAS handle the raid as stated in their best practices. 

It worked.  FreeNAS booted and was able to access the storage pool but in a "Degraded" state.  I was able to pull the VHD from the iscsi target that I needed with NO LOSS of data, and put it back into production.  I also grabbed a few other files while I had the server up, but the recovery was a complete success.

This is the first time I've tried this, but if anyone is having a issue with freenas not reading a storage pool after a hardware failure and your OS USB drive is in good shape give this Silicon Image RAID controllers a try it worked for me it might work for you.

How Windows 11 Pro and Workstation Pro handles VLANs and Virtual Switches

Windows 10 pre 1809 and Windows Server pre 2022 typically would be setup with a Teamed network adapter using LBFO (Load Balancing and Failov...