Wednesday, August 24, 2016

FreeNAS Disaster Recovery

FreeNAS is a fantastic NAS solution it is really robust when it comes to management, backup and restoring from a dead USB Key, Hard Drive failure are fantastic.  However what if your motherboard dies?  What if your using a RAID controller in JOBD and it dies?  

I had the RAID controller die on me and the consensus on the internet was I'm F*cked.  

I had setup a FreeNAS system on a Quadcore AMD Phenom System from 2007, because I needed a system with lots of storage space, for backup purposes, which this chassy allowed.  The Processor is 2.3 Ghz, and has 40 GB of RAM along with dual power supplies.  I had put in an Areca 1120 because the SATA Bus is faster then the onboard SATA (SATA2 Vs SATA1) on the motherboard.  The system was configured with 8 3TB WD RED NAS drives in a Z2RAID, and an 80GB Intel 520 SSD for a arc cache.  The system was working as an iSCSI target and SMB server and performed extremely well, with the slowest part of the system being the network connection.  Then when I came in to work this morning the system was completely unresponsive.  I had 2 virtual machines running off the iscsi target which were in production, a Linux Print Server and a mailforwarder / Scheduling server.  I had a backup up to a few days before which I recovered to but I had to get the the data for the the scheduling software.

After trying a few reboots hoping the ARC RAID controller might magically work I disconnected it and tried to plug the drives into the mainboard and the system booted but FreeNAS couldn't find the Storage Pool.  I was not surprised with this result and no backup raid controller, it was time to get creative.  I had access to 2 different PCI Silicon Image Software RAID cards a 4 port and a 2 port so I thought I would try put them in and hook up 6 of the drives that were on the failed raid controller to these software based controllers I might have a chance to get the data; after all I had configured the 1120 as a JOBD setup and let FreeNAS handle the raid as stated in their best practices. 

It worked.  FreeNAS booted and was able to access the storage pool but in a "Degraded" state.  I was able to pull the VHD from the iscsi target that I needed with NO LOSS of data, and put it back into production.  I also grabbed a few other files while I had the server up, but the recovery was a complete success.

This is the first time I've tried this, but if anyone is having a issue with freenas not reading a storage pool after a hardware failure and your OS USB drive is in good shape give this Silicon Image RAID controllers a try it worked for me it might work for you.

How to fix CURL call imporitng an RSS feed on a site blocking CURL calls

There is a 3rd party service provider that my organization uses called bibliocommons.  They have these nice book carousels.  However the car...