TrueNAS Server Setup

I setup a new TrueNAS system to work with my lab server for storage.  The system is running on an older platform an Intel Haswell 1150.  The system has 32 GB of ram and as you can see below the system only has 6 sata ports.  I put in a Matrox 10Gig card for connecting my server 2022 system to the iscsi target so that left me with one PCI Express port to add a raid card or a card for additional sata ports. 

Now I am a big VANTEC fan, I have purchased a lot of enclosures and adapters from them over my IT career and they have worked really well and I came across this nice little card.

https://www.vantecusa.com/products_detail.php?p_id=286  it was at my local computer retailer memory express https://www.memoryexpress.com/Products/MX00120961 where it is retailing for $59.99.

This card requires at least a Gen3 4x PCI-E port, I know my board supported Gen3 PCI-E so I thought I would give it a try.

Initially I had issues getting TrueNAS to boot with the system encountering re-scan errors when trying to initialize the card.

(noperiph:ahcich3:0:-1:ffffffff): rescan already queued

After some troubleshooting; I determined that it had to do with the system cold booting because after a couple of reboots the system would boot and recognized all the drives.  I also noted that the card did not initialize until just before they system was about to boot.  The system registered all my drives, I have 3 SSD drives on the motherboard SATA ports and the rest of my 3+ TB drives running off the card.

So for the disk setup I have a 120GB SSD boot drive, 40Gig SSD Z-Log Drive, and 120GB SSD Cache Drive.  The rest of the drives are NAS drives mostly WD REDS and two 3TB seagate iron wolfs.  

I setup a SMB share to test how the system might perform

Though a 1Gig network the samba share registered the following through put

When I setup the iSCSI target on my lab server I will update this but for now I am pretty happy about this and anyone looking for an inexpensive controller for TrueNAS, I would  defiantly recommend this controller if your on a budget.  The throughput on the NAS is pretty much max out on my 1Gig network, and the NAS did not sweat a bit!

FreeNAS What is a L2ARC Cache and why is it useful?

In FreeNAS nothing beats RAM. A general rule of thumb is a minimum is 1GB of Ram per TB of storage but might need to be adjusted depending on workload/application.  Personally I prefer to have 2 GB RAM to 1 TB of disk if possible. Generally in the FreeNAS (truenas) community it is well known that it is recommended that FreeNAS run on 32GB of ECC RAM minimum, however I have found that if you are using a the 2:1 Ram ratio you can use non-ecc ram and FreeNAS runs fine though is not recommend by FreeNAS; but you work with what you got.

FreeNAS uses ZFS which provides a read cache in RAM, known as the ARC, to reduce read latency. FreeNAS adds ARC stats to top(1) and includes the arc_summary.py and arcstat.py tools for monitoring the efficiency of the ARC. 

If you use a L2ARC cache it is recommended using an SSD as a cache device.  ZFS uses it to store more reads which can increase random read performance.  If you are using a low ram system you shouldn't use a L2ARC, it will not increase performance, and in most cases you will actually hurt performance and could cause system instability.  If the hit rate of the ARC is below 90%, the system can benefit from L2ARC. If the ARC is smaller than RAM or if the hit rate is 99.X%, adding L2ARC to the system will not improve performance, the L2ARC is really only useful if your asking for the same data over and over again. 

As far as selecting appropriate devices for L2ARC, they should be biased towards random read performance. The data on the L2ARC is not persistent, and the L2ARC will be cleared on a reboot.  If you have a drive failure on your L2ARC device there is no need to worry ZFS is resistant and only uses it for read caching; your NAS will continue to function, however read performance will most likely be affected.  There is no need to provision, mirror or otherwise make L2ARC devices redundant, nor is there a need for power protection on these devices.

For example on some of my backup NAS systems I've build where I am moving over 500+ GB of data within 20 minutes my L2ARC cache gets hit quite a bit.  However it depends on what your doing.  It might be more beneficial to have a mirrored SSD pool for more active data and spinning drives for less active content then having a L2ARC Cache if the data needs to be read.

I would suggest using the tools in FreeNAS and monitoring your L2ARC's effectiveness using tools such as arcstat. If you need to increase the size of an existing L2ARC, you can stripe another cache device using Volume Manager. The GUI will always stripe L2ARC, not mirror it. As stated above the L2ARC data is not persistent.

I usually add a L2ARC if I have a spare SSD handy; with a 2:1 RAM ratio my experience is that it does help more then hurt but it depends on what your setup is and what your using it for.  Most of the NAS Systems I use are used for smb backups, and I'm backing up severs virtual disks which are always changing and moving them from NAS to NAS.  Having an L2ARC cache doesn't help there but with a file server that servers out files that don't change much the L2ARC does help with the read performance.

Note: that a dedicated L2ARC device can not be shared between ZFS pools.

Reference:
http://www.freenas.org/blog/a-complete-guide-to-freenas-hardware-design-part-iii-pools-performance-and-cache/

http://www.freenas.org/hardware-requirements/

https://doc.freenas.org/9.3/zfsprimer.html