FreeNAS Disaster Recovery

FreeNAS is a fantastic NAS solution it is really robust when it comes to management, backup and restoring from a dead USB Key, Hard Drive failure are fantastic.  However what if your motherboard dies?  What if your using a RAID controller in JOBD and it dies?  

I had the RAID controller die on me and the consensus on the internet was I'm F*cked.  

I had setup a FreeNAS system on a Quadcore AMD Phenom System from 2007, because I needed a system with lots of storage space, for backup purposes, which this chassy allowed.  The Processor is 2.3 Ghz, and has 40 GB of RAM along with dual power supplies.  I had put in an Areca 1120 because the SATA Bus is faster then the onboard SATA (SATA2 Vs SATA1) on the motherboard.  The system was configured with 8 3TB WD RED NAS drives in a Z2RAID, and an 80GB Intel 520 SSD for a arc cache.  The system was working as an iSCSI target and SMB server and performed extremely well, with the slowest part of the system being the network connection.  Then when I came in to work this morning the system was completely unresponsive.  I had 2 virtual machines running off the iscsi target which were in production, a Linux Print Server and a mailforwarder / Scheduling server.  I had a backup up to a few days before which I recovered to but I had to get the the data for the the scheduling software.

After trying a few reboots hoping the ARC RAID controller might magically work I disconnected it and tried to plug the drives into the mainboard and the system booted but FreeNAS couldn't find the Storage Pool.  I was not surprised with this result and no backup raid controller, it was time to get creative.  I had access to 2 different PCI Silicon Image Software RAID cards a 4 port and a 2 port so I thought I would try put them in and hook up 6 of the drives that were on the failed raid controller to these software based controllers I might have a chance to get the data; after all I had configured the 1120 as a JOBD setup and let FreeNAS handle the raid as stated in their best practices. 

It worked.  FreeNAS booted and was able to access the storage pool but in a "Degraded" state.  I was able to pull the VHD from the iscsi target that I needed with NO LOSS of data, and put it back into production.  I also grabbed a few other files while I had the server up, but the recovery was a complete success.

This is the first time I've tried this, but if anyone is having a issue with freenas not reading a storage pool after a hardware failure and your OS USB drive is in good shape give this Silicon Image RAID controllers a try it worked for me it might work for you.

Pokemon Go, You, Your Child, and Privacy

Pokemon Go is all the rage right now, and if your a parent you might not be to sure how to handle this new app craze that is the Angry Birds of 2016. Pokémon Go is a free-to-play location-based, augmented reality game developed and published by Niantic, Inc. for iOS and Android devices.  What you might not be aware of is some of the privacy issues that you may be exposing you and/or your child to.

As you may or may not be aware the way Pokemon Go! uses two ways to sign up for an account.  Using a Pokemon Trainer Account and tieing it to a Google Account.  If you are signing up a child to play this game I highly recommend you use the Pokemon Trainer account.  If you sign up your child with a new google account there are some issues.

1) The Google Terms of service specifies that a user must be at least 13 years of age.

"At the moment, age restrictions don't warrant a single word in Google's Gmail Terms & Privacy details. Google does, however, spell them out on YouTube, buried down in item 12 of its Terms of Service. “If you are under 13 years of age, then please do not use the Service."

You can use the gmail account setup by the schools if the schools are setup for using google because there are additional privacy protections with those accounts.

2) The Pokemon Trainer account is suppose to compliance with the Children's Online Privacy Protection Rule "COPPA" as is the Google Accounts for schools, and has far more privacy restrictions then with a general Google Account. 

Granted that COPPA is a US based law and according so TWIL Episode 353 it is suppose to offer better privacy protection than your typical google account.

HOW TO CREATE POKÉMON TRAINER CLUB ACCOUNT

1. Go to the Pokémon.com official website or click on this link to go to the signup page https://club.pokemon.com/us/pokemon-trainer-club/sign-up/

2. Once you enter the webpage, below the “Join the Pokémon Trainer Club”” click on “Create an Account” button.
3. Click on the button marked, “Continue,” in the section on the right hand side of the page that is marked, “Sign Up! New to Pokémon.com ? Sign up for an account now!”
4. Select your country of residence from the drop-down menu, then enter your date of birth, Next, click on the button marked, “Continue.”
5. Complete the sign up process as directed to activate your account.
6. Done. Now go back to your phone and start the Sign up process. Please make sure to use your Pokémon Trainer Club Account instead of your Google Account.

Fixing Veeam Hyper-V Replication Failure Caused By VSS Writer

I just had an issue with the Veeam Backup and Replication Software where a replication job was failing.  I have Veeam setup to email error logs to me and I started getting error messages with a replication job where it was not reconciling the differences in the disk on the fail-over server.

Here is a sample of the email error.

After getting a few of these emails I checked the server and discovered on of the VM replication jobs appeared to be stuck on in progress.

A Sample of the log on the Veeam Server

---

VM {VMNAME} task has finished with 'InProgress' state.
Task details: Failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.

The writer experienced a transient error.  If the backup process is retried,
the error may not reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.

Make sure VM does not have 'iSCSI Software Target Storage Provider' feature installed.
Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.

The writer experienced a transient error.  If the backup process is retried,
the error may not reoccur.

--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)

Task has been rescheduled

---

Googling the issue I only found this blog http://www.insidetechnologies.eu/en/blog/veeam-br-resolve-hyper-v-vss-writer-error/  and after double checking that his solution wouldn't bring down my Hyper-V Server by checking Microsoft Technet https://social.technet.microsoft.com/Forums/en-US/69e6dcbd-fbe3-43e0-a7b3-65111fbdf00a/will-the-running-vms-be-impacted-if-i-restart-the-hyperv-virtual-machine-management-service?forum=winserverhyperv

I disabled the replication jobs and stopped the Hyper-V Virtual Management Management Service on the primary server; while it took a while for the management service to stop but once it did and I restarted it everything came back no problem and the replication jobs started, reconciled and finished without an issue.

Setting Up Veeam Backup and Replication in a Hyper-V Enviroment

This is a modification of a Veeam backup and Replication setup I have done that is in production.  This system is setup in a WORKGROUP enviroment.   This was done for 2 reasons, all the virtual machines are in a non-domain setup, and I didn't have the resources to setup a full domain for the Veeam setup.  I also have some recommended reading that I would suggest you do before you start to implement the Veeam Backup and Replication System.  This post is going to be a variation of the Simple Deployment Model based on the Veeam user guide.

Veeam Hyper-V Simple Deployment Model

I do have some recommendations that I would suggest putting in place if your organizations budget can handle it.

1. Two physical active directory controllers and 1 virtual read only controller if your using a clustered hyper-v system (Mostly used for the Cluster System)
2. A cluster system for the Veeam Backup and Proxy Server which are virtualized.  Having a separate server for the Guest interaction proxy and backup proxy will take load off the source host.
3. Freenas or other high storage and performance NAS System with dual power supplies and battery backup with at minimum 2 disk redundancy (for freenas known as RAIDZ2).
4. Smart switches with LACP LAGG and vlan support

For the purposes of this post we are going to assume a few things.  You have all the necessary licences you require for Microsoft Windows Server 2012R2 or better including all CALS, SQL Server, SQL Server Cals, Veeam licences etc.  There is have a working infrastructure in place and for our purposes we are going to use 192.168.42.x network.  This is a setup for a SMB with no available offsite hosting capabilities, this is just an onsite redundant setup for when the virtualized servers needs to be put into a maintenance mode or  have had some sort of failure either in the VM or the Hyper-V Host.

The setup

We are going to use a workgroup setup and the VM's don't need to be on the same network as the host or the storage essentially what you really need to talk to each other is the NAS, Veeam Server, the Primary and Fail-Over servers and the gateway/router.   For my sanity on this post everything is just on the same network.  If you use a domain you can simplify a lot of this due to having a domain level admin on all systems attached to the domain.

Gateway/Router Primary Host Failover Host Veeam Server VM 1 (SQL Server) VM 2 (Apache Server on internet) VM 3 (3rd party connection server) VM 4 (report server) FreeNAS SMB

192.168.42.1 192.168.42.2 192.168.42.3 192.168.42.4 192.168.42.5 192.168.42.6 192.168.42.7 192.168.42.8 192.168.42.9

In Veeam when you setup a VM for replication the wizard gives you a variety of questions to answer.

1. Create a new replication job



















2. Select your failover host and specify the save location.

3.  Verify The Job Settings.  Next I will be going though Advanced Settings

4.  This Veeam setup is on the same LAN, so optimizing for low bandwidth is not as important but you should always optimize your replication job for which ever environment your in.  In this case a LAN Network Target Server for Failover and Production.

5.  I've setup my Veeam instance for Application Aware Image Processing.  Guest Processing, which will allow you to create a transitionally consistent backup, configure transaction log handling settings, and enable guest file system indexing.

6.  Veeam also has the ability to excite PowerShell scripts when running fail-over/replication jobs.  I haven't set anything up yet but it is possible to do.

7.  If I had setup an off-host proxy server that would be handling the majority of the backup and not the Primary host server which does create quite a bit of additional load.  Currently as per the diagram in the beginning of this post there is the Veeam Server, The Primary Production Server and the Fail-Over/Backup Server.  I will be adding a off-host proxy server to take most of the load off the production server in an effort to speed up the backups and the replication time.

8.  If you have a copy of the VM on the backup server you can do what's called Replica Seeding.  This will significantly reduce the amount of bandwidth used.

9.  Here is where you define the guest processing for the Veeam Backup and Replication Server.  It is recommend that you set this up.

10.  Configure you backup/replication schedule

11.  This is the summary screen, it will tell you everything about your backup/replication job and it is pretty easy to find configuration errors.  One done your Veeam replication job is ready and running.

I would definitely recommend Veeam for any company that is looking for a easy fail-over solution especially if they need to be up 24/7.  The backup jobs work much the same way and when I have done testing of the fail-over I've setup it appears as just a blip in the process, and is not really noticed.  A few of my personal goals for this is to get an off-site proxy server set up, and move the backup server offsite and setup the reconfiguration ip addresses which Veeam is capable of.

I definitely recommend going though Veeam University https://www.veeam.com/university.html

Three Dumb Routers

Securing your network from IOT (Internet of Things)

IOT is the latest buzzword and a main feature when you go to buy a TV, fridge, thermostat, basically anything you want to buy it is all connected to the internet and that is a bad thing.   

First let me say based on manufacturer’s track records with routers, phones and other devices that connect to the internet this is just crying out to be exploited.

Second, manufacturers will not keep these devices up to date with the latest security patches to fix flaws that could allow a third party to intercept information, or worse gain access to these devices.

Third, unlike traditional computers, where there are alternatives to “throwing out” the device at the end of life, these devices are only good for a short while typically 3 years then you're expected to go buy a new one.

To best protect yourself and your home network I recommend using the “Three Dumb Routers” solution that is described by Steve Gibson in Episode 545 of the Security Now Podcast.

You need three routers, the make and models don’t matter, what matters is the network settings.  It would be best if you configure the other 2 routers before you connect them to your main network to do that consult the setup guide that came with your router.

One of them will be the “Root/Gateway” router or the router that you purchased or lease from your ISP is the Root router (Address Range 192.168.0.x/24) typically the default for all ISP gateway modem/routers; it provides the common link to the Internet for all your connected equipment in this case routers. This router is set up on what’s called a NAT configuration, which stands for Network Address Translations, and means that things on the Internet side, can’t access addresses on the “inside” of that router’s network directly.  The only thing that you would connect to this router on the Local Area Network side (the “inside”, not the Internet or WAN side) are the other two routers as shown in the diagram.  You shouldn’t connect any other devices to the inputs of the Main Gateway router, and it should have wif-fi turned off, if possible. In fact, it may just as well be a non-wifi router or a custom firewall like a PC running PFSense.

The Trusted Network

The trusted network we have on the address 192.168.1.x/24 and should also be set up for NAT-ing, which means anything on its Internet side won’t be able to access things on its LAN or “inside” segment.

The Untrusted (IOT) Network

Finally, we have the untrusted or IOT network; .again, this router is set up with NAT-ing and we have the address range set to 192.168.2.x/24. So, you will never have any trusted devices on the same segment as your untrusted devices. And the devices in each segment won’t be able to see the ones in the other segment.  Also on this network the only thing we want to allow in/out on this router are ports 80 and 443 as IOT devices most commonly only use those ports and if a IOT device requires additional ports it is best to add those ports as they are required.

Why this configuration provides the lowest risk

The reason that this set-up should be secure is that even if any potentially untrusted device is maliciously configured to scan its LAN environment, and if your important devices are not on that segment, they shouldn’t be visible to that untrusted device. There is a slight chance that the untrusted device might be able to find a way to “see” the network segment that is under the “Root” Gateway router but if the only things connected would be the trusted router.  Then there be no other computers or devices (e.g printers, NAS devices, etc.) that can be easily seen or exploited.

DISCLAIMER:

Routers have their own issues, such as vulnerabilities, and you should always check for updates to your routers do a backup of your settings before any upgrade.  Remember routers are your first defence to anything out there in cyberlan.

For additional information please consult Security Now Episode 545

Building A PFSense Firewall For Your Home

Build your own Firewall/Router with PFSense.

pfSense is an open source firewall/router computer software distribution based on FreeBSD. It is installed on a physical computer or a virtual machine to make a dedicated firewall/router for a network and is noted for its reliability and offering features often only found in expensive commercial firewalls. It can be configured and upgraded through a web-based interface, and requires no knowledge of the underlying FreeBSD system to manage. pfSense is commonly deployed as a perimeter firewall, router, wireless access point, DHCP server, DNS server, and as a VPNendpoint. pfSense supports installation of third-party packages like Snort or Squid through its Package Manager.

The system is fairly light weight an can be run on old or inexpensive hardware.  For my purpose which is a home/small business router, where I want VPN capability, squid proxy server and easy router for a web server and low power draw.

I went with a Asus J1800I-A SOC system.  It is a 2.4Ghz Dual core Celeron, and out fitted it with 4GB of DDR3L ram.

I had paid $98 for the SOC, $50 for the ram, $50 for the case and $45 for the PSU.  This SOC system has a PCI port and I used an INTEL Gigabit Adapter for the LAN port (49.99 to buy new).  I also have a 120 GB SSD for the system install.

Overall the system runs between 50 and 60% load.  The Disk usage is about 500mb for the ufs, tmp and var partition.  Memory Usage runs at about 6%.  The system is fast and easy to manage.

SOC:

DDR3 Low Power RAM $49.99

Case & Power Supply $82.00

Intel Gigabit PCI Nic $49.99

Total: 284.98 + Tax

Migrating and expanding virtualbox hard drives.

A user I work with using virtualbox as their hypervisor was having an issue with windows updates and saving files on their profile.  Upon investigation found out there was only about 500mb of storage space left.  To make things easier I worked from the directory the virtual hard disk was located.

Using the VBOXManage command I was able to resize the VM and convert the format without any changes to the users or data on the VM.  To accomplish this I used two commands the first being.

VBoxManage clonehd   --format VDI (the format of the drive was a paralles .hdd format)

Once Cloned, I then resized the drive using the following command

VBoxManage.exe modifyhd --resize $bytesize

The drive was a 40gb drive and I made the $bytesize value 80000 bytes or (80GB)

Everything since then has been running perfectly for that user.
Sources.

http://www.dedoimedo.com/computers/virtualbox-clone.html
http://superuser.com/questions/716649/how-to-change-fixed-size-vdi-with-modifyhd-command-in-windows