Upgrading Server 2016 to 2019

Just before the end of February, my workstation's RAID 10 decided that it would be fun if I lost 2 drives on the same mirror.  So thankfully I had my OS on an SSD which was running but I did end up losing all my data but none of it was essential and it is pretty hard to backup 8tb.

So now that I'm more/less starting from scratch I thought it would be a great time to try an in-place upgrade of server 2016 to 2019!  HOW EXCITING!  I do have some software I will running on my C: including SQL server so it will be interesting to see what still works after the upgrade.  Now that I have some new drives and I can now setup a backup solution for some of my dev work!

Following the guide from microsoft.  https://docs.microsoft.com/en-us/windows-server/upgrade/upgrade-2016-to-2019

I saved the info they requested to a network drive and a flash drive.

open the cmd prompt and cd to c:\Windows\system32, and then type systeminfo.exe.

Results from running systeminfo.exe.

Host Name:                 "MY WORKSTATION"
OS Name:                   Microsoft Windows Server 2016 Standard
OS Version:                10.0.14393 N/A Build 14393
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Server
OS Build Type:             Multiprocessor Free
Registered Owner:          Windows User
Registered Organization:
Product ID:                xxxxx-xxxxx-xxxxx-xxxxx
Original Install Date:     2018-06-18, 6:36:57 AM
System Boot Time:          2020-02-26, 9:58:13 AM
System Manufacturer:       COMPUTER INC.
System Model:              WORKSTATION
System Type:               x64-based PC
Processor(s):              2 Processor(s) Installed.
                           [01]:  ~1200 Mhz
                           [02]:  ~1200 Mhz
BIOS Version:              American Megatrends Inc.
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume4
System Locale:             en-ca;English (Canada)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC-07:00) Mountain Time (US & Canada)
Total Physical Memory:     130,958 MB
Available Physical Memory: 115,100 MB
Virtual Memory: Max Size:  150,414 MB
Virtual Memory: Available: 132,449 MB
Virtual Memory: In Use:    17,965 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    "MY DOMAIN"
Logon Server:              \\LoginServer
Hotfix(s):                 14 Hotfix(s) Installed.
                           [01]: KB3192137
                           [02]: KB4091664
                           [03]: KB4093137
                           [04]: KB4132216
                           [05]: KB4465659
                           [06]: KB4485447
                           [07]: KB4498947
                           [08]: KB4503537
                           [09]: KB4509091
                           [10]: KB4512574
                           [11]: KB4520724
                           [12]: KB4521858
                           [13]: KB4524244
                           [14]: KB4537764

NETWORK INFORMATION Gets listed with all details about your networks connected, such as VLAN Tags, IPs, etc

Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.

Run and save the information from ipconfig /all

RESULTS FROM IPCONFIG

Windows IP Configuration

   Host Name . . . . . . . . . . . . : "My Workstation"
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : Domain1
                                       Domain2

All you networking information gets listed here starting with your default NETWORK and it lists all vlan tags, mac addresses, much as you would expect from the
"ipconfig /all" command.

Record 2 registry key entries.   BuildLabEx (version) and EditionID (edition)

I did a Registry Keys Export for a Backup and a screen shot for reference

I saved the upgrade info to a text file on the flash drive along with a screen shot of the registry and an export of the current version key.

I create upgrade media using rufus, then ran the installer as an administrator

NOTE:  You will need your server key when you upgrade

First it will check for updates, if there are none it will move on

then it will check and see if your ready to install, checking for enough disk space etc.  Here is where you will be asked for your product key

Then you will need to select your version of server in this case I'm picking (Desktop Experience)

Accept the notices and license terms

You will be asked what you want to keep, I'm going to just keep everything as is since I need to redo everything anyway.

Then it will do a check to see if there is any kind of issue in the way of the upgrade.

 Then it will ask you to install windows

I started this process at 2:30 pm and to get to 75% took about an hour and a half so it really does take it's sweet time.  So I would plan for a minimum outage of about 3 hours.  The system seemed to have taken the upgrade just fine however I was unable to connect to the server using RDP and I lost access to some unauthenticated network shares. Though I really should setup some security on this network share it is something I am not able to arbitrarily do at the moment as others require access to this share. 

So to fix this problem; you need to edit a registry entry "AllowInsecureGuestAuth"

located here (you can copy and paste the path into regedit):

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters

The update change the REG_DWORD to 0

AllowInsecureGuestAuth Value after upgrade

You need to change the value to 1 if you want to be able to access unauthenticated shares.

Change the REG_DWORD to 1 to allow access to unauthenticated shares

you can read the reference link below with the details.

https://support.microsoft.com/en-ca/help/4046019/guest-access-in-smb2-disabled-by-default-in-windows-10-and-windows-ser

Microsoft is defaulting on the side of security which I completely applaud them for; however I do have to re-enable unauthenticated access.  

It was also shortly after fixing my network share issue I noticed some issues with most of my virtual networks with most of my networks showing an error.

Most networks no longer work after upgrade.

After trying serveral times to fix the issue with updating drivers, rebooting, system scans, etc I gave up and decided that my in-place upgrade wasn't an in-place upgrade.  So I formatted my OS drive and did a clean install.

After doing a clean install of Server 2019, setting up the dual port nic into a teamed interface I still lost network connectivity after installing the Hyper-V roll.  This happened after I reinstalled another 2 times.  What I didn't realize even looking though forums this issue was hard to find a resolution to.  The only solution that worked for me was to follow this process I found on Microsoft Technet

https://social.technet.microsoft.com/Forums/en-US/5906152c-1f22-449a-b223-15c4a3e5eb36/2019-hyperv-virtual-switch-error?forum=winserverhyperv

1. I had to remove Hyper-V role
2. Restart
3. Run  netcfg -d in a privileged powershell console
4. I made sure I setup all my teamed networks after this
5. Restart 
6. Install Hyper-v role

This was what fixed my network issue in server 2019.  After running the netcfg -d command, and re-installing the hyper-v roll my virtual switches came back without an issue.  This network issue after installing the Hyper-V roll also occurred when I had set up the server on a regular network interface (NOT TEAMED).  This would not be good if I had to keep the virtual machines running but at least I would be able to export them and re-import them.  This is compelling me to try and do a mockup in place upgrade; even though this one went sideways I think I can come up with a process to do it.

Performing a Rolling Upgrade of Windows Server 2012R2 to 2016

In January I performed a rolling cluster upgrade on my organizations Microsoft Hyper-V cluster going from server 2012R2 to server 2016.  I highly recommend reading the following resources as they were a great reference for me.

https://docs.microsoft.com/en-us/windows-server/failover-clustering/cluster-operating-system-rolling-upgrade
https://redmondmag.com/articles/2016/12/15/clustered-hyperv-deployment.aspx
https://redmondmag.com/articles/2016/12/19/hyperv-deployment-2.aspx

There were a couple of caveats with the upgrade as outlined below.

• Move from seven clustered nodes to a two node failover
• Consolidate 19 VMs to 12
• Upgrade NAS and Cluster Servers to 10 Gig networking
• virtualize all AD controllers

What is being kept:

• 2 X FreeNAS storage Servers

What is being removed:

• 6 X AMD Piledriver 15H Microcloud Server

What was being upgraded and kept:

• 2 X AMD Bulldozer 1U Servers

Upgrading the core to 10 gig

The migration to the 2X AMD bulldozer 1U servers was due to a lack of availability of getting new hardware ordered and a urgent need to remove the Microcloud system.  I felt it was a good idea to start with the storage as I had 4 servers I needed to upgrade with 10 Gig and with Freenas it is pretty straight forward.  We went with Intel X540 T2 cards because they were within our project budget and with having purchased Meraki MS350 24X which have 10 Gig, I though it was best to make use of them.  Unfortunately they don't support iWARP/RDMA but we were running 19 virtual servers off 2 freenas boxes on dual gigabit LACP LAGG.  Any limit we would reach I would think should be caused by disk (spinning rust) as they are not high speed SAS or NVMe but Enterprise SATA drives.

The upgrade on the FreeNAS servers went great all that was really required was having good documentation for any IP addresses on the storage and that the switches had all the required networks.  On the cluster management part it was a matter of shutting down the VM's, off-lining the cluster storage (maintenance mode), popping in the new network cards and configuring the networks to what they were everything came back without a hitch.  For verification we ran with the 10 gig in the systems for a month with no issues.  We then started with the Rolling upgrade

Step 1. - Run a cluster validation report.

It's hard to see where you're going without knowing where you've been and where you are.  This is why I recommend running a cluster validation report before you start anything.  This way you know what errors you were getting before hand and you will be able to know if something happened because of the upgrade or because there was a mistake or that you fixed something that might have been broken.  As shown below is a validation report before I started the upgrade from 2012R2 to 2016

Server 2012R2 Cluster Validation Report

As you can see I have several warnings on the validation report.  The errors included some network issues of dropping packets and other things such as the Cluster Aware Update Service being configured to be run in a separate monitor as shown below.

With that report being done, we can now start our Rolling upgrade.

Rolling Upgrade

I had to come up with a plan of more or less what I was going to do for the rolling upgrade.  This is it in a nutshell.



<!--IMPORTANT NOTE--!>


When the first Windows Server 2016 node joins the cluster, the cluster enters "Mixed-OS" mode, and the cluster core resources are moved to the Windows Server 2016 node. A "Mixed-OS" mode cluster is a fully functional cluster where the new nodes run in a compatibility mode with the old nodes. "Mixed-OS" mode is a transitory mode for the cluster. It is not intended to be permanent and customers are expected to update all nodes of their cluster within four weeks.


Cluster Upgrade Process

1. Disable Cluster Aware Updates (CAU), verify if CAU is currently running by using the Cluster-Aware Updating UI, or the Get-CauRun. Stop CAU using the Disable-CauClusterRole to prevent any nodes from being paused and drained by CAU during the Cluster OS Rolling Upgrade process. This really depends on how many resources you have to keep the cluster going in this case we have enough resources so it is not required.
2. Backup Cluster Database (requires third party software)
3. Remove/Drain Rolls from from cluster being upgraded for use in 2 node cluster Backup (Full Export) of Workload DATA (VMS)
4. After moving the VMs off of server, upgrade RAM and Reconfigure disks 1 SSD OS MIRROR and 1 HDD MIRROR for DATA reinstall with server 2016
5. Upgrade and Configure networking
6. Add Hyper-V and cluster rolls
7. Add node to the cluster 

Verify cluster rolls and nodes in hybird 2012r2 and 2016 setup. Once verified evict the other 6 2012R2 nodes ensuring all rolls are transferred to new 2016 nodes and verify the cluster is operating properly.   Then upgrade the Cluster Functionality level

1. Re-Enable CAU, restart it using the CAU UI or use the Enable-CauClusterRole
2. Open Powershell -> Run the Update-ClusterFunctionalLevel cmdlet - no errors should be returned. Fix any errors.
3. Upgrade the VM’s Hyper-V VM Configuration (Version)

Upgrade the Hyper-V Storage-Pool. Storage pools can be upgraded using the powershell command Update-StoragePool which is an online operation.

Cluster Changes

It is always easier and safer to start new or fresh, however this is not possible at this time.  When we go to upgrade the cluster infrastructure we will be doing a Mixed OS cluster until we remove all 2012R2 servers and upgrade the cluster functionality level. When the first Windows Server 2016 node joins the cluster, the cluster enters "Mixed-OS" mode, and the cluster core resources are moved to the Windows Server 2016 node. A "Mixed-OS" mode cluster is a fully functional cluster where the new nodes run in a compatibility mode with the old nodes. "Mixed-OS" mode is a transitory mode for the cluster. It is not intended to be permanent and customers are expected to update all nodes of their cluster within four weeks.

The two nodes I am upgrading to server 2016 from 2012R2 are almost identical, with two major differences. 2 X 2.8 GHZ quad core processors vs 2 x 3.2 GHz quadcore processors and one requires an old adaptec raid driver vs a dothill raid controller. Both machines have 80GB of DDR3 ECC RAM.

TIPS

• All Nodes need to have the exact same software installed.
• All Nodes need to have the exact same Virtual Switches and Named the same.
• All Nodes Should Have RDS Enabled
• Virtual Network Adapter TEAM: AdapterName ( I typically put the Cluster/Client communication on the default VLAN)
• Cluster Name: $CLUSTERNAME

Cluster Upgrade Process

Step 1 . Disable/Stop Cluster Aware Updates

• If you are running Cluster Aware Updates (CAU), verify if CAU is currently running by using the Cluster-Aware Updating UI, or the Get-CauRun. Stop CAU using the Disable-CauClusterRole to prevent any nodes from being paused and drained by CAU during the Cluster OS Rolling Upgrade process. This really depends on how many resources you have to keep the cluster going in this case we have enough resources so it is not required.
• Backup Cluster Database (use Veeam, Altaro, Windows Server Backup)
  Backing Up Failover Cluster Configuration
  https://social.technet.microsoft.com/Forums/en-US/5bdd1536-189a-42c6-a084-1de73848887a/backup-a-failover-cluster-configuration?forum=winserverClustering
• Drain the rolls from cluster node being updated. Ensure all rolls from node have been removed and transferred then evict from cluster
• Upgrade Node to Server 2016. I made sure that I had as many of the rolls and features that were on the other nodes re-setup on the new 2016 node.
• Rolls Required
• File and Storage Services everything but work folders
• Hyper-V

• Features Required
  .net 3.5
  .net 4.5
  Client for NFS
  Data Center Bridging
  Enhanced Storage
  Failover Clustering
  Ink and Handwriting services
  media foundation
  multipath io
  Remote Admin Tools
  Hyper-V management
  File services
  Failover cluster tools
  SMB1
  SMB Bandwidth limit
  User Interfaces
  Windows Internal Database
  Windows Server Backup
  Windows server migration tools
  Windows Standards-based storage management
  WoW64 Support
• Setup network. (Intel network setup) Cluster only allowed on cluster network. To do this we are setting up NIC Teaming to have easy access to VLAN Tagging.

Once done create the LAGG adding the active NIC to the team.

Repeat the steps for as many times as you need until you have all the VLANS you need for the server.

• Add the Hyper-V Switches to the network. I setup only one network for VM and Cluster Traffic. It is recommended that you have more then one network for doing this. Make sure you match the proper Microsoft Multiplexor adapter with the proper vlan you have setup on your host interface, otherwise your VM's won't be able to communicate with anything.

VM and Host Network for cluster communication

VM network.  No Host/Cluster Communication Permitted.

• Re-Connect the iSCSI targets on the host node after all networking is setup and configured.

  iSCSI Initiator

  IP: PORT -SSD

  IP: PORT -HDD

  IP: PORT - Quarcom

  IP: PORT - SSD

  IP: PORT - HDD

  IP: PORT - Backup Quarcom

• Rejoin cluster

Verify cluster rolls and nodes in hybird 2012r2 and 2016 setup.  Once verified evict the other 6 2012R2 nodes and verify the cluster is operating properly. Transfer Rolls such as the Cluster Core Resources and ensure are not running on nodes that you are decommissioning.

1. Re-Enable CAU, restart it using the CAU UI or use the Enable-CauClusterRole

2. upgrade the Cluster Functionality level

3. Open Powershell -> Run the Update-ClusterFunctionalLevel cmdlet - no errors should be returned. Fix any errors.

4. Upgrade the VM’s Hyper-V VM Configuration (Version)

Upgrade the Hyper-V Storage-Pool. Storage pools can be upgraded using the powershell command Update-StoragePool which is an online operation.

Verify Cluster Operating Level
Get-Cluster | Select ClusterFunctionalLevel (Server 2016 and above only)
https://serverfault.com/questions/848529/hyper-v-2012-r2-cluster-functional-level-missing

Run cluster verification and fix any issues that crop up.

Some tips from my Server 2012R2 to 2016 Upgrade:

• I had to limit the number of networks the cluster host runs off of due to some DNS resolution issues that worked using server 2012R2 but not under 2016.  I had to remove the cluster host from using these networks due to DNS Error 1966 showing up and under server 2016 causing cluster communication problems.  To do this I disabled the communication using the Hyper-V switch manager with the setting shown below.
  
  
• I did have to remove and re-add the cluster auto-update roll. 
• Cleanup the AD and remove any cluster nodes that did not get automatically removed.

Other then that I had a really successful cluster update to server 2016 and everything has been working well since.