Upgrading Windows 11 From 23H2 to 25H2 causes error 0x800f0991

 I had a heck of a time upgrading my Windows 11 Pro System from 23H2 to 25H2 and I really did not want to do a clean install.  I really shouldn't have had a problem but it was; the issue had to do with Hyper-V and Windows Subsystem for Linux.

My system is an AMD Ryzen 9 with all the bells and whistles, and it was constantly failing windows update where the error was 0x800f0991.

Running DISM & SFC did not help at all with the results displaying everything was fine with the system.  Booting from USB and trying an offline scan also proved to be useless.

I do a lot of work with Hyper-V and upgrading from 21h2 to 23H2 was painless which is why I was so puzzled why I couldn't go from 23H2 to 25H2.  I figured it had to be something with windows virtualization; it was the only thing it could be.  I have a number of 3rd party software installed a game or two but with the changes in Hyper-V, the network virtualization stack, it made the most sense that was going to be the best place to start.

So I backed up my VM's exporting them before removing Hyper-V.  I had to remove Data Center Bridging, Hyper-V, Virtual Machine Platform, Windows Hypervisor, and Windows Subsystem for Linux.

Before removing the virtualization components in windows it failed the 6 times when I tried to upgrade though a booted USB and PC Health Check.

Once that was done I was able to upgrade using PC Health Check.

Now that I'm updated to Windows 11 25H2; unfortunately Hyper-V will not work and causes a continuous boot loop.  However the upgrade to 25H2 from 23H2 worked very well so long as Hyper-V was removed.

How to upgrade your linux server

How do you do an in place upgrade on a older Linux Server and why would you want to.  Well why you would want to is obvious, security patches, and giving longer life to the server on the newer software; that is a good reason.  With browsers being constantly upgraded, along with security issues being found every other day it seems, staying up to date is just a good idea.

So how do we stay up to date?

Typically, if you have been staying up to date you will eventually get prompted to do an upgrade to an newer version of linux at some point.  All the following commands will need to be run as root (sudo).

sudo apt-get update

Downloads package information from all configured sources.  Running this command keeps your package information up-to-date

sudo apt-get upgrade

This command only upgrades.  The upgrades are only upgrades available to the platform, as defined in /etc/apt/sources.list or in /etc/apt/sources.list.d/.  This upgrades your packages, not your OS and it will not remove packages.

sudo apt-get dist-upgrade, 

Intelligently installs or removes packages as needed.  This command has a smart conflict resolution system.  The system attempts to upgrade the most important packages, at the expense of those deemed less important.  This command will also delete software if it is required to complete the upgrade process.

sudo apt-get autoremove

Removes packages that were automatically installed because another package required them but, if they are no longer needed this command removes them.

sudo apt-get autoclean

Clears the local repository of retrieved package files, but it only removes files that can no longer be downloaded and are virtually useless. It helps to keep your cache from growing too large

sudo do-release-upgrade

The do-release-upgrade command will upgrade the system from one release to another. This is the command you want if you want to upgrade from Ubuntu 18.04 to 20.04.  NOTE:  The system must be fully upgraded to run. use the apt-get upgrade, followed by sudo apt-get dist-upgrade commands before running the do-release-upgrade command.

Sources

https://www.digitalocean.com/community/questions/updating-ubuntu-16-04-to-20-04
https://www.techrepublic.com/article/how-to-tell-the-difference-between-apt-get-upgrade-apt-get-dist-upgrade-and-do-release-upgrade/

Performing an upgrade for Horizon Web Services

Upgrading Horizon Web Services is unfortunately more/less a full reinstall of the Apache tomcat server and software which you can read my post about here.  This means you will have to re-apply any TLS certificate you have setup with your Horizon Web Services (which is a pain but do able).

On my previous post we installed Horizon Web Services 201709 (you can see the full version number in the picture) and now we are going to upgrade to version 6.0.2R2 from 201709

Now this is a few versions behind and since an upgrade to Horizon 7.5.5 from 7.5.4.1 an app produced by sirsidynix called mobilecirc (mobile staff) available for both android and ios broke and will not allow you to create/register new users.

Before you begin your upgrade you will want to backup your classes directory in the webapps  directory webapps\"YourHorizonWebServicesInstanceName"\WEB-INF\classes

It is safest to stop your service before doing the backup.  In this case "Apache Tomcat HWS"

The web-inf\classes directory is in the web directory you setup when you installed tomcat typically "hzws" if you use the defaults.  I just backed up the whole web directory as a zip file.

Then you should uninstall the version of Web services your using thought the program and features in the control panel.  Since 201709 is running on tomcat 7, and version 6.0.2R2 is on tomcat 8, this will also upgrade our tomcat install.

Once uninstalled reboot the server, and start the installation of Web Services you want to install in this case we are doing 6.0.2R2.

Now for the java vm I am selecting the latest version of java for this.

Then the rest of the install works just like in my post to install Horizon Web Services.  Once the install is done don't forget to go to Services and change the startup type to automatic.

Once the service is started, you will have to fix the TLS certificate.  The installer resets all customized tomcat configurations.

You will have to edit the server.xml file located in Webservices\conf\server.xml and update your keystorefile,  keystorepass, and server values 

Restart Apache tomcat and the TLS Certificate should be working properly.

and our web services is now version 6.0.2.201291521

Upgrading Server 2016 to 2019

Just before the end of February, my workstation's RAID 10 decided that it would be fun if I lost 2 drives on the same mirror.  So thankfully I had my OS on an SSD which was running but I did end up losing all my data but none of it was essential and it is pretty hard to backup 8tb.

So now that I'm more/less starting from scratch I thought it would be a great time to try an in-place upgrade of server 2016 to 2019!  HOW EXCITING!  I do have some software I will running on my C: including SQL server so it will be interesting to see what still works after the upgrade.  Now that I have some new drives and I can now setup a backup solution for some of my dev work!

Following the guide from microsoft.  https://docs.microsoft.com/en-us/windows-server/upgrade/upgrade-2016-to-2019

I saved the info they requested to a network drive and a flash drive.

open the cmd prompt and cd to c:\Windows\system32, and then type systeminfo.exe.

Results from running systeminfo.exe.

Host Name:                 "MY WORKSTATION"
OS Name:                   Microsoft Windows Server 2016 Standard
OS Version:                10.0.14393 N/A Build 14393
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Server
OS Build Type:             Multiprocessor Free
Registered Owner:          Windows User
Registered Organization:
Product ID:                xxxxx-xxxxx-xxxxx-xxxxx
Original Install Date:     2018-06-18, 6:36:57 AM
System Boot Time:          2020-02-26, 9:58:13 AM
System Manufacturer:       COMPUTER INC.
System Model:              WORKSTATION
System Type:               x64-based PC
Processor(s):              2 Processor(s) Installed.
                           [01]:  ~1200 Mhz
                           [02]:  ~1200 Mhz
BIOS Version:              American Megatrends Inc.
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume4
System Locale:             en-ca;English (Canada)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC-07:00) Mountain Time (US & Canada)
Total Physical Memory:     130,958 MB
Available Physical Memory: 115,100 MB
Virtual Memory: Max Size:  150,414 MB
Virtual Memory: Available: 132,449 MB
Virtual Memory: In Use:    17,965 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    "MY DOMAIN"
Logon Server:              \\LoginServer
Hotfix(s):                 14 Hotfix(s) Installed.
                           [01]: KB3192137
                           [02]: KB4091664
                           [03]: KB4093137
                           [04]: KB4132216
                           [05]: KB4465659
                           [06]: KB4485447
                           [07]: KB4498947
                           [08]: KB4503537
                           [09]: KB4509091
                           [10]: KB4512574
                           [11]: KB4520724
                           [12]: KB4521858
                           [13]: KB4524244
                           [14]: KB4537764

NETWORK INFORMATION Gets listed with all details about your networks connected, such as VLAN Tags, IPs, etc

Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.

Run and save the information from ipconfig /all

RESULTS FROM IPCONFIG

Windows IP Configuration

   Host Name . . . . . . . . . . . . : "My Workstation"
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : Domain1
                                       Domain2

All you networking information gets listed here starting with your default NETWORK and it lists all vlan tags, mac addresses, much as you would expect from the
"ipconfig /all" command.

Record 2 registry key entries.   BuildLabEx (version) and EditionID (edition)

I did a Registry Keys Export for a Backup and a screen shot for reference

I saved the upgrade info to a text file on the flash drive along with a screen shot of the registry and an export of the current version key.

I create upgrade media using rufus, then ran the installer as an administrator

NOTE:  You will need your server key when you upgrade

First it will check for updates, if there are none it will move on

then it will check and see if your ready to install, checking for enough disk space etc.  Here is where you will be asked for your product key

Then you will need to select your version of server in this case I'm picking (Desktop Experience)

Accept the notices and license terms

You will be asked what you want to keep, I'm going to just keep everything as is since I need to redo everything anyway.

Then it will do a check to see if there is any kind of issue in the way of the upgrade.

 Then it will ask you to install windows

I started this process at 2:30 pm and to get to 75% took about an hour and a half so it really does take it's sweet time.  So I would plan for a minimum outage of about 3 hours.  The system seemed to have taken the upgrade just fine however I was unable to connect to the server using RDP and I lost access to some unauthenticated network shares. Though I really should setup some security on this network share it is something I am not able to arbitrarily do at the moment as others require access to this share. 

So to fix this problem; you need to edit a registry entry "AllowInsecureGuestAuth"

located here (you can copy and paste the path into regedit):

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters

The update change the REG_DWORD to 0

AllowInsecureGuestAuth Value after upgrade

You need to change the value to 1 if you want to be able to access unauthenticated shares.

Change the REG_DWORD to 1 to allow access to unauthenticated shares

you can read the reference link below with the details.

https://support.microsoft.com/en-ca/help/4046019/guest-access-in-smb2-disabled-by-default-in-windows-10-and-windows-ser

Microsoft is defaulting on the side of security which I completely applaud them for; however I do have to re-enable unauthenticated access.  

It was also shortly after fixing my network share issue I noticed some issues with most of my virtual networks with most of my networks showing an error.

Most networks no longer work after upgrade.

After trying serveral times to fix the issue with updating drivers, rebooting, system scans, etc I gave up and decided that my in-place upgrade wasn't an in-place upgrade.  So I formatted my OS drive and did a clean install.

After doing a clean install of Server 2019, setting up the dual port nic into a teamed interface I still lost network connectivity after installing the Hyper-V roll.  This happened after I reinstalled another 2 times.  What I didn't realize even looking though forums this issue was hard to find a resolution to.  The only solution that worked for me was to follow this process I found on Microsoft Technet

https://social.technet.microsoft.com/Forums/en-US/5906152c-1f22-449a-b223-15c4a3e5eb36/2019-hyperv-virtual-switch-error?forum=winserverhyperv

1. I had to remove Hyper-V role
2. Restart
3. Run  netcfg -d in a privileged powershell console
4. I made sure I setup all my teamed networks after this
5. Restart 
6. Install Hyper-v role

This was what fixed my network issue in server 2019.  After running the netcfg -d command, and re-installing the hyper-v roll my virtual switches came back without an issue.  This network issue after installing the Hyper-V roll also occurred when I had set up the server on a regular network interface (NOT TEAMED).  This would not be good if I had to keep the virtual machines running but at least I would be able to export them and re-import them.  This is compelling me to try and do a mockup in place upgrade; even though this one went sideways I think I can come up with a process to do it.

Upgrading Server 2012 to 2012R2 and Hyper-V Import Types

Just recently I had to upgrade a server from Windows Server 2012 to 2012R2, due to some driver changes in the intel 82576 adapter that prevents the addition and removal of VLAN tags in Windows Server and that 2012 mainstream support ended in May 2017.  Server 2012R2 doesn't really extend mainstream support by much (till October 2018), but most of our infrastructure is built on in so migration to 2012R2 is easier for that reason alone.  

I managed to do the whole migration over 2 days and most of that was due to an issue with adding SSD's to the system (Turns out the RAID controller doesn't support them properly) and a lack of documentation on the VM guests and on the Hyper-V host. 

There are 3 different ways to do an import of a Hyper-V Guest listed below; for simplicity I just used Register in-place for this procedure.

Hyper-V offers three import types:

• Register in-place – This type assumes export files are in the location where you'll store and run the virtual machine. The imported virtual machine has the same ID as it did at the time of export. Because of this, if the virtual machine is already registered with Hyper-V, it needs to be deleted before the import works. When the import has completed, the export files become the running state files and can't be removed.
• Restore the virtual machine – Restore the virtual machine to a location you choose, or use the default to Hyper-V. This import type creates a copy of the exported files and moves them to the selected location. When imported, the virtual machine has the same ID as it did at the time of export. Because of this, if the virtual machine is already running in Hyper-V, it needs to be deleted before the import can be completed. When the import has completed, the exported files remain intact and can be removed or imported again.
• Copy the virtual machine – This is similar to the Restore type in that you select a location for the files. The difference is that the imported virtual machine has a new unique ID, which means you can import the virtual machine to the same host multiple times.

For more detailed information please reference https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/deploy/export-and-import-virtual-machines 

For simplicity I've only used the Register in-place option.


STAGE 1: Migrate VM's on 2012 Server

My workstation is a Dual socket 6 core Xeon system (12 Physical cores 24 Cores Total) running 2012R2 for developing and testing and staging VMs for deployment.  In this case my Workstation was becoming the temporary home for hosting the guest VM's from the old system was being upgraded.  So with the exception of 2 VM guests, the VM's could be moved during regular office hours.

A list of the services running on the VM Host

• EmailRelay
• Imaging Server/WOL Server
• Minecraft Server
• Website X 2
• Print Release Server
• VRNetwork DHCP Server (New and not yet in production)

When doing the migration to the Temporary Hyper-V host the process was very simple.

1. Shutdown the VM Guest and Export the VM
2. Copy exported VM to new temporary location on the server
3. Do a Register In-Place Import of the VM

I had created a SMB share on the temporary host for the old host; shutdown and exported the guest server on 2012 and copied it to the 2012R2 host.  The transfer for the VM's took place during off hours, and since most of the services can be down during off hours no one really noticed the outage.  The whole process took a few hours, the websites which we didn't want to really bring down for any length of time were down for about 15 minutes while the VM's were copied.

STAGE 2: Register in-place import

As the VM's finished being copied over I did a register in-place import which doesn't move the VM's into the default location you specify when you had setup Hyper-V.  So If I did an export and called it websites I can copy the entire folder and doing the in-place import still keeps everything in the folder as shown below.

Exported VM, Doing an in-place import keeps everything together.

STAGE 3 - Restore VM's to the Server:

Once the old server was upgraded to 2012R2 and the NIC was upgraded to an Intel i350.  I restored the VM's to the system and decided to continue to use the register in-place import method, but I maybe should have gone with the Restore option for the final import (makes it easier if you want to just backup the VHDX files).  If I had backed up the VM's to a file share all the host had access to I might have used the Copy option.

Notes & Issues:

A couple of issues I had with this migration to the temporary VM Host was due to the lack of documentation.  I discovered that a couple of VM's had broke durning import because the "migrate to a machine with a different processor" wasn't checked, but also that they had Dynamic MAC Addresses.  This broke the IP Address they received from the DHCP as they had a different MAC Address.  I had to go though and double check each VM guest and verify all the MAC Addresses were set to static.  The other issue revolved around VLAN Tags.  There were VLAN tags assigned to the host in the switch that were not correct or improperly configured.  All in all there are only 5 VLANS that were required:

VLANs:

• NAVTIVE VLAN 0X
• VLAN 1X
• VLAN 2X
• VLAN 3X
• VLAN 4X - NOT REQUIRED FOR HYPER-V GUEST (HOST ONLY)

I made Hyper-V switches for VLANS 0X - 3X and left VLAN 4X just on the host and is used for doing VSS backups of the VHDX files.

All in all I am relatively happy with how the upgrade went, I am currently working on documentation for our Hyper-V Servers with information required to rebuild them in less then a day.

Here is an example of my VM Guest Documentation (work in progress)

https://docs.google.com/spreadsheets/d/1hJu2lJ_ahpJ_2JYY2J8r0mBozO3LhNPm9G_qtpYIjf0/edit?usp=sharing

Here is an example of my Hypervisor Host Documentation (work in progress)

https://docs.google.com/spreadsheets/d/1RusCrWB_538DWBPV96r_RrsscxaG2uk5ua70sL6yDuU/edit#gid=0