Storage Spaces Lost Communication or IO Error and Intel SSD Event ID 129

I had a major issue with our company's primary Hyper-V Server.  It was housing our primary AD controller, User Storage and Share, along with 2 other VMs.  The symptoms were major IO read/writing, reports of disconnects, and a loss of access to the AD, and user files.  According to our log files we were getting iANSMiniport, and Intel Nvmestor errors.

Here are some log file samples

According to the logs the system starting giving Event ID 129 warnings and this continued to happen every 10 seconds and affected our DHCP, DNS and user logons.  After a forced shutdown everything was fine according to the logs until later that morning when users started to have lag and login issues.



You can read more about Event ID 129 Here

We sent the server in for diagnostics and according to the Authorized Service Depot, they could not find anything out of date except for the bios.

Since this was a "Main Production Server" it was decided that we would move the virtual ad controller from the production server to a dev server to run until the failure of the drive could be determined so the organization could continue to run.

https://support.microsoft.com/en-ca/help/4052341/slow-performance-or-lost-communication-io-error-detached-or-no-redunda

After doing a second forced shutdown and boot up I was able to shut down the virtual machines running on the servers and did a full export of the data to an external drive.  This made sure we did not lose any data but it did inconvenience some users as the data had to be copied back from the dev server.

Unfortunately I didn't find the Microsoft article about Event ID 129 until about a week after our issue at that point I had  destroyed the storage spaces volume, but I did find it useful as I installed some additional software which seemed to help the server recognize the NVMe drives better.  For that you need to install not just the Intel SSD Toolbox, but also the SSD Data Center tool and the Data Center NVMe SSD Drivers.  The errors in the Event ID 129 before the Maintenance 8 release seemed to be a little too coincidental to what we were experiencing.

Symptoms

When this issue occurs, your cluster may experience any of the following symptoms:

• Slow workload performance
• Virtual disks in the cluster that have an Operational Status value of Detached or No Redundancy.
• Physical disks that report a status of Lost Communication or IO Error.

I haven't had a chance to verify the issues have been 100% corrected but from the testing I have done some major stress testing on the storage spaces setup using hyper-v to do mass exports of VM's to the storage array with no issues at all, HD tune and crystal disk mark have also shown the Storage Spaces Array to be in good shape.

https://www.intel.ca/content/www/ca/en/support/products/81000/memory-and-storage/data-center-ssds/intel-ssd-dc-p3600-series.html

https://downloadcenter.intel.com/download/28175/Intel-Solid-State-Drive-Toolbox?_ga=2.179938403.692315852.1538156647-366274783.1537922699

https://downloadcenter.intel.com/download/28035/Intel-SSD-Data-Center-Tool-Intel-SSD-DCT-?_ga=2.179938403.692315852.1538156647-366274783.1537922699

https://downloadcenter.intel.com/download/27517/Datacenter-NVMe-Microsoft-Windows-Drivers-for-Intel-SSDs?_ga=2.192759661.692315852.1538156647-366274783.1537922699

Enabling Trim on 3rd Party SSD on MacOS

Solid State Drives or SSD's as they are commonly referred to are now the easiest and cheapest way to bring new life to your old computer.  Many of the SSD's max out the bandwidth provided by the bus that your computer connects to the hard drive.  On MacOS though if your lucky enough to have an older mac then you can upgrade with any SSD you would like but what you don't know is that Apple doesn't enable TRIM by default even though most SSD's support TRIM. 

Why do you want Trim?  I'm not going to go into it in detail because Techspot has a great article you can read in detail here

Basically SSD's can not directly over write cells with new data, but must first be erase it at the block level.  This is called garbage collection.  The flash memory uses over provisioning used to improve performance and longevity of the SSD.   

The less space not consumed by your data becomes what is called dynamic over provisioning – because it changes as the amount of data stored changes.  When less data is stored by the user, the amount of dynamic over provisioning increases, further improving performance and longevity.

Here is what you need to do to enable Trim on your SSD on your Mac.

1. Close all programs except for Terminal if you

First you can check and see it if Trim is enabled.  You can do this 2 ways.

Terminal.

system_profiler SPSerialATADataType | grep 'TRIM'

You will get a response of TRIM Support: Yes or No

Or go under the Apple Menu -> about this Mac -> and run System Report

If you get a Yes you don't need to do anything trim is enabled.

If you get a No type the following command.

sudo trimforce enable

Then answer yes to the two questions and it will reboot.

Trim will be enabled after the reboot.  You can verify this by using the terminal command

system_profiler SPSerialATADataType | grep 'TRIM' or checking the system report.