Managing VM Storage On A Hyper-V Cluster

Managing storage on a cluster can be pretty dry but it is a challenge to keep your data files from becoming a sprawling mess much like my desktop.  File duplication from copies, backups, even testing VMs can create a confusing mess.  When it comes to your virtual machines and their storage you should make sure that you do everything you can to keep everything together; otherwise your not only asking for trouble. Microsoft has some great tools for managing your virtual machine storage and you should use them otherwise you could easily end up with duplicated virtual disks which could be outdated and can cause you all sorts of issues.

My Messy Desktop

In my case I want to move several VM's off of a storage server called STORAGE1 and move them to STORAGE2 as shown below.  Now before I go head and move these because I am cautious and paranoid; I make a backup of said VM, in this case let's say it's VM1.  I run a export out of the cluster node to which the VM is on (or you can move it to the node your on) and run a full export to a usb drive I have connected.  

VM File Management

Wanting to keep your File System Organization nice and clean, I highly recommend that you pre-setup your folders where you want to move your VM, so on STORAGE2 I would create a folder called VM2 and inside that folder I created a folder called Virtual Hard Disk.  The Cluster Server Manager will move your files but it won't create the same folders, the cluster storage manager will just put everything in a single directory, and I myself prefer to keep the default structure as much as possible.  So on Storage2 my directory pathing is setup like this.

STORAGE2 -> VM2 -> Virtual Hard Disk

As you can see below I have the Failover Cluster Manager open and I have the roles menu selected.  Here is where you can modify your Clustered Role Storage

Right click on the selected VM and Select Virtual Machine Storage as shown below.

You will get a dialog box showing you where the virtual machine storage resides.  Then you drag and drop the virtual machine storage elements into the folders on the left as shown below.

Once you move all the items to the new storage locations press start.  This will take some time to process which also depends on the VM and the size of the VM disk etc.  Obviously the larger the VM is the longer the process will take.  Unfortunately there isn't any progress bar to tell how where the progress is on the moving of the VM storage, but the text on in the information section will disappear when the process is complete.  You can run this while the VM is live, though I do recommend shutting down the VM then move it; but that is just me being extra cautious.

Once your VM has finished moving the folders on the storage server where the VM was will still be there but will be empty.  I do recommend tiding up after and removing these unused folders as shown below.  This really beats the alternative of removing the role of the VM on the cluster and moving your backed up file to where you want it on the storage and resetting up the role.

Fix For Kaspersky Security Center 11 Error Opening localhost:13291

 

Kaspersky Security Center Error Failed to establish connection with the
remote device (location: http://localhost:13291) connection has failed

Kaspersky Security Center 11 Error

SQL express is great if you have to run or install programs such as Kaspersky Security Center, however if you don't turn on your Database maintenance scripts you can have a problem with your database getting to large and it will run out of space.  This will cause security center to stop working which is a problem.  As you can see the SQL database only has .64 MB of space available, which is why Kaspersky Security Center crashes and won't launch

Below is how I came to fix my issue with Kaspersky Security Center when my SQL express database got to full.  If your looking for the really quick fix, using SQL management Studio run 

TRUNCATE TABLE dbo.wus_upd_metadata

on your Kaspersky Security Center Database mine is called KSC then shrink your database in sql management studio.  This will get your Kaspersky management studio back up and working, however my process to getting to the fix is below so enjoy!

I started the troubleshooting process by making a copy of the av server, in this case since it is a VM, I did an export and set it up on a different host and setup the virtual switch to be an Internal switch so I can transfer files using the VM host.

With the database being SQL Express and it being completely full I had to upgrade the edition of SQL server to a trial or developer database.  You will also need SQL Management Studio.  Since this is a copy of the production system, I upgraded the database to sql 2016 dev edition from 2014 express edition to run a shrink database script.  Since I am using SQL Server Express 2014 we will be upgrading to SQL Server 2016 developer edition to get Kaspersky Security Center to load so we can shrink the database.  You will need a myvisualstudio account to download the iso, in my case for SQL Server 2016

Run the installer and select Installation -> Upgrade from a previous version of SQL Server

Select The edition, I am changing from express to developer edition

Accept the terms of use

If your offline, you will get an error about not being able to contact Windows Update Service.  You can continue

Then you will get a list of checks for upgrade rules and warnings/errors if there are any

Select your sql server instance you want to upgrade

Select the features you want to upgrade

Select your instance configuration

Then upgrade your sql server

Now that the system is upgraded we need to reboot to enable the changes

After our reboot Kaspersky Security Center will now run because the dev version give you more space then express so I used this to do some checking on the database and to see if Kaspersky could shrink the database enough to go back to express.

Now we can run the script to shrink the database.  Under Administration Server go to tasks and database maintenance.  You can view this article on Kaspersky's support website

https://support.kaspersky.com/ksc/sp3/en-us/100999.htm

To enable shrinkage on the database right click on Database Maintenance -> go to properties -> Settings and make sure shrink database is checked.

then hit apply and ok.

Now we will run a maintenance script which was provided by Kaspersky, which can be downloaded from my server here.  However if you do want to get a hold of this I recommend opening a ticket with Kaspersky at support.kaspersky.com

In an administrator cmd prompt

go to the extracted folder path in commnad line, I put the script in C:\Users\Administrator\Desktop

run start.cmd

After running the script our database is now down to 9925.44 Mb with 2.66 MB of available space not much but it should be able to run on our system now.  However I do think that this problem will come up again so I needed to find a way to prune some of the data in the database.

The next set of processes you will need Microsoft SQL Server Management Tools.

What I did was run a sql query to find our largest tables.  Here is a great script from https://dataedo.com/kb/query/sql-server/list-10-largest-tables which I used to find the largest table.

select top 10 schema_name(tab.schema_id) + '.' + tab.name as [table], cast(sum(spc.used_pages * 8)/1024.00 as numeric(36, 2)) as used_mb, cast(sum(spc.total_pages * 8)/1024.00 as numeric(36, 2)) as allocated_mb from sys.tables tab join sys.indexes ind on tab.object_id = ind.object_id join sys.partitions part on ind.object_id = part.object_id and ind.index_id = part.index_id join sys.allocation_units spc on part.partition_id = spc.container_id group by schema_name(tab.schema_id) + '.' + tab.name order by sum(spc.used_pages) desc;

We see that the table dbo.wus_upd_metadata is our largest table with 4 gig of data being stored there.  Now I don't work for Kaspersky but I would guess that wus has probably something to do with Windows Update Services and I would bet it isn't critical to running Kaspersky Security Center

Table Name	used_mb	allocated_mb
dbo.wus_upd_metadata	4270.23	4271.41
dbo.wus_upd_prerequisite	1907.66	1907.79
dbo.ev_event	1399.92	1402.85
dbo.wus_upd_driver	1178.42	1178.52
dbo.wus_upd_revision	489.21	489.78
dbo.wus_upd_file	123.76	124.68
dbo.wus_upd_product	118.73	119.44
dbo.wus_upd_digest	73.41	73.72
dbo.wusx_upd_revision	41.91	43.09
dbo.wus_upd_superseded	41.28	42.24

Then I ran a new query TRUNCATE TABLE dbo.wus_upd_metadata, then I shrank the database using sql management studio.

After that we can see our database is now about 6Gb in size and to reclaim the space by right clicking on the database (KSC) and selecting Tasks -> Shrink -> Database

Once the database was finished shrinking you can see the properties on the database.

Now I don't want to run into this problem again, so I am going to go into Files and change the Autogrowth/Max Size to 1% and 8192MB for storage.

Now this process will work for fixing you Kaspersky Security Center if you happen to run out of database space, however it has a great side effect of upgrading your version of SQL server.  In this case going from 2014 to 2016.

Truncating the database table dbo.wus_upd_metadata and shrinking the database did not seem to have any major effect on our AV software, as now we can now go in and manage Kaspersky Security Center.

Giving User and Group Management Permission in Google Workspaces

Google workspaces has made it much easier to delegate tasks or give users such as HR administrators and personnel permissions for creating users and email groups.  

This can be handy in a small or large organization and has be done with Microsoft using snap-ins for Active Directory for a long time.

Some basic permission for a user to be able to do this they need essential four rolls.

• User Management Admin
• Help Desk Admin
• Groups Reader
• Groups Editor

Once given permission to the users and the groups, a user can then add and remove users, add and remove users to groups, create and rename groups without being able to access any other area in google workspaces such as organizational units, machine and domain management etc.

Microsoft SQL Management Studio Connection Parameters

Connecting to an SQL server remotely is pretty easy but occasionally you might need some additional help, such as if your sql database uses a non standard port.  You have a couple of options for logging into your sql server but I'm going to cover 2 ways of connecting using the Database Engine login method.

As shown if you change the server type to Database Engine then you can put either the machine name or IP address (I have ipaddress because it's easy) with a comma then the port; if your database uses a NON-STANDARD Port (something other then 1443) make sure there are no spaces before or after the comma as shown above.  Set the Authentication method to SQL Server Authentication, put in your database username and password, then connect.

The second method is using the "Additional Connection Parameters".  You must have the database hostname or ip in the server name, then select options.

You will see a number of tabs that will be displayed and you want to select "Additional Connection Parameters" but be warned this is sent in clear text.

You can then put in your connection parameters in the box displayed above.  You can't just put in the connection string you do have to put in the server name or ip in the server name field on the login tab; once that is done the Connection string overrides the GUI settings.

below is a sample connection string:

SERVER=$IP,$PORT;USER=$DBUSER;PASSWORD=$DBPASSWORD

A successful connection will then be displayed in the Object Explorer as shown below.

iTiva import error from Horizon ILS generated notices file

illion i-tiva automated phone software is pretty good but it is quirky.  After disabling the Phone Holds in the iTiva software we started getting import errors; but it wasn't a consistent failure.  As shown in the activity log shown below there was a successful import then 2 failures.

After further investigation by looking at the past notices located in the i-Tiva archive folder, I determined that i-Tiva requires a 1:1 mapping for the notices export from the ILS (in this case Horizon) to the notice.csv file.  When is imported into itivia.   With the covid-19 shutdown it was decided that we would disable the software from making calls when holds were available.  This was not disabled in the ILS and was being included in the generated notices.csv file. With the holds block "$HOLDSBLOCK" being included in the file, the import fails.

"V","","$HOLDSBLOCK","$HOLDSBLOCK","2222200*******","","Frist Name","Last Name","780-***-****","","**","Library Location","3222200*****","13/1/2021","Book Title.","*****-******"

After testing if the the file only had the block "$OVERDUEBLOCK" the import was successful.

"V","","$OVERDUEBLOCK","$NOTICEBLOCK1","2222200*******","","First Name","Last Name","780-***-****","","**","Library Location","3222200*******","19/12/2020","Book Title /","*****-*****"

To fix the export you have to make an edit in the Horizon Table Editor on the Telephone Messaging Control Block.

To apply the fix disable the For: hold notice.  It should appear unchecked as shown below.  That will fix your issue.

A special thanks to Mary Schmidt at SirsiDynix who worked with me as we determined the cause of the issue.

AD Time drift - and how to fix it

 

Over a period of time and windows updates especially if your using a virtualized ad controller or a mix of hardware and vm controllers, time can start to drift.  If your maintaining your own time server good on you but most of us don't; we sync to a time server and occasionally it needs to be resynced.  

After four years and several power outages this happened to a physical and virtual active directory controller, they had a time drift of over 2 minutes and it was causing issues with DNS and DHCP.

To check your current timesync settings open regedit and go to 

\HKEY_LOCAL_Machine\SYSTEM\CurrentControlSet\Services\W32Time\Parameters

You will find a parameter called NTPServer

As you can see in the image above, the ntpserver value is set to time.windows.com. 

The following command will update your server time and set your NTP server to timerserver1 and timeserver 2 which can be any valid time servers you want it to be.

C:\Windows\system32>w32tm /config /syncfromflags:manual /manualpeerlist:"0.time.timeserver1.com 1.time.timerserver2.com"

This will update your NTPServer value with the time servers you specify in timeserver1 and timeserver2.

You can check your windows task scheduler to see if you have had any issues with running the time sync by go to.

Task Scheduler -> Microsoft ->Windows -> Time Synchronization

You will want to run this on both Active Directory controllers and it should fix your time sync issue.