Jun 06
High CPU Utilization on a Terminal Server


We recently ran into a tricky problem on a terminal server.  The CPU utilization would spike to 100% for approximately one minute every 30 minutes.  When this happened, the Terminal Server would appear to freeze and become unresponsive.  Using the Task Manager, we noticed that the Webroot Core process took up most of the CPU resources during the spike.  We contacted Webroot support, and they recommended a uninstall/reinstall of the Webroot Software using the instructions below:

  1. Uninstall Webroot.
    1. Restart the server and press F8 in the boot screen.
    2. Select Safe Mode with networking.
    3. Start a cmd prompt
    4. Type in "C:\Program Files\Webroot\WRSA.exe" -uninstall or "C:\Program Files (x86)\Webroot\WRSA.exe" -uninstall for 64bit systems
    5. Enter the CAPTCHA.
    6. WRData & WRcore Removal
      1. Verify c:\programdata\WRData and c:\programdata\WRCore are removed.  The previously sent uninstall command should remove these folders however, if it hasn't, please delete both.
    7. Verify that the Webroot folders have been removed from both C:\Program Files and C:\Program Files (x86).  Delete them manually if they have not been removed.
  2. Reinstall Webroot
    1. After ensuring that the WRData and WRCore folders are gone, download the most current version of SecureAnywhere. Download the latest software version from the Resources tab in the Management Console.
    2. Do not use an installer that you have downloaded previously.  Please be sure to download a new installer from the console.  This ensures all recent fixes and improvements are included in the latest version.

If you have high CPU Utilization on a server running Webroot, try this fix.  It worked for us!

Jan 01
Exchange Mailflow Breaks on 1/1/2022 at 12:00 a.m. and the Y2k22 Bug


There is a major Bug on how some software stores the date/time of 1/1/2022.  Some software stores the date/time as a long integer variable.  The maximum value of a long integer variable is 2.147.483.647.  But when the date of is stored, systems break because this value is greater than the maximum allowed value of a long integer variable.  This Bug will impact any system that stores the date/time as a long integer variable.  As far as we know, it breaks mail flow on on-premise Exchange Servers and Mail Auditing and the Junk Box on the Sonicwall Email Security Appliance.


For Exchange, you have to bypass malware scanning on your Exchange Server to re-establish mail flow by issuing the following PowerShell commands:


  1. Get-ExchangeServer | % {Set-MalwareFilteringServer -BypassFiltering $true -Identity $_.Name} 
  2. Restart Transport service
  3. $ExchangeServers = Get-ExchangeServer | Select -ExpandProperty Name
  4. $ExchangeServers | % {Get-Service -ComputerName $_ -ServiceName MSExchangeTransport | Restart-Service -Force} 
  5. Restart the Exchange Front end Transport Service.
  6. Restart the Exchange Transport Service.


For SonicWall Email Security, a new version of the firmware will address this Bug – there currently is no workaround.

Nov 30
VMWare has pulled vSphere 7.0 Update 3


VMware has discovered multiple issues with vSphere ESXi 7.0 Update 3, 3a, and 3b and vCenter 7.0 Update 3b.  These versions are no longer available for download.  If you downloaded any of these builds before VMWare pulled them, DO NOT install these updates. 

The ESXi updates can cause the Purple Screen of Death (PSOD) and crash the host under certain circumstances.  Here are links to known issues with these updates:  https://kb.vmware.com/s/article/86287 and https://kb.vmware.com/s/article/86281.  One client installed vSphere 7.0 Update 3, and it changed the iSCSI Qualified Name (IQN) on the host, which caused all of the shared SAN storage to go missing.  If you have any issues that are not resolved by the VMware Knowledgebase articles, open up a support case with VMware.

Oct 04
Facebook and Instragram are down today


Both Facebook and Instgram sites are down for a lot of users today.  It appears to be a problem with Domain Name Services (DNS)​, because both Facebook and Instragram do not resolve to an IP address.

Feb 10
Beware of Tricky Relevant Phishing Email Messages


Some Phishing Emails are floating around that are particularly tricky.  They typically deal with the following subjects:


  1. Payroll Protection Program (PPP).  Quite a few companies have applied for a PPP loan and forgiveness of that loan.  Many fake phishing emails attempt to trick you into clicking on a link, downloading an attachment, or entering confidential information.  They are typically related to the Loan Forgiveness Process or other PPP related matter.  As a general rule, DO NOT open these emails.  Study them carefully, hover over any link to see where they will take you (usually over a cliff), review the sender, recipient, and any other clues that the message is fake.  If it didn't come from your bank or sba.gov, it's probably fake.  If you're not sure, contact your bank or sba.gov directly to confirm that the message is legitimate.
  2. Covid Vaccines.  We've seen numerous emails claiming you can get a vaccine early, pay to "jump" the line, or other methods and promises to get vaccinated early.  Do NOT click on any of these emails.  Instead, go to https://www.cdc.gov/vaccines/covid-19/index.html or other reputable web sites to get information about the availability of vaccinations in your area.

Hackers are aware that people are concerned, stressed, and may let their guard down during the pandemic.  This is the perfect opportunity for these phishing campaigns to be more effective.  Make sure to pause and review the email.  If you have any doubts, delete the email and contact the resource directly.  Stay safe, everyone!


Nov 19
Gift Card Scams


With the holidays fast approaching, be on the lookout for this widespread scam.

This type of scam will slip through most anti-spam and anti-phishing software because most anti-phishing software only analyzes attachments or scans for suspicious links in an email. Gift Card Scam emails are dangerous because they do not have attachments or links to click — they look like emails from someone important in your organization asking for help, but are actually from a Cybercriminal.

The "From:" line of an email can display any name, so a cybercriminal can write a CEO's name but send the email from a different email address.

Gift Card Scam emails often appear to come from your company leader asking for a favor and mentions that they are too busy to talk on the phone. The scammer expects the employee will respond quickly to their boss' request. 

Here are the details and how to identify this type of attack:

  1. Spoof Sender Email.  The person will receive an email that looks like it came from the CEO/CFO/COO requesting help or a favor.  This email will be malformed, so the reply will go to a different email address other than the CEO/CFO/COO.  The email address may be a Gmail or other free email account. The spoofed email address will show when a user replies, but it is often overlooked.  It is overlooked because users don't look at the reply address when responding to the email.
  1. Spoofed Reply Email Address.  The person replies to the email, thinking they are responding to the CEO, but they are really responding to the Cybercriminal.
  1. Correspondence.  The email thread goes back and forth, making the request more and more urgent.  Train your users to be aware of any change of tone for email threads that request gift card purchases.  If the person thinks – wow that doesn't sound like the CEO - it probably isn't.  If you review the email thread, the reply address will be to a different domain – another clue of a fraudulent request.
  2. Purchased Gift Cards.  If the scam is successful, the impersonators will ask you to take photos of the numbers on the back of the gift cards and send them back. Once you send those photos, you're never getting your money back. 

Please make all users aware of these fraudulent attempts and how to spot them.  Closely review how easy it is (usually VERY easy) to determine the CEO/CFO/COO/Controller emails address via your website and social media sites like LinkedIn, Twitter and Facebook. 

Do yourself a big favor, don't immediately respond to emails asking for a favor. Call the person and ask if they really need any extra help (they won't).  Make it a habit to always check the email address when replying to a message to verify that the message is going to the intended person.  Stay safe and Happy Holidays!

Oct 05
SimpliVity VMs may not Inherit a SimpliVity Datastore Backup Policy

SimpliVity Backup Policies.jpg

When you create a datastore with SimpliVity, it is assigned a Backup Policy.  Any new Virtual Machines (VMs) that reside on that datastore will inherit the Backup Policy.  However, if you Storage vMotion a VM to a different datastore, the VM is suppose to retain the original SimpliVity Backup Policy.  In reality, your results may vary.

This is an issue if you Storage vMotion a VM to a new datastore with a different SimpliVity Backup Policy, and assume that the VM will automatically inherit the SimpliVity Backup Policy assigned to the target datastore.  In our experience, we've seen an existing Storage vMotioned VM, sometimes inherit the new target datastore Backup Policy, and sometimes retain the original Backup Policy. 

In addition to our existing data center in Switch Las Vegas, we recently opened up a new data center in Switch Reno.  We created a new datastore in Las Vegas and a new datastore in Reno.  Both new datastores have a Backup Policy that backup locally and to the remote data center.  For clients who want their VMs replicated between the two data centers (or for migrations between the two datacenters), we place their VMs on the datastore with the local and remote replication Backup Policy.  If a client has existing VMs that they want to replicate to the remote data center, we perform a Storage vMotion to the datastore that has the local and replication Backup Policy.  We assumed that the VM would start replicating to the remote datacenter when the Storage vMotion was complete.  In practice, we've seen the VM sometimes get the target datastore Backup Policy and sometimes retain the source datastore Backup Policy. 

The fix here is relatively simple.  After you perform the Storage vMotion, right-click on the VM, and manually assign the target datastore Backup Policy to the migrated VM.  Don't assume that the VM will inherit the target datastore Backup Policy.  If you do not perform this step, the VM may not receive the target datastore Backup Policy.  You may not discover it's an issue until you attempt to restore the VM in the remote datacenter.

Sep 25
SimpliVity Saves the Day (Again)


Yesterday, I received a panicked call from one of our clients.  He cleaned up some disk space on their Enterprise Resources and Planning (ERP) server, and some production data was accidentally deleted.  This caused their ERP package to crash. 

Fortunately, they have a vSphere HPE SimpliVity Cluster that backs up their Virtual Machines every 30 minutes.  Based on the timestamp of the folder, it appeared that the files were deleted around 4:05 p.m.  I reviewed the SimpliVity backups that were available for that ERP server, and there was a backup of the VM that ran at 4:00 p.m. – just before the accidental deletion of the files.  For a file-level restore, SimpliVity has a maximum limit of 32GB.  In this case, they weren't sure how many files were involved, so we decided to restore the entire VM instead of performing a file-level restore.  With SimpliVity, you can restore an entire VM in seconds (it took about 10 seconds) regardless of the VM's size.  We were able to successfully restore the VM and get them back up and running in a few minutes.  The client was VERY HAPPY!

We also run SimpliVity in our ADS Cloud Service.  A few years ago, we received a call from one of our ADS Cloud Clients, a CPA firm.  During tax season, one of the partners accidentally deleted all of the tax information for a client.  Before migrating to ADS Cloud, we informed the client that we backed up hourly and could quickly restore data.  After determining the best restore point, we were able to restore all of the client's deleted data in a few minutes.  The client was THRILLED to get back their deleted files.

The ability to restore a VM in seconds regardless of its size is one of the best features of SimpliVity.  It's saved our SimpliVity and ADS Cloud clients countless hours of work and lost productivity.  The thought of restoring a large server (>1tb) without SimpliVity is an intimidating task and may require six hours to a day or more depending on the speed of your infrastructure.  With SimpliVity, the restore takes seconds, not hours, not days.  For more information about SimpliVity or our ADS Cloud Service, please send an email to info@adscon.com or call us at (310)541-8584 x 100.

Sep 18
Configure Server Core for Remote Management


As you know, Windows Server Core is Windows Server with the Graphical User Interface (GUI) removed from the Operating System.  The advantages of Server Core are:

  1. More Secure
  2. Requires fewer resources.
  3. Fewer patches to install.
  4. Smaller attack foot print.

All Microsoft Applications now support Server Core.  We recommend to using Server Core whenever possible.  However, the biggest disadvantage to server core is the lack of GUI for management.  It is still possible to use the GUI management tools by standing up a full installation of Windows Server with all of the server management tools and enable remote management of the Server Core Servers.  To enable remote management of Server Core:

  1. Verify RDP is enabled. 
    1. Log into the console. 
    2. Type sconfig if the Server Configuration is not loaded.
    3. Select 7 and type in E to enable RDP.
  2. Enable Remote Management.
    1. Select 4, 1 to enable remote management.
    2. Optionally select 3 to allow ping.
  3. Enable Firewall rules for Remote Management.
    1. Exit to a cmd prompt.
    2. Type Powershell.
    3. Enable-NetFirewallRule -Displaygroup "Remote Event*" to enable Remote Event Management.
    4. Enable-NetFirewallRule -Displaygroup "Remote Scheduled*" to enable Remote Scheduled Tasks.
    5. Enable-NetFirewallRule -Displaygroup "Remote Service*" to enable Remote Service Management.
    6. Enable-NetFirewallRule -DisplayGroup "Windows Defender Firewall Remote Management" (Windows 2016 and later) or Enable-NetFirewallRule -DisplayGroup "Windows Firewall Remote Management" (Windows 2012 R2 and earlier) to enable remote Firewall Management. 
    7. Enable-NetFirewallRule -Displaygroup "Remote Volume*" to enable Remote Volume Management.  Make to enable this rule on BOTH the Server Core Computer and the Remote Management Computer!!!
  4. Start the Virtual Disk Service.
    1. Remotely connect to the Services.msc applet on the server core computer.
    2. Start the Virtual Disk Service. 
    3. Make sure to set the Virtual Disk Server to start automatically.

To perform any Server Management tasks, remote into the management server and remotely manage all of your servers running Server Core.  Consider using Server Core to improve your company's security.

Aug 18
Data Backup and the 3 2 1 1 Backup Rule


Why is a backup strategy important?  According to the University of Texas 94% of companies that suffer a catastrophic data loss do not survive.  43% of businesses do not reopen and 51% are out of business within two years.  What's your company's backup strategy?  Make sure your company follows the best backup practices to protect your valuable data.  We suggest following the core strategy of the 3 2 1 1 backup rule.  What is the 3 2 1 1 backup rule?

  1. 3 copies of your data.
  2. 2 copies on separate media.
  3. 1 copy off-site
  4. 1 copy off-line.


3 copies of your data

In addition to the copy of your data that runs on production storage you should have at least two additional backup copies.  This gives you a much greater chance of successful data recovery with three copies of your data.  If there's a 1/1000 chance of errors on each copy of your data then the probability of each copy having errors simultaneously is

1/1000*1/1000*1/1000 = 1,000,000,000

That's a one in a billion chance that all three copies of your data will be unreadable – a very small percentage.

2 copies on separate media

It's important to have at least two copies of your data on separate media like disk AND tape.  If all of your backups reside on the same physical hardware and the hardware fails, you will not be able to restore from backup. 

1 copy off-site

You should keep one copy off-site to protect against disasters like fire, earthquakes and floods.  This copy of your data should be stored far enough away to avoid the impact of any local disaster. 

1 copy off-line

It's important to have at least one copy off-line.  This off-line copy protects you against ransomware.  Even if you have a copy off-site but still online, ransomware could encrypt all of your on-line backups and you will not be able to recover.  Tape is still one of the most cost effective ways to keep your data off-line.  An LTO8 tape can natively store 11 TB of data.

Auditing your Backup

It's a good idea to regularly audit your backup environment before you have to perform a restore.  Here are items to consider when auditing your backup.

  1. Is all necessary data backed up?  Verify that all production data is backed up.  If a server exists it should be backed up.  Be sure that any new servers are added to the backup rotation and any retired servers are taken out of the backup rotation.
  2. What is the backup schedule?  How often does the backup run?  It should be run at least once a day or more often for critical servers.
  3. What were the results of the last backup jobs?  Verify that the last backup was successful.  If not, why are they failing?  How long have they failed?  What was done to correct the backup failures?
  4. How long is disk backup data retained?  We suggest at least two weeks of backup history on disk.  The longer the better.  Your back up disk repository should be at least three times larger than your production data usage.
  5. How long is tape backup data retained? We suggest eight or more weeks for backup tape rotation.  The longer the better.  Make sure you are compliant with any backup retention requirements for your company. 
  6. How are off-line backups stored?  Make sure that off-line backups are securely stored.  Are the backups encrypted?  Are the off-line backups stored in a data approved fire safe? 
  7. What were the results of the last data restore request? Was the company able to successfully restore data when requested?  If you were unable to restore data, what was the cause?  What steps were taken to ensure this doesn't happen again?
  8. Do you run Full Backups?  We recommend full backups at least monthly, just in case a differential/incremental backup gets corrupted.
  9. Do you use Differential Backups?  Differential backups, backup all data that was changed since the last full backup.
  10. Do you use Incremental Backups?  Incremental backups only backup changed data since the last backup.
  11. If you're backing up Virtual Machines (VMs) is your backup software Virtualization aware?  We recommend using a virtualization specific backup solution like Veeam Backup and Replication.
  12. If you're backing up a Virtual Machines (VMs) are you getting Image backups of the VMs?  We suggest obtaining image backups of VMs. This significantly simplifies the VM recovery process.
  13. Are VMs that have transaction log data like Exchange and SQL Server Quiesced?  Quiescing temporarily stops the transaction flow of the database so no partial transactions are included in the backup.  Make sure that your backup solution properly truncates transaction logs after a successful backup.
  14. Are the backups following the 3 2 1 1 backup rule?

Backup Verification

We recommend regular restores of your test environment to ensure your data is properly backed up.  You don't want to find out that your backups are corrupted when you need to perform a data restore. Veeam's SureBackup Feature in their Enterprise and Enterprise Plus versions automatically verifies the integrity of the backup.  It runs a malware scan on VM data, verifies it can start the VM in a protected environment and optionally runs a Cyclic Redundancy Check (CRC) on the data to ensure the backup data is valid.  



In our opinion you can never be too careful with backups.  Having a robust backup strategy ensures you can recover from a data disaster.  For more information on creating a customized backup strategy for your company please send an email to info@adscon.com. 

1 - 10Next

Servers at Switch Las Vegas 

Home of ADS Cloud