Sep 18
Configure Server Core for Remote Management

RemotelyManageServerCore.jpg

As you know, Windows Server Core is Windows Server with the Graphical User Interface (GUI) removed from the Operating System.  The advantages of Server Core are:

  1. More Secure
  2. Requires fewer resources.
  3. Fewer patches to install.
  4. Smaller attack foot print.

All Microsoft Applications now support Server Core.  We recommend to using Server Core whenever possible.  However, the biggest disadvantage to server core is the lack of GUI for management.  It is still possible to use the GUI management tools by standing up a full installation of Windows Server with all of the server management tools and enable remote management of the Server Core Servers.  To enable remote management of Server Core:

  1. Verify RDP is enabled. 
    1. Log into the console. 
    2. Type sconfig if the Server Configuration is not loaded.
    3. Select 7 and type in E to enable RDP.
  2. Enable Remote Management.
    1. Select 4, 1 to enable remote management.
    2. Optionally select 3 to allow ping.
  3. Enable Firewall rules for Remote Management.
    1. Exit to a cmd prompt.
    2. Type Powershell.
    3. Enable-NetFirewallRule -Displaygroup "Remote Event*" to enable Remote Event Management.
    4. Enable-NetFirewallRule -Displaygroup "Remote Scheduled*" to enable Remote Scheduled Tasks.
    5. Enable-NetFirewallRule -Displaygroup "Remote Service*" to enable Remote Service Management.
    6. Enable-NetFirewallRule -DisplayGroup "Windows Defender Firewall Remote Management" (Windows 2016 and later) or Enable-NetFirewallRule -DisplayGroup "Windows Firewall Remote Management" (Windows 2012 R2 and earlier) to enable remote Firewall Management. 
    7. Enable-NetFirewallRule -Displaygroup "Remote Volume*" to enable Remote Volume Management.  Make to enable this rule on BOTH the Server Core Computer and the Remote Management Computer!!!
  4. Start the Virtual Disk Service.
    1. Remotely connect to the Services.msc applet on the server core computer.
    2. Start the Virtual Disk Service. 
    3. Make sure to set the Virtual Disk Server to start automatically.

To perform any Server Management tasks, remote into the management server and remotely manage all of your servers running Server Core.  Consider using Server Core to improve your company's security.

Aug 18
Data Backup and the 3 2 1 1 Backup Rule

DataBackup.jpg

Why is a backup strategy important?  According to the University of Texas 94% of companies that suffer a catastrophic data loss do not survive.  43% of businesses do not reopen and 51% are out of business within two years.  What's your company's backup strategy?  Make sure your company follows the best backup practices to protect your valuable data.  We suggest following the core strategy of the 3 2 1 1 backup rule.  What is the 3 2 1 1 backup rule?

  1. 3 copies of your data.
  2. 2 copies on separate media.
  3. 1 copy off-site
  4. 1 copy off-line.

 

3 copies of your data

In addition to the copy of your data that runs on production storage you should have at least two additional backup copies.  This gives you a much greater chance of successful data recovery with three copies of your data.  If there's a 1/1000 chance of errors on each copy of your data then the probability of each copy having errors simultaneously is

1/1000*1/1000*1/1000 = 1,000,000,000

That's a one in a billion chance that all three copies of your data will be unreadable – a very small percentage.

2 copies on separate media

It's important to have at least two copies of your data on separate media like disk AND tape.  If all of your backups reside on the same physical hardware and the hardware fails, you will not be able to restore from backup. 

1 copy off-site

You should keep one copy off-site to protect against disasters like fire, earthquakes and floods.  This copy of your data should be stored far enough away to avoid the impact of any local disaster. 

1 copy off-line

It's important to have at least one copy off-line.  This off-line copy protects you against ransomware.  Even if you have a copy off-site but still online, ransomware could encrypt all of your on-line backups and you will not be able to recover.  Tape is still one of the most cost effective ways to keep your data off-line.  An LTO8 tape can natively store 11 TB of data.

Auditing your Backup

It's a good idea to regularly audit your backup environment before you have to perform a restore.  Here are items to consider when auditing your backup.

  1. Is all necessary data backed up?  Verify that all production data is backed up.  If a server exists it should be backed up.  Be sure that any new servers are added to the backup rotation and any retired servers are taken out of the backup rotation.
  2. What is the backup schedule?  How often does the backup run?  It should be run at least once a day or more often for critical servers.
  3. What were the results of the last backup jobs?  Verify that the last backup was successful.  If not, why are they failing?  How long have they failed?  What was done to correct the backup failures?
  4. How long is disk backup data retained?  We suggest at least two weeks of backup history on disk.  The longer the better.  Your back up disk repository should be at least three times larger than your production data usage.
  5. How long is tape backup data retained? We suggest eight or more weeks for backup tape rotation.  The longer the better.  Make sure you are compliant with any backup retention requirements for your company. 
  6. How are off-line backups stored?  Make sure that off-line backups are securely stored.  Are the backups encrypted?  Are the off-line backups stored in a data approved fire safe? 
  7. What were the results of the last data restore request? Was the company able to successfully restore data when requested?  If you were unable to restore data, what was the cause?  What steps were taken to ensure this doesn't happen again?
  8. Do you run Full Backups?  We recommend full backups at least monthly, just in case a differential/incremental backup gets corrupted.
  9. Do you use Differential Backups?  Differential backups, backup all data that was changed since the last full backup.
  10. Do you use Incremental Backups?  Incremental backups only backup changed data since the last backup.
  11. If you're backing up Virtual Machines (VMs) is your backup software Virtualization aware?  We recommend using a virtualization specific backup solution like Veeam Backup and Replication.
  12. If you're backing up a Virtual Machines (VMs) are you getting Image backups of the VMs?  We suggest obtaining image backups of VMs. This significantly simplifies the VM recovery process.
  13. Are VMs that have transaction log data like Exchange and SQL Server Quiesced?  Quiescing temporarily stops the transaction flow of the database so no partial transactions are included in the backup.  Make sure that your backup solution properly truncates transaction logs after a successful backup.
  14. Are the backups following the 3 2 1 1 backup rule?

Backup Verification

We recommend regular restores of your test environment to ensure your data is properly backed up.  You don't want to find out that your backups are corrupted when you need to perform a data restore. Veeam's SureBackup Feature in their Enterprise and Enterprise Plus versions automatically verifies the integrity of the backup.  It runs a malware scan on VM data, verifies it can start the VM in a protected environment and optionally runs a Cyclic Redundancy Check (CRC) on the data to ensure the backup data is valid.  

 

Summary

In our opinion you can never be too careful with backups.  Having a robust backup strategy ensures you can recover from a data disaster.  For more information on creating a customized backup strategy for your company please send an email to info@adscon.com. 

Jul 15
Outlook Starts and Immediately Closes

Outlook.jpg

We've received a flood of support calls today regarding Outlook.  When you load Outlook, it starts and then closes after five to ten seconds.  It appears this is caused by a bad Outlook patch that was recently released.  You can roll back to a previous version by issuing the following commands from an Admin Cmd prompt:

  1. cd "\Program Files\Common Files\microsoft shared\ClickToRun"
  2. officec2rclient.exe /update user updatetoversion=16.0.12827.20470

DO NOT attempt a repair of Office.  The repair will fail and leave all Office programs unusable.  Any new installation of Office 365 will fail in the middle of the installation until this issue is resolved.  Until this issue is resolved, DO NOT install any Office updates – it will probably break Outlook.   We will post updates as they become available.

*** Update ***  Microsoft reports that they've fixed the issue and Outlook should automatically update.  It may take several hours before the fix progates to every computer.  The fix IS NOT related to the Security Patches that were released on 7/15/2020.  As a workaround, use Outlook Web App (OWA).  If Outlook does not load try to start Outlook every hour for four hours. If you're still having problems after four hours please contact Microsoft.

Jul 06
SimpliVity File Level Restore Fails

SimpliVityFileLevelRestore.jpg

When we attempted to perform a file level restore (FLR), we received an error on the OVC of fault.com.simplivity.task.error.228.summary.  When you perform an FLR, SimpliVity creates an ISO containing the restored files (as long as you have fewer than 32GB to restore) that is mounted on the Virtual Machine (VM).  Then you can copy over the restored files to any folder on the target VM.  Evidently there is a limit of 103 characters for the full path of the file.  The path and file name of the file we were attempting to restore was longer than 103 characters.  Here's a workaround:

  1. Attempt to restore the file with the long path and file name – this will fail.
  2. Restore a "dummy" file with a short name that is located on the root of the same drive as the "real" file you want to restore.  This should succeed.
  3. Review the contents of the ISO that was mounted on the VM.  In our case both the "dummy" and the "real" were on the ISO so we just copied over the necessary file.

If that doesn't work, you'll have to restore the entire VM in an isolated network, copy over the files to a different location and then move them over to the production VM.

Jun 30
Exchange 2019 Memory Requirements

Exchange2019andMemory.jpg

As you may know, Microsoft recommends:

Exchange 2019 RoleMemory Recommedation
Mailbox128GB
Edge Transport64GB

 

When we first learned about these memory recommendations, they seemed very high, especially when compared to Exchange 2016.  These recommendations are appropriate for installations that have 1000+ mailboxes.

But what about installations that have fewer mailboxes?  Does the server need this much memory?   It depends.  We've found that an Exchange Server with the Mailbox Role that has roughly ten to twenty mailboxes requires around 16GB of memory when running on Windows Server Core.  We typically configure the Exchange Server with four vCPUs and get excellent performance from Outlook Web App and the Exchange Admin Center.

Of course, your experience may vary but unless you are placing a hefty load on the Exchange Server, you probably don't need 128GB of the Exchange 2019 Server for ten to twenty users. 

May 24
Sonicwall SMA 500 Virtual Appliance does not work with NSX

Sonicwall_sma500_viritualappliance.jpg

With COVID-19, we've seen a massive rush for employees to work remotely.  Of course, for remote workers, one of the vital IT Infrastructure components is an SSL VPN.  For some of our clients, we use the Sonicwall SSL VPN SMA500 Virtual Appliance.  When deploying this in our ADS Cloud environment for a new client, we ran into a severe limitation of the SMA 500 Virtual Appliance.  If you attach the SMA 500 to an NSX (VMware's Software Defined Networking) backed network, it will NOT work.  It does work with a vLAN backed network.  Most likely, this is because the Virtual Appliance is compatible with vSphere 4.0, which was released over ten years ago.  Even with the latest build we could find of the SMA 500, which is 10.2.0 we could not ping the appliance after it was deployed with the correct IP address, subnet and default gateway.  The Virtual Appliance is configured with the Flexible Network Card, which doesn't work with NSX.  The workaround is quite simple:

  1. Deploy the appliance.
  2. Remove the three flexible Network Cards.
  3. Add one or more E1000 Network Cards. 
  4. Power on the appliance.
  5. Log in to the console and configure the
    1. IP Address
    2. Subnet
    3. Default Gateway
    4. DNS Servers
    5. Assign a name to the Virtual Appliance.

After we changed the Network Card on the Virtual Appliance from the Flexible to the E1000, we were able to access the Virtual Appliance.  It took us a day to troubleshoot this issue.  Hopefully, you found this article, and it saves you time.  Stay safe everyone!

May 14
Protect your remote workers now and avoid the next wave of Cyber Attacks

hackremoteworkers.jpg

With COVID 19 we've seen a massive rush of companies move to allow all employees to work from home.  Even as companies start to open up, there's been a significant paradigm shift where companies enable employees to work from home at least part-time permanently.   Working remotely has significantly improved the safety of employees; however it makes it much easier for a hacker to gain access to valuable company resources.  Why try to go through the front door that has cameras, guard dog, and an electric fence?  Instead, climb through an unprotected basement window or even tunnel underneath the building to gain access.  Often remote workers represent these poorly protected points of entry into your company.  Here's a list of the top eight items every company should do to protect remote workers:

  1. Remote access with an SSL VPN and Multi-Factor Authentication (MFA).  We consider MFA mandatory for all remote workers.  Ideally, the second factor should be a hardware token (RSA Key or Yubikey) or MFA Application installed on a Smartphone.  You can use a text or email message as your second factor, but they are not as secure as the other MFA methods.  Never exposure Port 3389 (Remote Desktop Protocol) to the Internet.
  2. Install Anti-Virus on all remote workstations.  Make sure that all remote computers have anti-virus installed on them.  The anti-virus solution should have a centrally managed console, so any identified threats can be centrally monitored and managed.
  3. Do NOT use Public Wi-Fi.  Avoid using Public Wi-Fi.  If you have to connect outside of your home, hotspot your phone with a strong password to provide Internet access to your computer.
  4. Install a Firewall at home.  If an employee works at home for an extended length of time, consider purchasing a small home firewall that performs Intrusion Prevention/Detection, Gateway Anti-virus, Stateful Inspection, and Cloud Analytics.  A home firewall that has these features typically costs $400-$500 with one-year support.
  5. Educate your remote workforce.  Notify employees of increased Cyberattacks and to be especially aware of Phishing and Spear Phishing attacks.  Of course, Phishing attacks are emails that trick a user into clicking on a link to gain access and/or install malware on their computer. Spear Phishing attacks include information that is typically only known to internal employees.  Often Spear Phishing attacks involve some type of financial transaction, including wire transfer fraud or redirecting an employee's paycheck to a different bank account.  These attacks will appear to come from an internal trusted employee which often catches other employees off guard.
  6. Train users to store all company data on servers, not home computers.  Train all users to store all company data only on company servers and NOT their local home computers.  If a hard drive crashes on a home computer or gets hit with malware, all of this data will be destroyed.
  7. Use a dedicated computer to access company resources.  Avoid using a shared computer to access company resources.  Ideally, companies should issue a laptop or other dedicated computer that a remote user can use when accessing company resources.  Don't allow remote users to access company resources on the same computer that is shared with their children.
  8. Keep up to date with patches on all remote computers.  Just like Corporate computers, make sure that all remote computers remain up to date with the latest patches.  This may involve including remote computers in your centralized monthly patching solution.  Over 90% of computer hacks are due to unpatched computers.

Following these guidelines will make it significantly more difficult to compromise your Company's valuable data.  If you need help securing your remote workforce, please contact us today at info@adscon.com. 

Apr 17
Snap Server is no longer accessible as a Linux Share after upgrading to Veeam 10

Veeam 10.jpg

Veeam 10.jpg

After upgrading to Veeam 10, you may have problems accessing your Veeam Snap Server Backup Repository if it's configured as a Linux Share.  We noticed that our Veeam Cloud Connect tape backup of the Snap Server was backing 0 bytes even though there was new data on the Snap Server.   If you're running the Snap Server Guardian OS earlier than 8.1.103, then you will need to install a hotfix to make your Snap Server Linux Share accessible to Veeam 10.  Here are the steps to fix the problem.

  1. For instructions on how to configure your Snap Server as a Linux backup repository with Veeam go to https://www.snapserver.com/support/support_files/veeam-backup-repository-gos-v7-7-r8-v8-0-8-1-099-r4.  The Snap Server performance should be a little better when Veeam accesses the Snap Server Veeam Backup Repository as a Linux versus an SMB/Windows Share.
  2. Download the Veeam 10 hotfix at https://www.snapserver.com/support/hotfixes/veeam-10-hotfix-gos-7861
  3. In case you need it, the Snap Server support page is at https://www.snapserver.com/support/.  Snap Server was purchased a few months ago by HVE. 
  4. Log into your Snap Server GUI.
  5. Click on Maintenance, OS Update. 
  6. Make a note of the Current GOS that's running on the Snap Server.
  7. Verify that you're running a Guardian OS earlier than 8.1.103.  There are two patch files included in the download:
    1. Guardian OS 7.7 and earlier:  GOS_7.7.x_veeam_10_GOS-7861_hotfix.gsu.
    2. Guardian OS 8.0 - 8.1.099:   GOS_8.0.074_8.1.099_veeam10_GOS-7861_fix.gsu
    3. Make sure to upload the correct file based on the OS that's running on your Snap Server.
  8. Click on Browse to upload the Hotfix.
  9. Click Upload File.
  10. Click Update.
  11. Test.  Verify that the Hotfix works.

It took a while to resolve this issue with Veeam technical support.  Hopefully, this blog post will save you some time.

Apr 16
Server slow and inaccessible with Vipre 12 Anti-Virus

Vipre.jpg

The current version of Vipre 12 Anti-Virus can cause significant issues with server performance.  Vipre 12 seems to cause more problems with Virtual Machines compared to physical servers.  We've seen the following issues with VMs running Vipre 12:

  1. Server slowness.
  2. Server disconnects.
  3. Servers are inaccessible.
  4. The server is only accessible via the console.
  5. Can't ping other servers in the same subnet.
  6. Can't ping out to the Internet.

Evidently, with Vipre 12 they re-worked the network stack, and this has caused issues with some builds of Windows Servers.  It seems to have a more significant impact on later versions of Windows Server - 2016 and 2019.  Some possible workarounds include from least to more aggressive:

  1. Disable the NDIS driver on the Networking Properties of the VM.
    1. Start, Control Panel, Network and Internet, Network and Sharing Center.
    2. Click on your Ethernet Adapter
    3. Click on Properties.
    4. Look for an NDIS driver and clear the checkbox.
    5. Click Ok.
    6. Test. 
    7. If that doesn't work, continue to step 2.
  2. Uninstall Vipre 12.  We suggest installing Vipre 11, which doesn't seem to have the network disconnect issue.  Do NOT let your server run without any anti-virus protection after uninstalling Vipre 12!
  3. Try vMotioning/Live Migrating your VM to another host.  I know this sounds weird, but for some servers that were still inaccessible after uninstalling Vipre 12, we tried migrating the VM to a different host, and the VM started working again. 

If you haven't upgraded to Vipre 12 yet, we suggest remaining on Vipre 11 until this issue is resolved.  Vipre support is aware of this issue and is actively working on a fix.

*** Update.  Vipre has released v12 Hotfix 1 (12.0.7874) to address this issue.  For more information on this hotfix refer to https://success.vipre.com/endpoint-security/cloud/release-notes/agent/release-notes-20200506

Feb 21
Department of Homeland Security Iranian Advanced Persistent Threat Spreadsheet

iranian_apt_attack_matrix.zip

Attached is a spreadsheet developed by the Department of Homeland Security summarizing Advanced Persistent Threats.

Here's a link to our YouTube Channel which describes details about this spreadsheet https://youtu.be/Zms4fh4Zxc4

1 - 10Next
switchservers.jpg

Servers at Switch Las Vegas 

Home of ADS Cloud