Feb 10
Beware of Tricky Relevant Phishing Email Messages

PPPPhishingEMail.jpg

Some Phishing Emails are floating around that are particularly tricky.  They typically deal with the following subjects:

 

  1. Payroll Protection Program (PPP).  Quite a few companies have applied for a PPP loan and forgiveness of that loan.  Many fake phishing emails attempt to trick you into clicking on a link, downloading an attachment, or entering confidential information.  They are typically related to the Loan Forgiveness Process or other PPP related matter.  As a general rule, DO NOT open these emails.  Study them carefully, hover over any link to see where they will take you (usually over a cliff), review the sender, recipient, and any other clues that the message is fake.  If it didn't come from your bank or sba.gov, it's probably fake.  If you're not sure, contact your bank or sba.gov directly to confirm that the message is legitimate.
  2. Covid Vaccines.  We've seen numerous emails claiming you can get a vaccine early, pay to "jump" the line, or other methods and promises to get vaccinated early.  Do NOT click on any of these emails.  Instead, go to https://www.cdc.gov/vaccines/covid-19/index.html or other reputable web sites to get information about the availability of vaccinations in your area.

Hackers are aware that people are concerned, stressed, and may let their guard down during the pandemic.  This is the perfect opportunity for these phishing campaigns to be more effective.  Make sure to pause and review the email.  If you have any doubts, delete the email and contact the resource directly.  Stay safe, everyone!

 

Nov 19
Gift Card Scams

GiftCardScams.jpg

With the holidays fast approaching, be on the lookout for this widespread scam.

This type of scam will slip through most anti-spam and anti-phishing software because most anti-phishing software only analyzes attachments or scans for suspicious links in an email. Gift Card Scam emails are dangerous because they do not have attachments or links to click — they look like emails from someone important in your organization asking for help, but are actually from a Cybercriminal.

The "From:" line of an email can display any name, so a cybercriminal can write a CEO's name but send the email from a different email address.

Gift Card Scam emails often appear to come from your company leader asking for a favor and mentions that they are too busy to talk on the phone. The scammer expects the employee will respond quickly to their boss' request. 

Here are the details and how to identify this type of attack:

  1. Spoof Sender Email.  The person will receive an email that looks like it came from the CEO/CFO/COO requesting help or a favor.  This email will be malformed, so the reply will go to a different email address other than the CEO/CFO/COO.  The email address may be a Gmail or other free email account. The spoofed email address will show when a user replies, but it is often overlooked.  It is overlooked because users don't look at the reply address when responding to the email.
  1. Spoofed Reply Email Address.  The person replies to the email, thinking they are responding to the CEO, but they are really responding to the Cybercriminal.
  1. Correspondence.  The email thread goes back and forth, making the request more and more urgent.  Train your users to be aware of any change of tone for email threads that request gift card purchases.  If the person thinks – wow that doesn't sound like the CEO - it probably isn't.  If you review the email thread, the reply address will be to a different domain – another clue of a fraudulent request.
  2. Purchased Gift Cards.  If the scam is successful, the impersonators will ask you to take photos of the numbers on the back of the gift cards and send them back. Once you send those photos, you're never getting your money back. 

Please make all users aware of these fraudulent attempts and how to spot them.  Closely review how easy it is (usually VERY easy) to determine the CEO/CFO/COO/Controller emails address via your website and social media sites like LinkedIn, Twitter and Facebook. 

Do yourself a big favor, don't immediately respond to emails asking for a favor. Call the person and ask if they really need any extra help (they won't).  Make it a habit to always check the email address when replying to a message to verify that the message is going to the intended person.  Stay safe and Happy Holidays!

Oct 05
SimpliVity VMs may not Inherit a SimpliVity Datastore Backup Policy

SimpliVity Backup Policies.jpg

When you create a datastore with SimpliVity, it is assigned a Backup Policy.  Any new Virtual Machines (VMs) that reside on that datastore will inherit the Backup Policy.  However, if you Storage vMotion a VM to a different datastore, the VM is suppose to retain the original SimpliVity Backup Policy.  In reality, your results may vary.

This is an issue if you Storage vMotion a VM to a new datastore with a different SimpliVity Backup Policy, and assume that the VM will automatically inherit the SimpliVity Backup Policy assigned to the target datastore.  In our experience, we've seen an existing Storage vMotioned VM, sometimes inherit the new target datastore Backup Policy, and sometimes retain the original Backup Policy. 

In addition to our existing data center in Switch Las Vegas, we recently opened up a new data center in Switch Reno.  We created a new datastore in Las Vegas and a new datastore in Reno.  Both new datastores have a Backup Policy that backup locally and to the remote data center.  For clients who want their VMs replicated between the two data centers (or for migrations between the two datacenters), we place their VMs on the datastore with the local and remote replication Backup Policy.  If a client has existing VMs that they want to replicate to the remote data center, we perform a Storage vMotion to the datastore that has the local and replication Backup Policy.  We assumed that the VM would start replicating to the remote datacenter when the Storage vMotion was complete.  In practice, we've seen the VM sometimes get the target datastore Backup Policy and sometimes retain the source datastore Backup Policy. 

The fix here is relatively simple.  After you perform the Storage vMotion, right-click on the VM, and manually assign the target datastore Backup Policy to the migrated VM.  Don't assume that the VM will inherit the target datastore Backup Policy.  If you do not perform this step, the VM may not receive the target datastore Backup Policy.  You may not discover it's an issue until you attempt to restore the VM in the remote datacenter.

Sep 25
SimpliVity Saves the Day (Again)

hpe-simplivity-saves-the-day.jpg

Yesterday, I received a panicked call from one of our clients.  He cleaned up some disk space on their Enterprise Resources and Planning (ERP) server, and some production data was accidentally deleted.  This caused their ERP package to crash. 

Fortunately, they have a vSphere HPE SimpliVity Cluster that backs up their Virtual Machines every 30 minutes.  Based on the timestamp of the folder, it appeared that the files were deleted around 4:05 p.m.  I reviewed the SimpliVity backups that were available for that ERP server, and there was a backup of the VM that ran at 4:00 p.m. – just before the accidental deletion of the files.  For a file-level restore, SimpliVity has a maximum limit of 32GB.  In this case, they weren't sure how many files were involved, so we decided to restore the entire VM instead of performing a file-level restore.  With SimpliVity, you can restore an entire VM in seconds (it took about 10 seconds) regardless of the VM's size.  We were able to successfully restore the VM and get them back up and running in a few minutes.  The client was VERY HAPPY!

We also run SimpliVity in our ADS Cloud Service.  A few years ago, we received a call from one of our ADS Cloud Clients, a CPA firm.  During tax season, one of the partners accidentally deleted all of the tax information for a client.  Before migrating to ADS Cloud, we informed the client that we backed up hourly and could quickly restore data.  After determining the best restore point, we were able to restore all of the client's deleted data in a few minutes.  The client was THRILLED to get back their deleted files.

The ability to restore a VM in seconds regardless of its size is one of the best features of SimpliVity.  It's saved our SimpliVity and ADS Cloud clients countless hours of work and lost productivity.  The thought of restoring a large server (>1tb) without SimpliVity is an intimidating task and may require six hours to a day or more depending on the speed of your infrastructure.  With SimpliVity, the restore takes seconds, not hours, not days.  For more information about SimpliVity or our ADS Cloud Service, please send an email to info@adscon.com or call us at (310)541-8584 x 100.

Sep 18
Configure Server Core for Remote Management

RemotelyManageServerCore.jpg

As you know, Windows Server Core is Windows Server with the Graphical User Interface (GUI) removed from the Operating System.  The advantages of Server Core are:

  1. More Secure
  2. Requires fewer resources.
  3. Fewer patches to install.
  4. Smaller attack foot print.

All Microsoft Applications now support Server Core.  We recommend to using Server Core whenever possible.  However, the biggest disadvantage to server core is the lack of GUI for management.  It is still possible to use the GUI management tools by standing up a full installation of Windows Server with all of the server management tools and enable remote management of the Server Core Servers.  To enable remote management of Server Core:

  1. Verify RDP is enabled. 
    1. Log into the console. 
    2. Type sconfig if the Server Configuration is not loaded.
    3. Select 7 and type in E to enable RDP.
  2. Enable Remote Management.
    1. Select 4, 1 to enable remote management.
    2. Optionally select 3 to allow ping.
  3. Enable Firewall rules for Remote Management.
    1. Exit to a cmd prompt.
    2. Type Powershell.
    3. Enable-NetFirewallRule -Displaygroup "Remote Event*" to enable Remote Event Management.
    4. Enable-NetFirewallRule -Displaygroup "Remote Scheduled*" to enable Remote Scheduled Tasks.
    5. Enable-NetFirewallRule -Displaygroup "Remote Service*" to enable Remote Service Management.
    6. Enable-NetFirewallRule -DisplayGroup "Windows Defender Firewall Remote Management" (Windows 2016 and later) or Enable-NetFirewallRule -DisplayGroup "Windows Firewall Remote Management" (Windows 2012 R2 and earlier) to enable remote Firewall Management. 
    7. Enable-NetFirewallRule -Displaygroup "Remote Volume*" to enable Remote Volume Management.  Make to enable this rule on BOTH the Server Core Computer and the Remote Management Computer!!!
  4. Start the Virtual Disk Service.
    1. Remotely connect to the Services.msc applet on the server core computer.
    2. Start the Virtual Disk Service. 
    3. Make sure to set the Virtual Disk Server to start automatically.

To perform any Server Management tasks, remote into the management server and remotely manage all of your servers running Server Core.  Consider using Server Core to improve your company's security.

Aug 18
Data Backup and the 3 2 1 1 Backup Rule

DataBackup.jpg

Why is a backup strategy important?  According to the University of Texas 94% of companies that suffer a catastrophic data loss do not survive.  43% of businesses do not reopen and 51% are out of business within two years.  What's your company's backup strategy?  Make sure your company follows the best backup practices to protect your valuable data.  We suggest following the core strategy of the 3 2 1 1 backup rule.  What is the 3 2 1 1 backup rule?

  1. 3 copies of your data.
  2. 2 copies on separate media.
  3. 1 copy off-site
  4. 1 copy off-line.

 

3 copies of your data

In addition to the copy of your data that runs on production storage you should have at least two additional backup copies.  This gives you a much greater chance of successful data recovery with three copies of your data.  If there's a 1/1000 chance of errors on each copy of your data then the probability of each copy having errors simultaneously is

1/1000*1/1000*1/1000 = 1,000,000,000

That's a one in a billion chance that all three copies of your data will be unreadable – a very small percentage.

2 copies on separate media

It's important to have at least two copies of your data on separate media like disk AND tape.  If all of your backups reside on the same physical hardware and the hardware fails, you will not be able to restore from backup. 

1 copy off-site

You should keep one copy off-site to protect against disasters like fire, earthquakes and floods.  This copy of your data should be stored far enough away to avoid the impact of any local disaster. 

1 copy off-line

It's important to have at least one copy off-line.  This off-line copy protects you against ransomware.  Even if you have a copy off-site but still online, ransomware could encrypt all of your on-line backups and you will not be able to recover.  Tape is still one of the most cost effective ways to keep your data off-line.  An LTO8 tape can natively store 11 TB of data.

Auditing your Backup

It's a good idea to regularly audit your backup environment before you have to perform a restore.  Here are items to consider when auditing your backup.

  1. Is all necessary data backed up?  Verify that all production data is backed up.  If a server exists it should be backed up.  Be sure that any new servers are added to the backup rotation and any retired servers are taken out of the backup rotation.
  2. What is the backup schedule?  How often does the backup run?  It should be run at least once a day or more often for critical servers.
  3. What were the results of the last backup jobs?  Verify that the last backup was successful.  If not, why are they failing?  How long have they failed?  What was done to correct the backup failures?
  4. How long is disk backup data retained?  We suggest at least two weeks of backup history on disk.  The longer the better.  Your back up disk repository should be at least three times larger than your production data usage.
  5. How long is tape backup data retained? We suggest eight or more weeks for backup tape rotation.  The longer the better.  Make sure you are compliant with any backup retention requirements for your company. 
  6. How are off-line backups stored?  Make sure that off-line backups are securely stored.  Are the backups encrypted?  Are the off-line backups stored in a data approved fire safe? 
  7. What were the results of the last data restore request? Was the company able to successfully restore data when requested?  If you were unable to restore data, what was the cause?  What steps were taken to ensure this doesn't happen again?
  8. Do you run Full Backups?  We recommend full backups at least monthly, just in case a differential/incremental backup gets corrupted.
  9. Do you use Differential Backups?  Differential backups, backup all data that was changed since the last full backup.
  10. Do you use Incremental Backups?  Incremental backups only backup changed data since the last backup.
  11. If you're backing up Virtual Machines (VMs) is your backup software Virtualization aware?  We recommend using a virtualization specific backup solution like Veeam Backup and Replication.
  12. If you're backing up a Virtual Machines (VMs) are you getting Image backups of the VMs?  We suggest obtaining image backups of VMs. This significantly simplifies the VM recovery process.
  13. Are VMs that have transaction log data like Exchange and SQL Server Quiesced?  Quiescing temporarily stops the transaction flow of the database so no partial transactions are included in the backup.  Make sure that your backup solution properly truncates transaction logs after a successful backup.
  14. Are the backups following the 3 2 1 1 backup rule?

Backup Verification

We recommend regular restores of your test environment to ensure your data is properly backed up.  You don't want to find out that your backups are corrupted when you need to perform a data restore. Veeam's SureBackup Feature in their Enterprise and Enterprise Plus versions automatically verifies the integrity of the backup.  It runs a malware scan on VM data, verifies it can start the VM in a protected environment and optionally runs a Cyclic Redundancy Check (CRC) on the data to ensure the backup data is valid.  

 

Summary

In our opinion you can never be too careful with backups.  Having a robust backup strategy ensures you can recover from a data disaster.  For more information on creating a customized backup strategy for your company please send an email to info@adscon.com. 

Jul 15
Outlook Starts and Immediately Closes

Outlook.jpg

We've received a flood of support calls today regarding Outlook.  When you load Outlook, it starts and then closes after five to ten seconds.  It appears this is caused by a bad Outlook patch that was recently released.  You can roll back to a previous version by issuing the following commands from an Admin Cmd prompt:

  1. cd "\Program Files\Common Files\microsoft shared\ClickToRun"
  2. officec2rclient.exe /update user updatetoversion=16.0.12827.20470

DO NOT attempt a repair of Office.  The repair will fail and leave all Office programs unusable.  Any new installation of Office 365 will fail in the middle of the installation until this issue is resolved.  Until this issue is resolved, DO NOT install any Office updates – it will probably break Outlook.   We will post updates as they become available.

*** Update ***  Microsoft reports that they've fixed the issue and Outlook should automatically update.  It may take several hours before the fix progates to every computer.  The fix IS NOT related to the Security Patches that were released on 7/15/2020.  As a workaround, use Outlook Web App (OWA).  If Outlook does not load try to start Outlook every hour for four hours. If you're still having problems after four hours please contact Microsoft.

Jul 06
SimpliVity File Level Restore Fails

SimpliVityFileLevelRestore.jpg

When we attempted to perform a file level restore (FLR), we received an error on the OVC of fault.com.simplivity.task.error.228.summary.  When you perform an FLR, SimpliVity creates an ISO containing the restored files (as long as you have fewer than 32GB to restore) that is mounted on the Virtual Machine (VM).  Then you can copy over the restored files to any folder on the target VM.  Evidently there is a limit of 103 characters for the full path of the file.  The path and file name of the file we were attempting to restore was longer than 103 characters.  Here's a workaround:

  1. Attempt to restore the file with the long path and file name – this will fail.
  2. Restore a "dummy" file with a short name that is located on the root of the same drive as the "real" file you want to restore.  This should succeed.
  3. Review the contents of the ISO that was mounted on the VM.  In our case both the "dummy" and the "real" were on the ISO so we just copied over the necessary file.

If that doesn't work, you'll have to restore the entire VM in an isolated network, copy over the files to a different location and then move them over to the production VM.

Jun 30
Exchange 2019 Memory Requirements

Exchange2019andMemory.jpg

As you may know, Microsoft recommends:

Exchange 2019 RoleMemory Recommedation
Mailbox128GB
Edge Transport64GB

 

When we first learned about these memory recommendations, they seemed very high, especially when compared to Exchange 2016.  These recommendations are appropriate for installations that have 1000+ mailboxes.

But what about installations that have fewer mailboxes?  Does the server need this much memory?   It depends.  We've found that an Exchange Server with the Mailbox Role that has roughly ten to twenty mailboxes requires around 16GB of memory when running on Windows Server Core.  We typically configure the Exchange Server with four vCPUs and get excellent performance from Outlook Web App and the Exchange Admin Center.

Of course, your experience may vary but unless you are placing a hefty load on the Exchange Server, you probably don't need 128GB of the Exchange 2019 Server for ten to twenty users. 

May 24
Sonicwall SMA 500 Virtual Appliance does not work with NSX

Sonicwall_sma500_viritualappliance.jpg

With COVID-19, we've seen a massive rush for employees to work remotely.  Of course, for remote workers, one of the vital IT Infrastructure components is an SSL VPN.  For some of our clients, we use the Sonicwall SSL VPN SMA500 Virtual Appliance.  When deploying this in our ADS Cloud environment for a new client, we ran into a severe limitation of the SMA 500 Virtual Appliance.  If you attach the SMA 500 to an NSX (VMware's Software Defined Networking) backed network, it will NOT work.  It does work with a vLAN backed network.  Most likely, this is because the Virtual Appliance is compatible with vSphere 4.0, which was released over ten years ago.  Even with the latest build we could find of the SMA 500, which is 10.2.0 we could not ping the appliance after it was deployed with the correct IP address, subnet and default gateway.  The Virtual Appliance is configured with the Flexible Network Card, which doesn't work with NSX.  The workaround is quite simple:

  1. Deploy the appliance.
  2. Remove the three flexible Network Cards.
  3. Add one or more E1000 Network Cards. 
  4. Power on the appliance.
  5. Log in to the console and configure the
    1. IP Address
    2. Subnet
    3. Default Gateway
    4. DNS Servers
    5. Assign a name to the Virtual Appliance.

After we changed the Network Card on the Virtual Appliance from the Flexible to the E1000, we were able to access the Virtual Appliance.  It took us a day to troubleshoot this issue.  Hopefully, you found this article, and it saves you time.  Stay safe everyone!

1 - 10Next
switchservers.jpg

Servers at Switch Las Vegas 

Home of ADS Cloud