Description

The ability to clone a server on the Cloud to create a Client Image is a handy feature. One of the compelling use-cases is the ability to clone a server multiple times after you've installed software and configured the server for its appropriate function, thus saving time and effort. However, When working with Windows servers and the various roles it can assume, you must be careful with the creation of Guest OS Customization Images because some server roles won't support the method of cloning currently in use, such as the Active Directory services

Content / Solution:

Overview

The notes here apply only to Guest OS Customization Images, not to the Non Guest OS Customization version. For more details on the difference, see Introduction to Cloud Server Provisioning, OS Customization, and Best Practices 

As part of a Guest OS Customization Deployment, each Cloud Server gets reconfigured with new networking information and administrator password. The means by which this happens is a Microsoft technology known as sysprep. The sysprep tool is capable of customizing the Windows OS to a high degree. In the Cloud, VMware uses sysprep to set the networking parameters (IP, subnet, gateway, hostname, etc.) for the first NIC, reset the administrator password, generate a new SID and relicense the OS with Microsoft. A consequence of this technique is many server roles may cease to function properly due to the changing of these values. Microsoft has an article on TechNet that lists the various server roles and their tolerance to sysprep in Sysprep Support for Server Roles. The net result of this Windows limitation is that you can NOT create Client Images of servers with these server roles.

Outside of Windows Server roles, 3rd party applications may also fail to function properly when the server has been reconfigured via sysprep or the software may interfere with the sysprep process, causing a clone operation to fail. We have found that some antivirus products cause cloning to fail. While we can't exhaustively test 3rd party software for sysprep compatibility, users are encouraged to utilize the forums to discuss what does work, what may be failing, etc.

Finally, cloning running servers could render servers deployed off the clone unbootable. This is usually due to the file system not being properly closed. Many times when this happens, Windows boots into recovery mode. If this happens during the  deployment process, it will cause the deployment to fail because it interferes with guest customization. It is best to clone servers after they've been cleanly shut down.

Best Practices Around Creating Microsoft Windows Guest OS Customization Images

Please find the list of best practices we recommend while you plan uploading your image to our cloud infrastructure for future scalability of your virtual farm with us:

  1. We strongly recommend you powering off the Image/Virtual machine before initiating hot cloning off it. We have noticed ample failure results when a Guest OS Customization Image is cloned off a running Cloud Server.
  2. Ensure the server doesn’t have Domain Roles configured. Microsoft supports cloning of Domain Services only starting from Windows Server 2012. For details, see Microsoft Technet Reference
  3. Ensure ALL antivirus suites and security products are disabled. Don’t just stop them - disable these services from services.msc. Also, ensure Windows Defender and Windows Firewall are disabled as well. We have noticed security products causing hinderance during cloning/guest customization phase of cloning. Once your servers are deployed, you may enable them back on the deployed images.
  4. It is recommended to check with application vendors if they support cloning/duplicating their applications, especially if they are licensed bonded (or) bonded with system SID/Hostname/IP Addresses/MAC addresses.
  5. If your image is a result of an P2V (physical-virtual conversion), make sure to delete all hardware drivers/hidden devices. Please refer this Microsoft Article to ensure you have got rid of all the hidden hardware/devices on your image.
  6. A few hardware vendors use applications that interact directly with system board. For example, NIC binding applications. Another example could be a system utility/hardware diagnostic tool. Since you are moving into a virtual environment, you would not need them. Please refer your vendor guide and do a clean un-installation of these applications.

Windows 2008 Daisy Chain License Key Issue with Guest OS Customization Deployments

There is a known limitation with Windows 2008 operating systems that prevents from cloning a server 4 times in a daisy-chained fashion. In other words, if you deploy a server, create a customer clone, then deploy from the clone, then repeat that process two more times, the license key will fail. The Windows 2008 activation process allows you to clone it three times before the new VMs fails to customize. MS sysprep throws an error and kills the process

To avoid this problem, you need to do a few extra steps to prepare your Windows 2008 VM prior to your clone. If you do not do these steps, eventually your VM will fail to customize and the deployment will fail.

  1. Open an elevated command prompt and run

    • cscript c:\windows\system32\slmgr.vbs /rearm
       
  2. Edit the registry to reset the rearm counter by setting the ‘SkipRearm’ key to ‘1’.

HKLM\Software\Microsoft\WindowsNT\CurrentVersion\SoftwareProtectionPlatform\SkipRearm

Please refer Microsoft Article explaining Windows Re-Arm procedure. Also, here is a Technet Reference for more information on slmgr.vbs

If you have chain cloned VMs to the point where your new VMs will not customize,  it is quite possible that one of the above criteria has not been met/might have been overlooked. In such a situation, we recommend you going back to the last known good VM, verify and ensure above conditions are met and then retry cloning.

Additional Troubleshooting Tips

If you are still running into issues and have a copy of this image in your own VMware environment, try these steps:

  1. Clone your image and wait till guest customization fails (there are chances that guest customization might not fail when you are cloning it in your environment)
  2. Grab the vmdk from the failed virtual machine and attach it in a running server (in your environment)
  3. Open explorer and look at the below log entries on the failed hard drive:
  • <drive letter>\Windows\Temp\vmware-cust-nativeapp.log

  • <drive letter>\Windows\Panther\setuperr.log

  • <drive letter>\Windows\Temp\VMware-Imc\guestcust.log

One of these log files will be able to give you a clear understanding on the failure and a little bit of searching these errors on the web can get you to the resolution.