Description

This article provides an overview of both Guest OS Customization and Non-Guest OS Customization images and provisioning process. It also provides best practice guidelines to maximize efficiency and performance when deploying servers.

Overview

There two different types of Cloud Server deployment methods each of which use one of two different types of Images:

  1. Guest OS Customization - This is the "standard" method for deploying a Cloud Server that is offered for use with most of our OS Images. It uses VMware's Guest OS Customization feature to make it easy for you to provision a Cloud Server with the correct IP addressing, licensing, and other configuration elements. This VMware feature clones a Cloud Server from the OS Image or Customer Image you have chosen, but modifies elements within the guest operating system to "customize" the Cloud Server in terms of the following:
    1. Configuring each Cloud Server NIC with the requested IP addresses
    2. Setting up requested DNS entries in the Cloud Server
    3. For Windows Servers, the customization uses Microsoft's Sysprep tool to apply the administrator "root" password and provide licensing for the Windows Server, regardless of whether the Server is deployed from an OS Image or a Customer Image. Note there are some Windows applications (such as Active Directory) that do not work well with Sysprep. For details, see Best Practices and Tips around Microsoft Windows Guest OS Customization Client Images
    4. For RedHat Servers:
      1. If the server is deployed from an OS Image. the customization will also apply the "root" password and provide licensing for the RedHat Server only
      2. If the server is deployed from a Customer Image, neither the root password nor licensing is set. The root password will match the password of the Image. You can set up licensing as described in Introducing Red Hat Update Infrastructure
  2. Non-Guest OS Customization - This is an alternative method that is available only in MCP 2.0 locations. With this deployment method, the system simply clones the source Image and makes no changes inside the Guest OS. This provides a combination of advantages and disadvantages:
    1. Since VMware only supports Guest OS Customization for a limited number of OSes, additional Operating Systems are now supported under this method. For details, see What Operating Systems are Currently Supported on CloudControl?
    2. Since no integration within the Guest OS is required with this method, VM Tools is not required to be installed on these images. This opens up the ability to support virtual appliances, various applications, and a number of other Operating Systems that could not be supported with VMware's Guest OS Customization process.
    3. Since no changes are made at the Operating System, Images for applications that do not work well with Guest OS Customization can be supported with this method.
    4. The one downside is that since no interaction is made with the Guest OS, it's up to the user (or an automated script) to configure the IP addresses, DNS entries, etc. after deployment. The server from a Non-Guest OS Customization deployment is an exact replica of the source Image, including NIC configurations (even IP addresses), passwords, etc.

Which deployment process is used depends on whether the Image (Customer or OS) is a standard image (meaning it will use the Guest OS Customization process) or a Non-Guest OS Customization (NGOC) image. For Customer Images, the type of Image is dictated by the decision made when you created the Customer Image through cloning as described in How to Clone a Cloud Server to Create a Client Image or when you imported the image as described in How to Import an OVF Package as a Client Image. Once an image is created, all Cloud Servers will use the deployment process dictated by the Image. In the Updated UI, you can identify a Non-Guest OS Image by the presence of a "plug"  in the upper left-hand corner of the image:

In either case, the provisioning process for a virtual machine is a multi-step procedure that takes place asynchronously. After you make a deployment request through the UI or API, the system provides updates as the Server makes its way through the provisioning process. Each deployment passes through a maximum of eleven steps as the Cloud software interfaces with VMware vCenter, the heart of the VMware virtualization platform. The specific steps are described in the two sections below.  In each case, the step name returned on the API, what is displayed in the Admin UI, and the meaning of the step is given.

When the Data Center location's vCenter accepts the clone request, it places it into its own internal queue. Virtual Center executes these tasks in parallel and on a first come/first served basis. If there are a large number of tasks in the queue, Virtual Center will dynamically scale the number of simultaneously executing tasks according to available memory and CPU. The time it takes for a clone virtual machine task to complete is dependent upon the size of the disks attached to the source image - the more disk space the source has, the longer it takes to make a physical copy on the back-end media. Another factor that can impact step one is the amount of disk I/O on the source and destination datastores. Virtual disks are fully provisioned on the datastores (if the VM has a 50 GB disk, then a 50 GB disk image is created, not some smaller image that is grown to fill the 50 GB as data is written.) It is also worth noting that there may be multiple SAN devices involved. If the source and destination VMs are using the same SAN device, then cloning will be faster as all the reads and writes take place within the same device. When the source and destination are located in different SANs, then the data must traverse the SAN fabric.

Best Practices for Both Deployment Types

In order to ensure the efficient and successful server deployments, ensure your deployments meet the following best practices:

  1. Avoid Large Client Image Sizes To Reduce Deployment Times - The larger the amount of disk on the image, the slower the deployment.
  2. Avoid Simultaneous Deployments from the same Parent Image - As described above, the clone process involves copying files from the source OS image to the target virtual machine. When you spawn multiple simultaneous deployments from the same image, you increase disk contention on the source Image file, which slows down the process. If you space out your requests, they will likely complete more quickly. If you initiate too many simultaneous deployments from the same parent Customer Image, some may fail as they "time out".
  3. Additional Tips on Handling Large-Scale Deployments - One of the advantages of Cloud infrastructure is the ability to quickly scale servers up and down as needed. When dealing with large numbers of deployments, there are a couple other points that should be considered:
    • Consider "Pre-Deploying" servers in a Stopped State - Unlike many competing Cloud solutions, the majority of our charges occur only when your server is actually running. When in a stopped state, your servers incur only minimal charges from Storage Hours. Therefore, rather than deploying servers from an Image as needed, it may be cost-effective to "pre-deploy" these servers in a stopped state and then simply turn them on as needed. You'll carry the costs of the Storage Hours from the stopped servers, but the big advantage is that you simply turn them on and have them available in seconds, rather than the minutes involved in a full deployment.
    • Deploy Servers from Multiple Images - To the extent that you do need to deploy multiple servers simultaneously, create duplicate Client Images and spread your simultaneous deployments across all multiple Images instead of deploying them from the same parent Image. This will reduce disk contention on the reading of the Image file.
    • Space Out Deployment Requests - By spreading out deployment requests only once every 2-5 minutes, you ensure that the system will have deployments at different stages of the process, which will also reduce load.
    • Avoid More than 10 Simultaneous Deployments - We may, at our discretion, throttle requests from a client if they are adversely impacting other customers by simultaneously deploying large numbers of servers. If you have a need to deploy a large environment in a short period of time, please submit a case so we can discuss your situation and find a workable solution.

Guest OS Customization Server Deployment Process

The Guest O/S Customization process involves changes within the operating system. After the physical copy is complete, the hypervisor boots the new virtual machine. During the boot process, VMware Tools loads and begins the customization process. For Linux (Redhat, CentOS and Ubuntu) this involves setting the IP address, gateway, network & broadcast addresses, and, if deploying from an OS image, setting the root password and registering Redhat systems with RHN. For Windows, this involves running the Microsoft tool 'sysprep.' Sysprep is responsible for setting the network parameters, much like with Linux systems, setting the Administrator password and licensing the system with Microsoft. The sysprep process requires rebooting the system once or twice, depending on which version of Windows is being deployed. In either case, the Guest O/S process can be slowed down by 3rd party software that runs before VMware Tools loads and customizes the system. Products like antivirus software, firewalls, auto-logon programs, databases, application servers, etc. can hinder the process if they are CPU intensive or block the boot process. Some products may even cause the customization procedure to fail entirely.

The specific deployment steps associated with deployment from a Guest OS Customization Image are:

DescriptionAdmin UI ValueAPI Value
Step 1: Create Folder
This step occurs quickly and relates to identifying the correct location for server deployment.
Create FolderCREATE_FOLDER
Step 2: Query Removeable Drive Sizes
Initiates an asynchronous query in the VMware vCenter to determine the size of any removable drives attached to the source server image.
Query Removable Drive SizesQUERY_REMOVABLE_DRIVE_SIZES
Step 3: Wait for Query Removable Drive Sizes
Waits for the result of the asynchronous query for the size of any removable drives attached to the source server image.
Wait for Query Removable Drive SizesWAIT_FOR_QUERY_REMOVABLE_DRIVE_SIZES
Step 4: Prepare Clone VM Task
Another quick step as CloudControl examines the storage configuration of the new Cloud Server prior to beginning the clone process.
Prepare Clone VM TaskPREPARE_CLONE_VM_TASK
Step 5: Wait For Datastore Reservation ID Task
The system locates an appropriate datastore on which to place the storage disk(s) required by the new Cloud Server.
Wait For Datastore Reservation ID TaskWAIT_FOR_DATASTORE_RESERVATION_ID_TASK
Step 6: Clone VM Task
Another quick step as the Cloud software passes the request to deploy to VMware vCenter.
Clone VM TaskCLONE_VM_TASK
Step 7: Wait for Clone VM Task
VMware copies the VMDK files of the source OS or Client Image to create your new server and makes any configuration changes to the virtual hardware. This is usually the longest step in the process and the system will provide periodic updates as to the percentage complete.
Wait For Clone VM TaskWAIT_FOR_CLONE_VM_TASK
Step 8: Configure VM
Settings for the disk speed configuration for the server's local disks are sent to VMware.
Configure VMCONFIGURE_VM
Step 9: Wait for Configure VM
CloudControl waits for VMware to apply the final disk settings.
Wait For Configure VMWAIT_FOR_CONFIGURE_VM
Step 10: Prepare Start VM
The system determines the resources required to start the Server.
Prepare Start VMPREPARE_START_VM
Step 11: Wait for Start VM Datastore Reservation
The system locates the resources required to start the Server.
Wait For Start VM Datastore reservationWAIT_FOR_START_VM_DATASTORE_RESERVATION
Step 12: Prepare Move VMX and Files Task
The system calculates the resources required for any removable drives attached to the Server.
Prepare Move VMX and Files TaskPREPARE_MOVE_VMX_AND_FILES_TASK
Step 13: Wait for Move VMX Datastore Reservation
The system locates the resources required for removable drives attached to the Server.
Wait for Move VMX Datastore ReservationWAIT_FOR_MOVE_VMX_DATASTORE_RESERVATION
Step 14: Move VMX and Files
If necessary the system moves the removable drives and related configuration to a datastore with a sufficient level of space available.
Move VMX and FilesMOVE_VMX_AND_FILES
Step 15: Wait for Move VMX and Files
The system waits for the relocation of the Server in the VMware vCenter to complete.
Wait for Move VMX and FilesWAIT_FOR_MOVE_VMX_AND_FILES
Step 16: Power on VM Task
Guest OS Customization occurs when the server is first started. This step powers on the server to allow Guest OS Customization to start.
Power on VM TaskPOWER_ON_VM_TASK
Step 17: Wait for Power on VM Task
VMware powers on the server.
Wait for Power on VM TaskWAIT_FOR_POWER_ON_VM_TASK
Step 18: Wait for Guest OS Customization
VMware assigns the private IP address, network, password, and other variables to the virtual server. This process normally takes a couple of minutes for Linux machines, but takes longer (as long as 10-15 minutes) for Microsoft Windows servers as the process for those Operating Systems involves several reboots.
Wait for Guest OS CustomizationWAIT_FOR_GUEST_OS_CUSTOMIZATION
Step 19: Wait for Guest IP Address
The system ensures the private IP addresses have been correctly installed and are available on your Cloud Network.
Wait for Guest IP AddressWAIT_FOR_GUEST_IP_ADDRESS
Step 20: Power Off VM Task
If you have requested that the server be stopped on deployment, this optional step initiates the shutdown.
Power Off VM TaskPOWER_OFF_VM_TASK
Step 21: Wait for Power Off VM Task
VMware powers off the server.
Wait for Power Off VM taskWAIT_FOR_POWER_OFF_VM_TASK
Step 22: Read VM Configuration
Details of the Server are read from the VMware vCenter
Read VM ConfigurationREAD_VM_CONFIGURATION

Guest OS Customization Additional Notes and Best Practices

  1. Guest OS Customization Images Have Only one NIC - Guest OS Customization Images have only one NIC embedded in them. Additional NICs in MCP 2.0 locations are added as part of the Guest OS customization process (see following note regarding ordering). When you clone a server to create a Guest OS Customization Client Image, a copy of the Source Server is made but all NICs except the Primary NIC are removed - this is different than the Non-Guest OS Customization Image, where the NIC structure is retained.
  2. Guest OS Customization NIC ordering is Unpredictable - When you deploy from a Guest OS Customization Image, you can choose to add additional NICs, but the order in which these NICs will be perceived by the OS is unpredictable. The Primary NIC will always be consistently the first one but others may vary per deployment. If you need to switch NICs after deployment to make sure the NIC is connected to a specific named interface in the OS, see How to Exchange VLANs between Two Different NICs.
  3. Ensure your Images Have VM Tools Running - VM Tools must be running in order for a server to work with Guest OS Customization.
  4. If using VMware Tools, ensure your Images use an Up-To-Date Version -  If you are using VMware Tools on a Cloud Server, it is recommended that they be "up to date" with the current version of vSphere running in your data center location. Ensure your Cloud Server is up-to-date before cloning it to make a Customer Image. For details, see How to View the Status of VM Tools on a Cloud Server and How to Update VMware Tools on a Cloud Server
  5. Avoid Cloning Running Systems - It is common to customize a server, then clone it into a Client Image for deploying off of. Cloning a system that is running will often cause servers deployed from the image to fail step 4 of the provisioning process because the OS boots into a recovery mode, waiting indefinitely for someone to intervene. It is best to gracefully shut down a server before cloning it.
  6. Microsoft Windows-Specific Information - The Guest OS Customization process uses Sysprep when working with Microsoft Windows deployments. This introduces some additional complexity - for details, see Best Practices and Tips around Microsoft Windows Guest OS Customization Client Images

For a step by step deployment guide please see: How to Deploy a Cloud Server from a Guest OS Customization Image.

Non-Guest OS Customization Server Deployment Process

The Non-Guest O/S Customization (NGOC) process literally clones the Image and connects it's NICs to the requested VLANs. Unlike Guest OS Images, NGOC images can have multiple NICs. Since you are making an exact clone of the server, you must specify a VLAN for each NIC on the source NGOC Image. However, none of the NICs are configured within the Guest OS by the deployment process - so although the virtual hardware configuration may have changed, the configuration inside the OS will remain identical to the state of the original image.

The specific deployment steps associated with deployment from a Non-Guest OS Customization Image are:

DescriptionAdmin UI ValueAPI Value
Step 1: Create Folder
This step occurs quickly and relates to identifying the correct location for server deployment.
Create FolderCREATE_FOLDER
Step 2: Query Removeable Drive Sizes
Initiates an asynchronous query in the VMware vCenter to determine the size of any removable drives attached to the source server image.
Query Removable Drive SizesQUERY_REMOVABLE_DRIVE_SIZES
Step 3: Wait for Query Removable Drive Sizes
Waits for the result of the asynchronous query for the size of any removable drives attached to the source server image.
Wait for Query Removable Drive SizesWAIT_FOR_QUERY_REMOVABLE_DRIVE_SIZES
Step 4: Prepare Clone VM Task
Another quick step as CloudControl examines the storage configuration of the new Cloud Server prior to beginning the clone process.
Prepare Clone VM TaskPREPARE_CLONE_VM_TASK
Step 5: Wait For Datastore Reservation ID
The system locates an appropriate datastore on which to place the storage disk(s) required by the new Cloud Server.
Wait For Datastore Reservation IDWAIT_FOR_DATASTORE_RESERVATION_ID
Step 6: Clone VM Task
Another quick step as the Cloud software passes the request to deploy to VMware vCenter.
Clone VM TaskCLONE_VM_TASK
Step 7: Wait for Clone VM
VMware copies the VMDK files of the source OS or Client Image to create your new server and makes any configuration changes to the virtual hardware. This is usually the longest step in the process and the system will provide periodic updates as to the percentage complete.
Wait For Clone VMWAIT_FOR_CLONE_
Step 8: Configure VM
Settings for the disk speed configuration of the server's local disks are sent to VMware.
Configure VMCONFIGURE_VM
Step 9: Wait for Configure VM
CloudControl waits for VMware to apply the final disk settings.
Wait for Configure VMWAIT_FOR_CONFIGURE_VM
Step 10: Prepare Start VM
The system determines the resources required to start the Server.
Prepare Start VMPREPARE_START_VM
Step 11: Wait for Start VM Datastore Reservation
The system locates the resources required to start the Server.
Wait For Start VM Datastore reservationWAIT_FOR_START_VM_DATASTORE_RESERVATION
Step 12: Prepare Move VMX and Files Task
The system calculates the resources required for any removable drives attached to the Server.
Prepare Move VMX and Files TaskPREPARE_MOVE_VMX_AND_FILES_TASK
Step 13: Wait for Move VMX Datastore Reservation
The system locates the resources required for removable drives attached to the Server.
Wait for Move VMX Datastore ReservationWAIT_FOR_MOVE_VMX_DATASTORE_RESERVATION
Step 14: Move VMX and Files
If necessary the system moves the removable drives and related configuration to a datastore with a sufficient level of space available.
Move VMX and FilesMOVE_VMX_AND_FILES
Step 15: Wait for Move VMX and Files
The system waits for the relocation of the Server in the VMware vCenter to complete.
Wait for Move VMX and FilesWAIT_FOR_MOVE_VMX_AND_FILES
Step 16: Power on VM Task
Guest OS Customization occurs when the server is first started. This step powers on the server to allow Guest OS Customization to start.
Power on VM TaskPOWER_ON_VM_TASK
Step 17: Wait for Power on VM Task
VMware powers on the server.
Wait for Power on VM TaskWAIT_FOR_POWER_ON_VM_TASK
Step 18: Read VM Configuration
Details of the Server are read from the VMware vCenter
Read VM ConfigurationREAD_VM_CONFIGURATION

Non-Guest OS Customization Additional Notes and Best Practices

  1. Non-Guest OS Customization Client Images Can Have Multiple NICs - Non-Guest OS Customization Images can have multiple NICs. On deployment, they must always have a VLAN connected to each NIC (see note below). When you clone a server to create a Non-Guest OS Customization Client Image, the resulting Image will maintain the same number of NICs and NIC order. This is different than the Non-Guest OS Customization Image where the NIC structure is not retained.
  2. Every NIC Must Have a VLAN Assigned - As noted above, Non-Guest OS Customization Images have a fixed number of NICs. Therefore, on deployment, you must specify a different VLAN for each NIC associated with the source Image. You cannot add or subtract NICs as part of the deployment process - these changes can be made only after deployment. The upside is that unlike Guest OS Customization, the NIC ordering inside the Guest OS will remain the same.
  3. Remember To Configure the Guest OS After Deployment - After deployment, you have to assign both IPv4 and IPv6 addresses to each NIC, add DNS entries, adjust Windows timezone configurations, and change the root/administrator password (if desired). In many cases, this will require logging in via Console as described in How to Get Virtual Console Access to a Cloud Server.
  4. Potential for IP Conflicts Exists - Nothing inside the Guest OS is updated from the original clone. If the Image was cloned from a Cloud Server, this means you may need to take care to avoid IP conflicts on the deployed server if you deploy the Cloud Server onto the same VLANs as the server from which the Image was created since those same IP addresses will still be tied to the NICs on the Operating System.
  5. Software Licenses May Need to be Updated - Since the deployed Cloud Server is an exact clone of the original Image, all license keys for the OS and applications will remain the same. Depending on the software involved, you may need to install a new license key for "one use" software licensed applications or OS. 

For a step-by-step deployment guide, please see: How to Deploy a Cloud Server from a Non-Guest OS Customization Image.