Relax and virtualize it!

2c10159This is a guest post from Kristian Wæraas
Senior Consultant Datacenter at Datametrix AS
VMware VCP3/5, MCTS Hyper-V, Horizon View and Trend Micro Security Expert.

I am a curious by nature, and when my colleagues start talking frantic about some system that has crashed, I get curious and have to ask questions. Usually this ends up in me doing a lot of work.

– This, however, was not one of those times.

A few weeks ago, one of my colleagues came in late after a long night trying to fix a reoccurring bluescreen on a critical customer database server. Quite drawn in his face he sat down, picked up his phone and called Microsoft Support. I have to admit that I did some eavesdropping on that conversation, as it contained a few interesting tidbits that aroused my curiosity.

-“Physical server” (We still use those?)

-“Database on FC SAN”.

-“Critical data!” (Oh my!)

The minutes went by and turned into hours, and they still tried to fix the server. Diagnostics, rescue disks, rescue console, driver reinstalls, system file checking, fixing mbr and so on – but the server refused to cooperate. At some point Microsoft gave up on fixing the server, and asked if we could just reinstall it, which in this case would take even more hours.

When lunchtime came they had taken a break, and I started asking my colleague questions; not regarding the bluescreen and possible fixes, but more on the basic layout of the system. It turns out that it was an old physical server running Windows Server 2008 R2, it had Oracle database installed with the database-files placed on SAN mounted directly in to the server via FC – A normal setup for database servers I guess. We had a little chat on possible solutions to the problem during lunch, and my colleagues first thought was actually to find an identical physical server so he could install it parallel to the faulty server, then physically moving the fiber cable from the unstable server to the new. I of course asked if we could virtualize the server instead.

My colleague thought the idea was intriguing, but not knowing all the details in VMware’s possibilities had many questions.

-“How will it perform, how do we get the database files copied into the server, how long will it take to get a server ready, we need at least 2 CPUs and 8GBs of RAM, will there be cake?”

I explained to him that performance wise, the virtual server would do just fine, and that we could give it as much resources as it needed. As for getting a server up and running, I suggested using already prepared templates, which would take no more than a few seconds to deploy. Also, and this was my key point in this solution, the file copy is unnecessary:

-“You don’t have to copy the files from SAN into the server; you can just do zoning on the FC switches, and directly attach the datastore as raw disks on the virtual server. The disks will then appear inside the OS as you are used to”.

-“Is all this possible? How do we do it? If it is as easy as you say this would save us hours of work!”

Having done similar setups before, I was quite confident. However “saving” a physical server with real critical data from humiliation by moving datastore into a virtual server was new to me, so I did a quick tweet to my good friend Christian Mohn (vNinja extraordinare) to run my theory by him. We both agreed that the theory was spot on, but none of us had done this job before.

Being afraid of data loss, data corruption and the procedure in its whole, I agreed to do some tests to see if my theory was viable in our situation. We started with a basic SAN backup of the datastore, and then we did the necessary zoning by adding the backup LUN to the VMware host zone-group. After a quick rescan of datastores on the hosts, we saw the new LUNs available for the hosts.

1

The next thing was to add a new disk on a test-server, choose Raw Device Mappings (physical compability mode).

2

Found the correct LUNid

3

When all this was done, we logged into the test-server, went into Disk management and did a “Rescan Disk”. The disk appeared, drive-letter and all:

4

After verifying that the data was there, and that everything looked good, we felt confident that this approach worked, and we did the entire process again with the “live” data.

I always get a satisfied feeling inside when I am able to help a colleague solve an annoying issue. In this case, my actual work took no time at all; I also managed to open the eyes of my colleague who is now planning more p2v migrations. The customer was also happy, which in the end is what really matters.

I think the moral of the story is that “knowledge is power”. If you know what different solutions/products are capable of and you know how to use them correctly, you will be able to solve most problems quite quickly.

And yes, there was cake!

Creating and Using a Virtual Floppy in vSphere

My new colleague Olav Tvedt asked me if I could test his method of enabling Bitlocker in a VM, on VMware vSphere. Of course, I was happy to oblige.

I followed the same steps as he did in his Running Bitlocker on a Virtual computer post, and it worked perfectly.

The only real difference between doing this in Hyper-V and on ESXi, is that the virtual floppy drive on ESXi by default doesn’t emulate an empty floppy. So, in order to mount a virtual floppy you need to create a new floppy image. Thankfully the vSphere Client can do this for you!

To use the vSphere Client to create a floppy image you can later mount in a VM, you need to edit a VM’s settings. Find the floppy drive, if the VM doesn’t have one add one, close the window and return to the VM settings once the floppy drive has been added, and select “Create new floppy image in datastore: “.

Click on the Browse button and browse to your preferred location for the floppy image. Name it, and click on Ok.

Click on Ok again to close the VM settings window and return to the vSphere Client.

There you go, an empty virtual floppy image that you can mount in a VM is now created.

To mount the image, find the floppy drive icon in the vSphere client and select the Connect to floppy image on a datastore option.

Browse to the location where you created the floppy image, and select it.

Now, the VM has an empty floppy that you’ll need to format before you can use it.

Follow Olav’s guide to encrypt the boot drive with Bitlocker, without the need for a TPM chip or USB device connected to the VM!

And yes, it works as you can see here:

So much for never needing a floppy disk again. Oh, and by the way, you can of course do this is VMware Workstation 8 as well.

Setting Up Automated ESXi Deployments

Automating ESXi installs was made much easier after the release of vSphere 4.1 where the Scripted Install feature was added, and by using VMware Auto Deploy from VMware Labs. VMware Auto Deploy requires that you have vCenter and Host Profiles in your environment, and that again requires that you have Enterprise Plus licenses in your environment.

It is, however, possible to deploy ESXi in an automated fashion completely without vCenter and Host Profiles! By using a combination of a PXE based installation and PowerCLI for automating the setup of ESXi after the initial deployment. As this setup has been put together for a specific work project, my PowerCLI script also copies a VM template to the deployed ESXi host as well as the vMA for administrative tasks. There is one caveat with regards to this setup though, and that is that the free version of ESXi only allows PowerCLI in read-only mode. This means that you will either need to get licenses for the ESXi install, or use trial licenses. With the price drop from VMware on the Remote Office / Branch Office (ROBO) licenses, we’re looking at using that licensing model for our fleet of vessels.

Overview

The “complete package” consists of the following components:

  • Deployment VM
    This is a custom VM, built to provide DHCP and PXE services to do the actual ESXi installation
  • Powershell + PowerCLI
    Scripts that configure the ESXi host, post installation, and copy your initial VMs to the new host

Our current process looks like this:

  1. Connect physical host to deployment laptop via ethernet
  2. Start deployment VM on deployment laptop
  3. When deployment VM is finished booting, start physical host
  4. Physical host boots of network and PXE and installs ESXi
  5. When ESXi installation finishes, run PowerCLI script against host
  6. Disconnect deployment laptop and physical host, and connect physical host to vessel network
  7. Connect vSphere Client to ESXi install and start server VM

Deployment VM

Since our deployment scenario might be a bit out of the ordinary, we have the deployment VM set up on VMware Workstation or VMware Player on a laptop. The reason for this is that we need a mobile deployment model as the locations we are deploying this on are not static. Not only are they mobile, they are actually floating around on rather large oceans. That’s right, we’re deploying ESXi hosts on our vessels world wide!

Basic VM Setup

The basic setup is a standard Windows Server 2008 R2 with IIS installed. We will not be using any of the DHCP or other networking features included in Server 2008. In our environment it’s configured with a static ip of 172.16.200.1

DHCP + PXE Setup

For DHCP and PXE services, we are using Tftpd32 a free and open source application that provides us with all the required services for deployment eg. both DHCP and PXE.

Kickstart Script

Our very basic kickstart script – ks.cfg – looks like this

vmaccepteula
rootpw password
autopart --firstdisk --overwritevmfs
install url http://172.16.200.1/ESXi
network --bootproto=dhcp --device=vmnic0
reboot

Basically this sets the root password, automatically deletes all partitions and sets up a new vmfs, tells the installer that it will find the installation files via http on the server and sets the networking configuration to DHCP. This will of course need tweaking in your environment, but this should at least get you started with building your own. More details on the ks.cfg bootstrap commands can be found in the ESX and vCenter Server Installation Guide

PowerCLI Configuration Script

########################################################
#
# Created by Christian Mohn
# for Seatrans AS
#
# No warranty suggested or implied
#
########################################################</code>

#connect to VirtualCenter or ESX host
#
function Register-VMX {
param($entityName = $null,$dsNames = $null,$template = $false,$ignore = $null,$checkNFS = $false,$whatif=$false)

function Get-Usage{
Write-Host "Parameters incorrect" -ForegroundColor red
Write-Host "Register-VMX -entityName -dsNames [,...]"
Write-Host "entityName : a cluster-, datacenter or ESX hostname (not together with -dsNames)"
Write-Host "dsNames : one or more datastorename names (not together with -entityName)"
Write-Host "ignore : names of folders that shouldn't be checked"
Write-Host "template : register guests ($false)or templates ($true) - default : $false"
Write-Host "checkNFS : include NFS datastores - default : $false"
Write-Host "whatif : when $true will only list and not execute - default : $false"
}

if(($entityName -ne $null -and $dsNames -ne $null) -or ($entityName -eq $null -and $dsNames -eq $null)){
Get-Usage
break
}

if($dsNames -eq $null){
switch((Get-Inventory -Name $entityName).GetType().Name.Replace("Wrapper","")){
"Cluster"{
$dsNames = Get-Cluster -Name $entityName | Get-VMHost | Get-Datastore | where {$_.Type -eq "VMFS" -or $checkNFS} | % {$_.Name}
}
"Datacenter"{
$dsNames = Get-Datacenter -Name $entityName | Get-Datastore | where {$_.Type -eq "VMFS" -or $checkNFS} | % {$_.Name}
}
"VMHost"{
$dsNames = Get-VMHost -Name $entityName | Get-Datastore | where {$_.Type -eq "VMFS" -or $checkNFS} | % {$_.Name}
}
Default{
Get-Usage
exit
}
}
}
else{
$dsNames = Get-Datastore -Name $dsNames | where {$_.Type -eq "VMFS" -or $checkNFS} | Select -Unique | % {$_.Name}
}

$dsNames = $dsNames | Sort-Object
$pattern = "*.vmx"
if($template){
$pattern = "*.vmtx"
}

foreach($dsName in $dsNames){
Write-Host "Checking " -NoNewline; Write-Host -ForegroundColor red -BackgroundColor yellow $dsName
$ds = Get-Datastore $dsName | Select -Unique | Get-View
$dsBrowser = Get-View $ds.Browser
$dc = Get-View $ds.Parent
while($dc.MoRef.Type -ne "Datacenter"){
$dc = Get-View $dc.Parent
}
$tgtfolder = Get-View $dc.VmFolder
$esx = Get-View $ds.Host[0].Key
$pool = Get-View (Get-View $esx.Parent).ResourcePool

$vms = @()
foreach($vmImpl in $ds.Vm){
$vm = Get-View $vmImpl
$vms += $vm.Config.Files.VmPathName
}
$datastorepath = "[" + $ds.Name + "]"

$searchspec = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec
$searchspec.MatchPattern = $pattern

$taskMoRef = $dsBrowser.SearchDatastoreSubFolders_Task($datastorePath, $searchSpec)

$task = Get-View $taskMoRef
while ("running","queued" -contains $task.Info.State){
$task.UpdateViewData("Info.State")
}
$task.UpdateViewData("Info.Result")
foreach ($folder in $task.Info.Result){
if(!($ignore -and (&{$res = $false; $folder.FolderPath.Split("]")[1].Trim(" /").Split("/") | %{$res = $res -or ($ignore -contains $_)}; $res}))){
$found = $FALSE
if($folder.file -ne $null){
foreach($vmx in $vms){
if(($folder.FolderPath + $folder.File[0].Path) -eq $vmx){
$found = $TRUE
}
}
if (-not $found){
if($folder.FolderPath[-1] -ne "/"){$folder.FolderPath += "/"}
$vmx = $folder.FolderPath + $folder.File[0].Path
if($template){
$params = @($vmx,$null,$true,$null,$esx.MoRef)
}
else{
$params = @($vmx,$null,$false,$pool.MoRef,$null)
}
if(!$whatif){
$taskMoRef = $tgtfolder.GetType().GetMethod("RegisterVM_Task").Invoke($tgtfolder, $params)
Write-Host "`t" $vmx "registered"
}
else{
Write-Host "`t" $vmx "registered" -NoNewline; Write-Host -ForegroundColor blue -BackgroundColor white " ==> What If"
}
}
}
}
}
Write-Host "Done"
}
}

# Register-VMX -entityName "MyDatacenter"
# Register-VMX -dsNames "datastore1","datastore2"
# Register-VMX -dsNames "datastore1","datastore2" -template:$true
# Register-VMX -entityName "MyDatacenter" -ignore "SomeFolder"
# Register-VMX -dsNames "datastore3","datastore4" -ignore "SomeFolder" -checkNFS:$true
# Register-VMX -entityName "MyDatacenter" -whatif:$true

if ($args[0] -eq $null)
#
{$hostName = Read-Host "Enter ESX Host Name or IP"}
#
else
#
{$hostName = $args[0]}
#
#
#connect to selected Virtualcenter or host server
#
Connect-VIServer $hostName

# Set Datastore Name
$dsName = "datastore1"
$ds = Get-Datastore -Name $dsName
New-PSDrive -Name $dsName -Root \ -PSProvider VimDatastore -Datastore $ds

# Copy and Register VM from local drive to ESXi Host
Copy-DatastoreItem C:\VMs\MyTestVM\* datastore1:\MyTestVM\ -Force
Register-VMX -dsNames $dsName

# Import vMA ovf
Import-VApp -Source c:\VMs\vMA\vMA-4.1.0.0-268837.ovf -VMHost $hostName -Datastore $ds

# Lets configure the host

########################################################
# Host Configuration #
########################################################

#Disable IPv6
Get-VMHostNetworkAdapter | where { $_.PortGroupName -eq "Service Console 1" } | Set-VMHostNetworkAdapter -IPv6Enabled $false

# Configure networking
# Not finished

# Configure NTP Server
Add-VMHostNtpServer -VMHost $hostName -NtpServer "0.vmware.pool.ntp.org"
Add-VMHostNtpServer -VMHost $hostName -NtpServer "1.vmware.pool.ntp.org"
Add-VMHostNtpServer -VMHost $hostName -NtpServer "2.vmware.pool.ntp.org"

# Set VM Start Policy
$vmstartpolicy = Get-VMStartPolicy -VM MyTestVM
Set-VMStartPolicy -StartPolicy $vmstartpolicy -StartAction PowerOn

This PowerCLI script is not 100% finished yet, the networking part remains to be automated to provide correct configuration based on which vessel we are deploying to, but in general it’s pretty much good to go.

Of course, there are loads of ways to extend and improve this process, but for now this suits our needs very well. I’m sure I’ll need to revise it once vSphere.next is out and ready for deployment.

Using a model like this, combined with some interesting usage patterns for vMA you can create an automated ESXi deployment scenario that let’s you deploy, patch and manage your remote vSphere infrastructure in a pretty streamlined fashion.