Archives for posts with tag: realworld

One of the last projects I’ve been involved with at Seatrans, is to auto­mate the instal­la­tion and con­fig­u­ra­tion of vSphere ESXi 5 hosts for deploy­ment on ves­sels. I’ve talked a bit about this before, both on vSoup and in Set­ting Up Auto­mated ESXi Deploy­ments where I out­lined my PXE and Pow­er­CLI based instal­la­tion and con­fig­u­ra­tion scheme. Not much has changed since then, except updat­ing the PXE server to offer ESXi 5, instead of ESXi 4 and a lot of work has been put into the script­ing, includ­ing a front-end GUI for the Pow­er­CLI script itself. The end “prod­uct” is now in place for mass deploy­ments for inter­nal use.

The fol­low­ing video shows how the PXE based instal­la­tion works, as well as a run through the now GUI based con­fig­u­ra­tion tool aptly called Seatrans Hyper­vi­sor Instal­la­tion Tool.

The video jumps a bit between two VMs, one run­ning Win­dows Server 2008 R2, that runs the DHCP/PXE ser­vices and the Pow­er­CLI script, and one that gets ESXi installed and configured:

This goes to show that you can cre­ate your own, spe­cial­ized and portable deploy­ment solu­tion with­out requir­ing elab­o­rate net­work con­fig­u­ra­tions or recon­fig­ur­ing of exist­ing infrastructure.

Note: I will not be pro­vid­ing down­load­able ver­sions of the final script at this time. The rea­son for this is quite sim­ple, it’s very spe­cific and tai­lored for a non-generic envi­ron­ment. If I can man­age to find the time, I’ll post a generic ver­sion later but in order for any­one else to uti­lize the Pow­er­CLI scripts I’ve cre­ated, a lot of work is required.

The last cou­ple of days I’ve been in train­ing class, tak­ing the 6451B Plan­ning, Deploy­ing and Man­ag­ing Microsoft Sys­tem Cen­ter Con­fig­u­ra­tion Man­ager 2007 course.

One of the first things that got men­tioned, was that for larger deploy­ments you should not run Sys­tem Cen­ter Con­fig­u­ra­tion Man­ager vir­tu­ally. Of course, this caught my eye as I’m a pro­po­nent for the vir­tu­al­ize first “movement”.

It runs out that the rea­son for this is that Con­fig­u­ra­tion Man­ager is some­what poorly designed, as just about every­thing it receives from the clients in the net­work is placed in text based log files (inbox folder) before being processed and pumped into the back-end SQL DB. SCCM lives for and eats log files for a living.

I’m sure there are good rea­sons for this, espe­cially back in the day when SCCM 2007 was devel­oped, but in ret­ro­spect this seems to be a poor design choice. Espe­cially since the IO inten­sity of writ­ing text-based log files is high, and doesn’t scale well when you have loads of clients which SCCM 2007 sup­pos­edly is designed for in the first place.

There are ways of alle­vi­at­ing the strain on the machine run­ning SCCM, like run­ning the SQL server on a dif­fer­ent server and run­ning the man­age­ment con­sole on your local com­puter (remem­ber a Win­dows server is tuned, by default, to pri­or­i­tize back­ground tasks), but the fact remains that sum of all the small write oper­a­tions SCCM con­stantly does to your stor­age puts a heavy strain on it.

So, if you want to run SCCM 2007 vir­tu­al­ized in your envi­ron­ment, make sure your stor­age is up to the task and that you don’t sat­u­rate it by deploy­ing what is in essence man­age­ment soft­ware. Per­haps it is bet­ter to run it on a phys­i­cal server with ade­quate local stor­age, but don’t blame it on vir­tu­al­iza­tion, blame it on poor design in SCCM 2007.

Hope­fully Con­fig­u­ra­tion Man­ager 2012, which is cur­rently in beta 2, behaves bet­ter when it’s released. If not, how will Microsoft defend not get­ting real per­for­mance when run­ning it in Hyper-V (or any other Hypervisor).

Automat­ing ESXi installs was made much eas­ier after the release of vSphere 4.1 where the Scripted Install fea­ture was added, and by using VMware Auto Deploy from VMware Labs. VMware Auto Deploy requires that you have vCen­ter and Host Pro­files in your envi­ron­ment, and that again requires that you have Enter­prise Plus licenses in your environment.

It is, how­ever, pos­si­ble to deploy ESXi in an auto­mated fash­ion com­pletely with­out vCen­ter and Host Pro­files! By using a com­bi­na­tion of a PXE based instal­la­tion and Pow­er­CLI for automat­ing the setup of ESXi after the ini­tial deploy­ment. As this setup has been put together for a spe­cific work project, my Pow­er­CLI script also copies a VM tem­plate to the deployed ESXi host as well as the vMA for admin­is­tra­tive tasks. There is one caveat with regards to this setup though, and that is that the free ver­sion of ESXi only allows Pow­er­CLI in read-only mode. This means that you will either need to get licenses for the ESXi install, or use trial licenses. With the price drop from VMware on the Remote Office / Branch Office (ROBO) licenses, we’re look­ing at using that licens­ing model for our fleet of vessels.

Overview

The “com­plete pack­age” con­sists of the fol­low­ing components:

  • Deploy­ment VM
    This is a cus­tom VM, built to pro­vide DHCP and PXE ser­vices to do the actual ESXi installation
  • Pow­er­shell + Pow­er­CLI
    Scripts that con­fig­ure the ESXi host, post instal­la­tion, and copy your ini­tial VMs to the new host

Our cur­rent process looks like this:

  1. Con­nect phys­i­cal host to deploy­ment lap­top via ethernet
  2. Start deploy­ment VM on deploy­ment laptop
  3. When deploy­ment VM is fin­ished boot­ing, start phys­i­cal host
  4. Phys­i­cal host boots of net­work and PXE and installs ESXi
  5. When ESXi instal­la­tion fin­ishes, run Pow­er­CLI script against host
  6. Dis­con­nect deploy­ment lap­top and phys­i­cal host, and con­nect phys­i­cal host to ves­sel network
  7. Con­nect vSphere Client to ESXi install and start server VM

Deploy­ment VM

Since our deploy­ment sce­nario might be a bit out of the ordi­nary, we have the deploy­ment VM set up on VMware Work­sta­tion or VMware Player on a lap­top. The rea­son for this is that we need a mobile deploy­ment model as the loca­tions we are deploy­ing this on are not sta­tic. Not only are they mobile, they are actu­ally float­ing around on rather large oceans. That’s right, we’re deploy­ing ESXi hosts on our ves­sels world wide!

Basic VM Setup

The basic setup is a stan­dard Win­dows Server 2008 R2 with IIS installed. We will not be using any of the DHCP or other net­work­ing fea­tures included in Server 2008. In our envi­ron­ment it’s con­fig­ured with a sta­tic ip of 172.16.200.1

DHCP + PXE Setup

For DHCP and PXE ser­vices, we are using Tftpd32 a free and open source appli­ca­tion that pro­vides us with all the required ser­vices for deploy­ment eg. both DHCP and PXE.

Kick­start Script

Our very basic kick­start script — ks.cfg — looks like this

vmac­cep­teula
rootpw pass­word
autopart first­disk over­writevmfs
install url http://172.16.200.1/ESXi
net­work boot­proto=dhcp device=vmnic0
reboot

Basi­cally this sets the root pass­word, auto­mat­i­cally deletes all par­ti­tions and sets up a new vmfs, tells the installer that it will find the instal­la­tion files via http on the server and sets the net­work­ing con­fig­u­ra­tion to DHCP. This will of course need tweak­ing in your envi­ron­ment, but this should at least get you started with build­ing your own. More details on the ks.cfg boot­strap com­mands can be found in the ESX and vCen­ter Server Instal­la­tion Guide

Pow­er­CLI Con­fig­u­ra­tion Script

########################################################
#
# Cre­ated by Chris­t­ian Mohn
# for Seatrans AS
#
# No war­ranty sug­gested or implied
#
########################################################</code>

#con­nect to Vir­tu­al­Cen­ter or ESX host
#
func­tion Reg­is­ter-VMX {
param($enti­ty­Name = $null,$dsNames = $null,$tem­plate = $false,$ignore = $null,$check­NFS = $false,$whatif=$false)

func­tion Get-Usage{
Write-Host “Para­me­ters incor­rect” –Fore­ground­Color red
Write-Host “Register-VMX –enti­ty­Name –dsNames [,…]“
Write-Host “enti­ty­Name : a cluster-, dat­a­cen­ter or ESX host­name (not together with –dsNames)“
Write-Host “dsNames : one or more data­s­tore­name names (not together with –enti­ty­Name)“
Write-Host “ignore : names of fold­ers that shouldn’t be checked“
Write-Host “tem­plate : reg­is­ter guests ($false)or tem­plates ($true) — default : $false“
Write-Host “check­NFS : include NFS data­s­tores — default : $false“
Write-Host “whatif : when $true will only list and not exe­cute — default : $false“
}

if(($enti­ty­Name –ne $null –and $dsNames –ne $null) –or ($enti­ty­Name –eq $null –and $dsNames –eq $null)){
Get-Usage
break
}

if($dsNames –eq $null){
switch((Get-Inven­tory –Name $enti­ty­Name).Get­Type().Name.Replace(“Wrap­per”,””)){
“Clus­ter”{
$dsNames = Get-Clus­ter –Name $enti­ty­Name | Get-VMHost | Get-Data­s­tore | where {$_.Type –eq VMFS –or $check­NFS} | % {$_.Name}
}
“Dat­a­cen­ter”{
$dsNames = Get-Dat­a­cen­ter –Name $enti­ty­Name | Get-Data­s­tore | where {$_.Type –eq VMFS –or $check­NFS} | % {$_.Name}
}
“VMHost”{
$dsNames = Get-VMHost –Name $enti­ty­Name | Get-Data­s­tore | where {$_.Type –eq VMFS –or $check­NFS} | % {$_.Name}
}
Default{
Get-Usage
exit
}
}
}
else{
$dsNames = Get-Data­s­tore –Name $dsNames | where {$_.Type –eq VMFS –or $check­NFS} | Select –Unique | % {$_.Name}
}

$dsNames = $dsNames | Sort-Object
$pat­tern = “*.vmx“
if($tem­plate){
$pat­tern = “*.vmtx“
}

fore­ach($dsName in $dsNames){
Write-Host “Check­ing ” –NoNew­line; Write-Host –Fore­ground­Color red –Back­ground­Color yel­low $dsName
$ds = Get-Data­s­tore $dsName | Select –Unique | Get-View
$dsBrowser = Get-View $ds.Browser
$dc = Get-View $ds.Par­ent
while($dc.MoRef.Type –ne “Dat­a­cen­ter”){
$dc = Get-View $dc.Par­ent
}
$tgt­folder = Get-View $dc.VmFolder
$esx = Get-View $ds.Host[0].Key
$pool = Get-View (Get-View $esx.Par­ent).Resour­ce­Pool

$vms = @()
fore­ach($vmImpl in $ds.Vm){
$vm = Get-View $vmImpl
$vms += $vm.Config.Files.VmPathName
}
$data­s­torepath = “[“ + $ds.Name + ”]“

$search­spec = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec
$search­spec.Match­Pat­tern = $pat­tern

$taskMoRef = $dsBrowser.SearchDatastoreSubFolders_Task($data­s­torePath, $search­Spec)

$task = Get-View $taskMoRef
while (“run­ning”,“queued” –con­tains $task.Info.State){
$task.Updat­e­View­Data(“Info.State”)
}
$task.Updat­e­View­Data(“Info.Result”)
fore­ach ($folder in $task.Info.Result){
if(!($ignore –and (&{$res = $false; $folder.FolderPath.Split(”]”)[1].Trim(” /”).Split(”/”) | %{$res = $res –or ($ignore –con­tains $_)}; $res}))){
$found = $FALSE
if($folder.file –ne $null){
fore­ach($vmx in $vms){
if(($folder.Fold­er­Path + $folder.File[0].Path) –eq $vmx){
$found = $TRUE
}
}
if (–not $found){
if($folder.Fold­er­Path[-1] –ne “/”){$folder.Fold­er­Path += “/”}
$vmx = $folder.Fold­er­Path + $folder.File[0].Path
if($tem­plate){
$params = @($vmx,$null,$true,$null,$esx.MoRef)
}
else{
$params = @($vmx,$null,$false,$pool.MoRef,$null)
}
if(!$whatif){
$taskMoRef = $tgt­folder.Get­Type().Get­Method(“RegisterVM_Task”).Invoke($tgt­folder, $params)
Write-Host ‘t $vmx “reg­is­tered“
}
else{
Write-Host ‘t $vmx “reg­is­tered” –NoNew­line; Write-Host –Fore­ground­Color blue –Back­ground­Color white ” ==> What If“
}
}
}
}
}
Write-Host “Done“
}
}

# Register-VMX –enti­ty­Name “MyDat­a­cen­ter“
# Register-VMX –dsNames “datastore1”,“datastore2“
# Register-VMX –dsNames “datastore1”,“datastore2” –template:$true
# Register-VMX –enti­ty­Name “MyDat­a­cen­ter” –ignore “Some­Folder“
# Register-VMX –dsNames “datastore3”,“datastore4” –ignore “Some­Folder” –checkNFS:$true
# Register-VMX –enti­ty­Name “MyDat­a­cen­ter” –whatif:$true

if ($args[0] –eq $null)
#
{$host­Name = Read-Host “Enter ESX Host Name or IP}
#
else
#
{$host­Name = $args[0]}
#
#
#con­nect to selected Vir­tu­al­cen­ter or host server
#
Con­nect-VIS­erver $host­Name

# Set Data­s­tore Name
$dsName = “datastore1“
$ds = Get-Data­s­tore –Name $dsName
New-PSDrive –Name $dsName –Root \ –PSProvider Vim­Data­s­tore -Data­s­tore $ds

# Copy and Reg­is­ter VM from local drive to ESXi Host
Copy-Data­s­tor­e­Item C:\VMs\MyTestVM\* datastore1:\MyTestVM\ –Force
Reg­is­ter-VMX -dsNames $dsName

# Import vMA ovf
Import-VApp -Source c:\VMs\vMA\vMA-4.1.0.0-268837.ovf -VMHost $host­Name -Data­s­tore $ds

# Lets con­fig­ure the host

########################################################
# Host Con­fig­u­ra­tion #
########################################################

#Dis­able IPv6
Get-VMHost­Net­workAdapter | where { $_.Port­Group­Name –eq “Ser­vice Con­sole 1″ } | Set-VMHost­Net­workAdapter -IPv6Enabled $false

# Con­fig­ure net­work­ing
# Not fin­ished

# Con­fig­ure NTP Server
Add-VMHost­NtpServer -VMHost $host­Name -NtpServer “0.vmware.pool.ntp.org“
Add-VMHost­NtpServer -VMHost $host­Name -NtpServer “1.vmware.pool.ntp.org“
Add-VMHost­NtpServer -VMHost $host­Name -NtpServer “2.vmware.pool.ntp.org“

# Set VM Start Pol­icy
$vmstart­pol­icy = Get-VMStart­Pol­icy -VM MyTestVM
Set-VMStart­Pol­icy -Start­Pol­icy $vmstart­pol­icy -Star­tAc­tion PowerOn

This Pow­er­CLI script is not 100% fin­ished yet, the net­work­ing part remains to be auto­mated to pro­vide cor­rect con­fig­u­ra­tion based on which ves­sel we are deploy­ing to, but in gen­eral it’s pretty much good to go.

Of course, there are loads of ways to extend and improve this process, but for now this suits our needs very well. I’m sure I’ll need to revise it once vSphere.next is out and ready for deployment.

Using a model like this, com­bined with some inter­est­ing usage pat­terns for vMA you can cre­ate an auto­mated ESXi deploy­ment sce­nario that let’s you deploy, patch and man­age your remote vSphere infra­struc­ture in a pretty stream­lined fashion.

Let me start this post out with a lit­tle story. I am nor­mally a hard­core vir­tu­al­iza­tion and stor­age guy. Some­times my career in this sec­tor brings me into work­ing with stuff I haven’t worked with before because vir­tu­al­iza­tion encom­passes so much. As I con­tinue to work with other teams I learn more and more about what they do every­day. I usu­ally find myself involved in every per­for­mance trou­bleshoot­ing ses­sion and every new project these days. My per­sonal phi­los­o­phy is the IT guy of the future will be truly con­verged just as all the tech­nolo­gies are con­verg­ing into 1 box or “stack”. Spe­cial­ties in smaller sub­sets will fall away and a more spe­cial­iza­tion in every­thing Dat­a­cen­ter may become the norm.

Early Mon­day morn­ing I over­heard a con­ver­sa­tion about con­nec­tion issues with our new Exchange 2010 envi­ron­ment while drink­ing some cof­fee and review­ing my brand new vSphere design. I didn’t think about it very much until my boss came to my desk and asked me to have a look at the prob­lem. Our mes­sag­ing guy was on vaca­tion and I was the only other per­son on staff who had some mes­sag­ing expe­ri­ence. It seems that all of our global and even local offices were com­plain­ing about ran­dom exchange dis­con­nec­tions and also includ­ing email deliv­ery delays from 30 min­utes to 4 hours! It seems Activesync devices and OWA users were not affected by these delays at all. Being always up for learn­ing new stuff I took the challenge.

First let’s start with the quick facts I could put together. We had users in every coun­try we have offices com­plain­ing about the ran­dom dis­con­nec­tions and delays. I had one actu­ally con­firmed in China but had some slight trou­ble get­ting exact user names from the local IT per­son. Also we had con­nec­tions ran­domly dis­con­nect­ing and show­ing dis­con­nected in the lower right hand cor­ner of the out­look client. I did not have any con­fir­ma­tion of who exactly was hav­ing these prob­lems. To start I dug through the event logs on all the servers in the Exchange 2010 envi­ron­ment and the amount of errors I found was over­whelm­ing. To shorten this up a bit and not write a novel most all of those I inves­ti­gated were directly related to run­ning Exchange 2010 SP1 with­out any update rollups in place. There were cor­re­spond­ing KB arti­cles from Microsoft con­firm­ing these fixes in var­i­ous update rollups.

I noticed an Event ID 2915 on our CAS servers that stuck out. I noticed sev­eral EWS and RPC con­nec­tions report­ing “Ses­sion Limit Over Bud­get”. I cor­re­lated this with the Default Throt­tling Pol­icy Exchange 2010 uses. It seems that the more mail­boxes a user opens the more con­nec­tions Exchange cre­ates. It doesn’t some­how trun­cate these con­nec­tions. To under­stand more about the Default Throt­tling Pol­icy see Under­stand­ing Client Throt­tling Poli­cies. So I quickly whipped up a pow­er­shell script that set the Throt­tling Pol­icy defaults to Null so there was no restric­tions (funny Microsoft states as a workaround just to do this if you encounter an issue).
If you are inter­ested in see­ing this script or want me to go deeper about Throt­tling Poli­cies con­tact me, but this arti­cle isn’t quite about this so I will move quickly on.

After the Throt­tling Pol­icy was changed the reported dis­con­nec­tions stopped but the deliv­ery delays con­tin­ued as men­tioned all around the globe. With the other prob­lem out of the way I began to real­ize that the prob­lem seemed very ran­dom. Some users expe­ri­enced it, some not, some couldn’t tell me whether they expe­ri­enced it or not. This is when the hours of fruit­lessly dig­ging through con­fig­u­ra­tions to learn them and read­ing about Exchange 2010 on Google began. I noticed our mail­box servers were set up in an active — active con­fig­u­ra­tion with bidi­rec­tional repli­ca­tion using DAGs. This is when I decided to go back to basics of trou­bleshoot­ing. I went over to my col­league sit­ting next to me and sent var­i­ous test mes­sages to him. All of them were promptly deliv­ered with­out any prob­lems. I noted down what server his mail­box was run­ning on and moved on. Then I walked around the IT depart­ment until I was able to find a col­league that con­firmed they had the deliv­ery delays up to 4 hours. Just for kicks I turned off their cache mode on the Out­look client and their prob­lem mag­i­cally van­ished. Then I turned cached mode back on and left it bro­ken since I was deter­mined to fix it on the server side and not just band aid the prob­lem. When I went back to my desk and noticed what mail­box server the col­league with the delays was expe­ri­enc­ing a light bulb went off and every­thing seemed to be com­ing together. Now all I had to do was note the dif­fer­ences between the 2 servers.

First of all to stop the global issue from occur­ring while I could resolve the prob­lem I failed all the DAG vol­umes over to the 1 server that did not seem to be hav­ing the prob­lem. Reports quickly came in that the prob­lem was resolved. Then I quickly moved on to exam­in­ing dif­fer­ences between the 2 servers. After com­par­ing win­dows updates between boxes I noticed that some updates from Feb­ru­ary were recently applied to both servers, how­ever, there was 1 dif­fer­ence. It seems Microsoft KB2393802 was applied to one server but not the other. I googled regard­ing this but only found one vague thing about delays in Exchange 2010 mail deliv­ery men­tioned in the mid­dle of a tech­net arti­cle relat­ing to this patch, but noth­ing offi­cial at all from Microsoft. I removed the patch, rebooted, and tested with a test mail­box data­base run­ning on the server I had cre­ated for this pur­pose. The prob­lem was fixed as I thought.

I tried to research on what about this patch could be caus­ing this prob­lem but came up with noth­ing. If any of you read­ers have an idea please com­ment and let me know your thoughts! I have attempted to con­tact Microsoft regard­ing this issue so they could pos­si­bly append to the KB arti­cle but they cur­rently have not replied.

Dur­ing Tech Field Day #6 in Boston one par­tic­u­lar gen­eral fea­ture request has become increas­ingly prominent;

Can we have it inside the vCen­ter client?

In short, what we want is for all those great third party ven­dors like VKer­nel, Solar­winds and oth­ers to be able to put their fea­ture addons directly into the vCen­ter client. Cur­rently most third party apps “inte­grate” by offer­ing a new tab where you can access it, but I would love to see that being expanded even further.

As a con­crete exam­ple, I would love to see for instance VKer­nel iden­tify per­for­mance prob­lems for a VM and then tell me, inside the sum­mary tab for that VM that there is a prob­lem. Show it to me where I do my work, which is in vCen­ter, and mostly on the sum­mary screen. Dash­boards are great, but we’re all suf­fer­ing from dash­boardi­tis, and the more dis­parate dash­boards and tabs we get to have rela­tions with, the less we’re actu­ally able to use them properly.

Also, raise a vCen­ter alert to lever­age my exist­ing alert­ing scheme to get my atten­tion. If a 3rd party plu­gin sees that some­thing is wrong, tell me through the exist­ing infra­struc­ture I already have in place. No need to rein­vent the wheel each and every time, from each and every vendor.

Cou­ple that with my back-end user authen­ti­ca­tion (eg. Active Direc­tory) for sign-on into your solu­tion, and we’re get­ting much closer to the “sin­gle pane of glass” nir­vana sce­nario we’re all craving.

From what I gather the vCen­ter client (I am in no way, shape or form a devel­oper), doesn’t allow this level of inte­gra­tion on the VM sum­mary screen right now but VMware needs to make it pos­si­ble in the future.

We admins want this, and we need it, and frankly, I think we deserve it (and so does the 3rd party ven­dors con­tribut­ing to this ecosystem).

Update:

Read Multi-Hypervisor Man­age­ment and the Future for an updated and more for­ward think­ing post on the same sub­ject matter.

A while ago I got a sur­pris­ing email, stat­ing the following:

You could con­sider this a sin­cere com­pli­ment from the Gestalt IT com­mu­nity as your name was sug­gested and we think you are the kind of per­son we all would love to have as part of our com­mu­nity. This means you’re independent-minded, technology-focused, community-oriented, and a thought leader in the area of IT infrastructure. 

Of course, this is pretty much hog­wash, but nev­er­the­less I’m extremely hon­ored to be invited as a del­e­gate for Tech Field Day #6 in Boston, Mass.

Thank­fully I have a very under­stand­ing employer (and wife!) that pretty much imme­di­ately gave the go ahead and let me take a few days off work to attend.

Look­ing at the del­e­gate list, it’s cer­tain that I’ll be in extremely good com­pany for the event, and there is a good chance we will be able to do some vSoup Live action while in Boston!

Thank you Stephen Fos­kett and the rest of Gestalt IT com­mu­nity who voted me in as a del­e­gate, I’m sure you’re as sur­prised as I am.

Look out Boston, the Del­e­gates They Are A-Coming!

Links

Gestalt Tech Field Day – Boston 6/10 June by Mike Lav­er­ick
vSoup Pod­cast & Tech Field Day 6 by Christo­pher Wells

VMware vCen­ter Oper­a­tions was released to the gen­eral pub­lic a week or so ago and is avail­able for down­load right now. As usual you can down­load a 60 day trial, and get started immediately.

Like other recent man­age­ment util­i­ties from VMware, vCen­ter Oper­a­tions comes in the form of a .OVF tem­plate (like vCMA/vMA).

Installing VMware vCen­ter Operations

Down­load VMware vCen­ter Oper­a­tions and import by start­ing vCen­ter Client, nav­i­gate to the “File” menu and select “Deploy OVF tem­plate…

Browse to the down­load loca­tion, and find the “VMware-vcops-1.0.0.0-373027_OVF10.ova” file. Select it and click open.

Click on “Next” and review the details.

Hit “Next” once more, and click on “Accept” to accept the VMware EULA and enable the “Next” but­ton.

Spec­ify the name and loca­tion of the VMware vCen­ter Oper­a­tions VM, and click “Next” to continue.


Now we get to select which host or clus­ter the VM should be deployed to. Make your choices, and click on “Next

Select your pre­ferred resource pool, if you have any, and once again click “Next

Now select your pre­ferred data­s­tore, and guess what? We get to click “Next” one more time!

Decide if you want a thin or thick pro­vi­sioned VM, the default is thick but I went with thin pro­vi­sioned in this par­tic­u­lar setup.

The last con­fig­u­ra­tion item, for now, is to map the net­works. Select your net­work map­pings and click on “Next”.

Review the final setup screen, and once you are sat­is­fied that your set­tings are cor­rect, click on “Fin­ish” to start the OVF tem­plate import.

The import starts, and after a few min­utes it should be ready to go!


Suc­cess!

Time to start it up!

Con­fig­ur­ing VMware vCen­ter Operations

After the vCen­ter Oper­a­tions VM has fin­ished boot­ing, it dis­plays a lit­tle infor­ma­tion screen show­ing the IP address and other tid­bits of infor­ma­tion. The most impor­tant piece of infor­ma­tion right now is the DHCP assigned IP address. Make a note of that IP for later.

To make sure we don’t run into prob­lems with time syn­chro­niza­tion we need to make sure that the vCen­ter Oper­a­tions VM time is syn­chro­nized with the ESX host time. To do so, right click on the VMware vCen­ter Oper­a­tions VM inside of the vCen­ter Client, select “Edit Set­tings”.

Select the “Options” tab, and find the VMware Tools sec­tion.

Find the “Syn­chro­nize guest time with host” option, and select it.

Open the vCen­ter Oper­a­tions web page in a browser, and log in. The default username/password for vCen­ter Oper­a­tions is admin/admin

Log in, and fol­low the direc­tions on screen to change the default username/password.

The new pass­word must be at least 8 char­ac­ters, and at least one digit and one char­ac­ter.
Note: This also changes the root account pass­word for the vCen­ter Oper­a­tions VM.

Next up is con­fig­ur­ing the vCen­ter Oper­a­tions con­nec­tion to the vCenter.

Fill out the vCen­ter Server infor­ma­tion form, with infor­ma­tion per­ti­nent to your infra­struc­ture.
Note that the reg­is­tra­tion cre­den­tials needs to have admin­is­tra­tor priv­i­leges on the vCen­ter Server. You can use the same cre­den­tials for both reg­is­tra­tion and col­lec­tion, or you can dif­fer­en­ti­ate them if required in your envi­ron­ment.

Click on “Save”, and a test is per­formed to make sure that the infor­ma­tion pro­vided is cor­rect.

If reg­is­tra­tion is suc­cess­ful, a new popup appears explain­ing that you need to use the vSphere Client to assign the vCen­ter Oper­a­tions licenses.

Click on “Ok” and the vCen­ter Oper­a­tions setup dash­board appears in your browser.

Go back to your vCen­ter Client and nav­i­gate to the “Home” screen.

You should now see the new “vCen­ter Oper­a­tions” icon under “Solu­tions and Appli­ca­tions”. If it does not appear imme­di­ately, close the vCen­ter Client and restart it to have it pick up the instal­la­tion.
To install the vCen­ter Oper­a­tions license, go to “Home” and find the “Licens­ing” icon.

Click on it, and change the “View by:” option to “Asset
Right click on “vCen­ter Oper­a­tions” and select “Change License Key

Select “Assign new license key to this solu­tion”, click on “Enter Key…” and enter your license key and option­ally a label for the key. Click on “OK” to return to the “Assign License” win­dow, and click on “OK” again to install the license.


Your license should now be installed and active.

Go back to the vCen­ter Client “Home” screen and find the vCen­ter Oper­a­tions icon under “Solu­tions and Appli­ca­tions”. Click on it, and vCen­ter Oper­a­tions should already be active mon­i­tor­ing your infra­struc­ture.

Thats it! You now have VMware vCen­ter Oper­a­tions run­ning in your envi­ron­ment. For details on how it works and reports your oper­a­tions refer to the VMware vCen­ter Oper­a­tions offi­cial doc­u­men­ta­tion. Eric Sloof has also posted a cou­ple of great videos in his VMware vCen­ter Oper­a­tions — Trou­bleshoot­ing Work­flow post that gives an in-depth overview of what VMware vCen­ter Oper­a­tions is capa­ble of doing.

When Rich Bram­b­ley posted “A Pirate Invented Server Vir­tu­al­iza­tion” today, it reminded me of a lit­tle story from my own pro­duc­tion environment.

This story is a cou­ple of years old, but sadly it’s still valid. A very spe­cial­ized appli­ca­tion that we run, requires a SQL Server Express instance, a proxy/licensing server and client instal­la­tion with license files to work. This appli­ca­tion isn’t very advanced, nor very resource inten­sive so by nature it’s prime for run­ning in a vir­tu­al­ized envi­ron­ment. None of the appli­ca­tion devel­op­ers cus­tomers had ever attempted set­ting it up in a vir­tu­al­ized envi­ron­ment, and so we offered to be a pilot cus­tomer and test it live in our envi­ron­ment. The test­ing con­firmed what we thought, it worked per­fectly when vir­tu­al­ized! We were happy, and so were the devel­op­ers. At least, that’s what we thought.

In fact, what hap­pened next is one of the more bizarre fall outs I’ve per­son­ally seen when it comes to imple­ment­ing a “vir­tu­al­ize first” strategy.

When we got the final ver­sion of the soft­ware, out of the test­ing phase, I got about my busi­ness of installing and con­fig­ur­ing it. All was fine, until I got to the part where I was sup­posed to install the proxy/licensing ser­vice. It turns out that the appli­ca­tion devel­op­ers had put checks in place to pre­vent instal­la­tion on vir­tual machines!

After we tested and ver­i­fied the solu­tion in our envi­ron­ment, the devel­oper turned around and blocked us from imple­ment­ing it as we wished. In fact, we still run this ser­vice on one of the few phys­i­cal servers we still have in place in our data cen­ter. If it were up to me, I would have kicked the appli­ca­tion right out of our envi­ron­ment, but sadly core parts of our busi­ness relies on this appli­ca­tion and we are stuck with it.

The devel­op­ers rea­son­ing for doing this was that it was way to easy to dupli­cate the proxy/licensing ser­vice in a vir­tual envi­ron­ment, and by doing that we could poten­tially bypass the con­cur­rent user license model they had put in place. Their checks are based on a hard­ware id gen­er­ated by the phys­i­cal hard­ware thus it could poten­tially be the same if we dupli­cated the VMs. They could of course work around that, by using the server DNS name as one of the hard­ware id checks or even the NIC MAC address, but sadly they opted to com­pletely block the instal­la­tion and oper­a­tion of that par­tic­u­lar part of their infra­struc­ture if you run virtualized.

Bad call? Very much so, and I have made this very clear to them to no avail.

So, per­haps Dil­bert is right? Was vir­tu­al­iza­tion indeed cre­ated by pirates? I would rather be a Ninja than a Pirate any day.