From the labs: Building an Ubuntu 14.04 Appliance with VMware Studio 2.6

VMware Studio 2.6 was released way back in March 2012,  and surprisingly there seems to be no new update in sight. While VMware Studio technically still works, even with newer versions of ESXi and vCenter, the supported operating systems for the appliances it can build is somewhat outdated:

  • Red Hat Enterprise Linux 5.5/6.0
  • SUSE Linux Enterprise Server 10.2/11.1
  • Ubuntu 8.04.4/10.04.1
  • Windows Server 2003 R2 / 2008 R2 w/SP1

The Problem:

For a yet to be announced project we are working on internally at EVRY, we needed to build an appliance that was based on newer software packages and development tools. Recent events like heartbleed and shellshock also highlight the need to build new appliances on new and supported distributions.

Attempts at upgrading an existing Ubuntu 10.04.1 appliance to 14.04 failed miserably, due to architectural changes between the Ubuntu versions and how Virtual Appliance Management Infrastructure (VAMI) is installed by VMware Studio and in the end we were pretty much left to two options:

  1. Build the appliance from scratch, and lose VAMI, which was one of the primary reasons for building the appliance with VMware Studio in the first place.
  2. Find a way to build the appliance with Ubuntu 14.04, with VMware Studio.

Option 1 felt a bit like giving up, and option 2, well that was a challenge we couldn’t just walk away from.

The Solution:

Thanks to the brilliant mind of my coworker Espen Ødegaard, we were able to come up with a set of solutions that does the trick.

First we added a new profile to VMware Studio connecting to the VMware Studio VM via SSH, and by issuing the following command:
[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”1″]
studiocli –newos –osdesc “Ubuntu14.04” –profile /opt/vmware/etc/build/templates/ubuntu/10/041/build_profile.xml
[/cc]
Basically this creates a new OS template in VMware Studio by copying the existing Ubuntu 10.04.1 profile.

Next edit line 462 in /opt/vmware/etc/build/templates/Ubuntu14.04/Ubuntu14.04.xsl and change scd0 to sr0.

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
if ! mount -t iso9660 -r /dev/scd0 /target/${cdrom_dir} ; then \
if ! mount -t iso9660 -r /dev/hda /target/${cdrom_dir} ; then \
cdrom_mounted=0; \
fi ; \
[/cc]

This fixes a problem with the appliance not being able to mount the Ubuntu installation iso when it is built. Place the Ubuntu 14.04 ISO in /opt/vmware/www/ISV/ISO, and create a new Build profile using the VMware Studio Web Interface.

Now, before an appliance can be built, we need to fix some other problems that prevent VAMI from starting up, and preventing login to the VAMI web interface. First of, make sure that libncurses5 is added as package under Application -> List of packages from OS install media. Next, add the following to the first boot script under OS->Boot Customization, we can move around those problems.

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
# Create symlinks required for Ubuntu 14.04 and VAMI

# Copy and symlink libncurses to the location VAMI looks for them

cp /lib/i386-linux-gnu/libncurses.so.5.9 /opt/vmware/lib/
rm /opt/vmware/lib/libncurses.so.5
ln -s /opt/vmware/lib/libncurses.so.5.9 /opt/vmware/lib/libncurses.so.5

# Symlink PAM libraries in order for them to work with VAMI
# This “unbreaks” authentication in the web interface
mkdir /lib/security
ln -s /lib/i386-linux-gnu/security/* /lib/security/
[/cc]

For details on how all of this works, and how to create build profiles, check the official VMware Studio documentation.

The build process should now complete successfully, and you should have an Ubuntu 14.04 based appliance built with VMware Studio!
vStealth1
vStealth2

Note that all of this is completely unsupported by VMware, and you are pretty much on your own. Hopefully there will be a new version of VMware Studio available soon, and we won’t have to rely on unsupported hacks to get it working with newer operating systems.

VMware vShield Manager Upgrade – Password Issues

While upgrading a vShield Manager 5.1.1 install to 5.1.4 at a client, I ran into an issue with logging in after a completed upgrade. The username and password used to log in, and subsequently upload the upgrade file, was no longer working after the upgrade finished and the vShield Manager appliance had been rebooted.

vShieldManagerError

It turns out that this was due to using international characters in the password for the admin user, in this case the Norwegian specialities æ, ø or å.

I was “lucky” enough to have taken a snapshot of the 5.1.1 vShield Manager appliance before performing the upgrade. After reverting back to the pre-upgrade snapshot, logging in, changing the password to one not containing international characters,  and then performing the upgrade procedure again the appliance was upgraded to 5.1.4 and logon could be done with out problems.

Be careful when using international characters in passwords for any of the VMware appliances, as I suspect this might be an issue also for other components, not only the vShield Manager.

vPostgres Database Backup in vCSA 5.5

With the new vSphere 5.5 release, the VMware vCenter Appliance (vCSA) has grown up to be a viable alternative to the traditional Microsoft Windows based vCenter deployment scenario. The new vCSA version supports up to 100 hosts and 3000 (with an external Oracle database the values change to 1000/10000) virtual machines, a big improvement from 5 hosts and 50 virtual machines in the previous version.

Sadly, the only external database option available for vCSA 5.5 is Oracle, which means there is still no external Microsoft SQL Server support. For those clients who don´t have an existing Oracle infrastructure, this might be a problem especially with regards to backup of the vCSA database.

The internal database in vCSA is a modified version of Postgres, that VMware has called vPostgres. Thankfully this also includes the native Postgres command to create database dumps, pg_dump.

In order to create a database dump of the vPostgres database, the following command needs to be run by opening a SSH connection to the vCSA:

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
~ # /opt/vmware/vpostgres/1.0/bin/pg_dump EMB_DB_INSTANCE -U EMB_DB_USER -Fp -c > VCDBBackupFile
[/cc]

EMB_DB_INSTANCE and EMB_DB_USER are to be replaced with values from the /etc/vmware-vpx/embedded_db.cfg file

In my case, the exact command would be:

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
~ # /opt/vmware/vpostgres/1.0/bin/pg_dump VCDB -U vc -Fp -c > /tmp/VCDBBackupFile
[/cc]

Note that EMB_DB_INSTANCE is replaced with VCDB and EMB_DB_USER is replaced by vc in the command above. Both values come from /etc/vmware-vpx/embedded_db.cfg

Off it goes, and creates a dump file in /tmp. So far so good, we have a database dump but how to we automate it?

Add a new file in /etc/cron.daily/ (for instance, use /etc/cron.hourly/ if that fits your RPO) called vcdbbackup.sh with the following content:

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
#!/bin/sh
/opt/vmware/vpostgres/1.0/bin/pg_dump VCDB -U vc -Fp -c > /tmp/VCDBBackupFile
[/cc]

Then we need to make the cron script executable by running:
[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
chmod u+x /etc/cron.daily/vcdbbackup.sh
[/cc]

And you should be ready to go! Of course, you should probably create the backup files somewhere else than in /tmp, and even mount a file share from another server in your environment and place the dump file there for safe keeping and further backup in your existing scheme. In addition to this, you can add other variables to the script like datestamping the file etc, but for now this is what I have done.

By doing backups of the vCSA in this manner, you can have a standby vCSA laying around in case of a primary vCSA failure. If that happens, fire up the standby one, ssh to it and restore the vPostgres dump. I won’t go into details on that right now, but the command for restoring looks like this:

[cc lang=”bash” width=”100%” theme=”blackboard” nowrap=”0″]
PGPASSWORD=EMB_DB_PASSWORD ./psql -d EMB_DB_INSTANCE -Upostgres -f VCDBBackupFile
[/cc]

Feature Request
What VMware should do for the next version of the vCSA is to add external Microsoft SQL Server support, but if that´s off the table for some reason, at least let us manage database dumps via the vCSA admin interface? Please create pre-defined scripts and crontabs, and let us manage those. It might not be as good as external database support, when it comes to backup, but a little goes a long way in protecting this valuable resource in your infrastructure.

References:

Backing up and restoring the vCenter Server Appliance (vPostgres) database (2034505)