First things first, I want to upgrade vCloud Director to what exactly ? The current latest version is vCloud Director 5.5, but I will only show the upgrade to 5.1.3 .. why ? Because first, an upgrade to 5.1.x is mandatory anyway when coming from version 1.5, and second, I never run into any issues when upgrading vCloud Director 5.1.x to 5.5 to be honest.
The main problems I always faced was getting to 5.1.x to begin with 🙂
There are a lot of articles out there in how to upgrade vCloud Director 1.5 – but to be fair – the upgrade of vCloud Director itself is quite straight forward. It is just a matter of upgrading the binaries – which the installer does for you – upgrade the DB – which it says how to do – and that is it.
When I was working on an multiple-site upgrade project, this is what most guides were talking about – the upgrade itself. The problem with vCloud Director though is, there are a lot of other components you need to consider. For example, the upgrade usually worked fine but the vShield Manager didn’t and so on.
So here I am trying to go for a whole upgrade process. Obviously every environment is different, but the guide here should be comprehensive enough for you to get an idea what to look out for.
The first important thing you need is making sure you actually know what components you have in your environment. The reason for it is that you need to upgrade each component in the correct sequence ! If you don’t, you will break things.
Important Read:
Update sequence for vSphere 5.x and its compatible VMware products (2059455)
Update sequence for vSphere 5.5 and its compatible VMware products (2057795)
Also make sure you read the actual Release Notes.
This is important in order to understand limitations, known issues etc. I have seen setups where the upgrade would have broken the environment. Needless to say – test .. test it all .. and everything .. I mentioned testing, right 😉 ?
The Release Notes also include links to the install and upgrade documentations so really make sure you read that. So .. I mentioned testing didn’t I 😀
Another important document you need to read
VMware Product Interoperability Matrixes
You need to make sure that your new vCloud Director version is actually compatible with your current build of ESXi, vCenter etc.
Anyway, let’s get started.
I am using a nested environment for testing. The “to be upgraded” vCloud Infrastructure consists of
– One vCenter Server (vCenter 5.0.0 Build 1300600)
– Two ESXi hosts (ESXi 5.0.0 Build 1311175)
– Two vCloud Director Cells (vCD 1.5.2 Build 924077)
– One vShield Manager (vSM 5.0.2 Build 791471)
Once all the VMs have been installed and configured, I created a test Organization, a test Organization vDC and made sure the virtual network used is routed rather than directly connected to a port group.
This way I can make sure that after the upgrade, the vShield Manager (and subsequent the vShield Edges) are still working as desired.
Based on the above KBs from VMware, the following upgrade sequence applies:
1. vCloud Director Cells (both of them obviously) 2. vShield Manager (in later versions called vCloud Networking and Security) 3. vCenter Server (out of scope of this guide really) 4. vSphere ESXi (also out of scope, including the VMware Tools you should upgrade at this stage) 5. vShield Edge
Now there are a few gotchas when it comes to upgrading / re-deploying vShield Edge I’d like to show.
As mentioned in the official upgrade Sequence document, vCenter and ESXi should be upgraded BEFORE the vShield Edges.
I am honestly not sure why this is. I don’t see the reasoning myself (please comment if you do). The production environment I worked on was quite large so each step pretty much needed a maintenance window. So whilst the vCenter and ESXi upgrade was scheduled for a later date, the vShield Edges had sometimes a different idea.
Some just died and had to be re-deployed, some customers “reset” their routed network, which also in turn re-deploys the Edges, and so on. So by the time you actually get to the point of upgrading your vCenter and ESXi, the majority of Edges have already been upgraded. Plus sometimes you are unable to perform any changes until you do upgrade the Edges, so if a customer needs to change firewall or NAT rules, you are pretty much forced to upgrade the Edges and you can’t really wait for the ESXi / vCenter upgrade.
Anyway, here I will skip the ESXi and vCenter upgrade, but you know how you should do it 🙂
vCloud Director Upgrade / Cell Upgrade
Before you start running the new vCloud Director Binary, you need to run a script.
The script won’t change anything in the VMs etc., it just fixed database references. If you don’t do this, VMs might lose network connectivity once you upgraded. Or rather, affected VMs will lose connectivity.
Here an example output of the script in ‘resolve’ mode. If unsure run the script in detect mode (./NicTypeValidator.sh <Full Path of db.properties file> detect). This will just show the conflicts in the log file without performing any changes in the database.
Loaded properties from db.properties
NIC type validation script set to run in ‘resolve’ mode
Connected to DB successfully!
NIC type MISMATCH for [VM – Ubuntu 11.10 64 Bits(vm-25618), VAPP – Ubuntu 11.10 64 Bits,Org – London-Templates,GuestOS -ubuntu64Guest,MAC – 00:50:56:01:01:11,VCD nic type = VMXNET3, VC nic type = E1000]
SQL – UPDATE network_interface SET nic_type = 3 WHERE CONVERT(varchar(255),vm_id,2) = ‘B77CB32A6E564F1BBD56E8682FE151E0′ AND mac_address = ’00:50:56:01:01:11’ resulted in 1 row(s) update
NIC type MISMATCH for [VM – CentOS 6.2 64 Bits(vm-22427), VAPP – CentOS 6.2 64 Bits,Org – London-Templates,GuestOS -centos64Guest,MAC – 00:50:56:01:01:0e,VCD nic type = VMXNET3, VC nic type = E1000]
SQL – UPDATE network_interface SET nic_type = 3 WHERE CONVERT(varchar(255),vm_id,2) = ‘4737166C83AB4F58A423AD5C62945D20′ AND mac_address = ’00:50:56:01:01:0e’ resulted in 1 row(s)update
Next step would be to suspend the cells .. A good idea is also to have backups (or if virtual, snapshots) of both, vCloud Cells and DB.
On both your cells check if there are any active jobs
[[email protected] bin]# ./cell-management-tool -u administrator cell -t
Please enter the administrator password:
Job count = 0
Is Active = true
[[email protected] bin]#
This is good. No current jobs so I am ok to suspend the cell. Killing the cell while a job is running could cause jobs stay hanging in the DB, requiring you to dig around SQL, trying to get rid of it. So better save than sorry.
1. Quiesce Cell
[[email protected] bin]# ./cell-management-tool -u administrator cell -q true
Please enter the administrator password:
[[email protected] bin]#
2. Shutdown Cell
[[email protected] bin]# ./cell-management-tool -u administrator cell -s
Please enter the administrator password:
[[email protected] bin]#
3. Stop Service
[[email protected] bin]# service vmware-vcd stop
Stopping vmware-vcd-watchdog: [ OK ]
Stopping vmware-vcd-cell: [ OK ]
[[email protected] bin]#
Now you can start the upgrade process.
First, run the installer
[[email protected] ~]# ./vmware-vcloud-director-5.1.3-1881640.bin
Checking architecture…done
Checking for a supported Linux distribution…You are not running a Linux distribution supported by vCloud Director.
Would you like to proceed anyway? [y/n] y
Will attempt to install vCloud Director in unsupported mode. The installation
may fail if the necessary RPM prerequisites are not already installed.
Checking free disk space…done
Extracting VMware vCloud Director. Please wait, this could take a few minutes…
vmware-vcloud-director-5.1.3-1881640.x86_64.rpm
done
Verifying RPM signatures…done
An older version of VMware vCloud Director has been detected. Would you like
to upgrade it to 5.1.3? The installer will stop the vmware-vcd service,
back up any configuration files from the previous release and migrate the
product configuration as necessary.Would you like to upgrade now? (y/n)? y
Upgrading VMware vCloud Director…
Installing the VMware vCloud Director 5.1.3 RPM…
warning: vmware-vcloud-director-5.1.3-1881640.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 66fd4949: NOKEY
Preparing… ########################################### [100%]
1:vmware-vcloud-director warning: /opt/vmware/vcloud-director/etc/global.properties created as /opt/vmware/vcloud-director/etc/global.properties.rpmnew
########################################### [100%]
Update completed. You will need to manually start the vCD services using
the following command: service vmware-vcd start
doneUpgrade installation complete.
Next steps:You will need to upgrade the database schema before starting the
vmware-vcd service. The product upgrade tool should be run only once per
vCloud Director group. The tool may be run with the following command:
/opt/vmware/vcloud-director/bin/upgrade
The result above is quite clear what is next – upgrade the database (you do this only on your first cell).
[[email protected] ~]# /opt/vmware/vcloud-director/bin/upgrade
Welcome to the vCloud Director upgrade utilityThis utility will apply several updates to the database. Please
ensure you have created a backup of your database prior to continuing.Do you wish to upgrade the product now? [Y/N] Y
Examining database at URL: jdbc:jtds:sqlserver://sql:1433/vcd-vcloud;socketTimeout=90;instance=
Running 337 upgrade tasks
Executing upgrade task: Start UpdateStatementManager
…[3]
Successfully ran upgrade task
Executing upgrade task: Upgrade to 2.0.0<snip>
Database upgrade complete
Do you wish to rebuild the database indexes? This may take several minutes. [Y/N] y
Rebuilding database indexes
…[3]
Done rebuilding indexes
Do you wish to update the database statistics? This may take several minutes. [Y/N] y
Updating database statistics
..[2]
Unable to update database statistics. Database user has insufficient privileges to update database statistics. To complete this step manually, run ‘EXEC sp_updatestats’ as the DBO or a member of the sysadmin fixed server role.Would you like to start the vCloud Director service now? If you choose not
to start it now, you can manually start it at any time using this command:
service vmware-vcd startStart it now? [y/n] n
As you can see above shows an error that the DB user does not have sufficient rights to upgrade the statistics. I therefore selected “n” when it asked me whether to start the service.
This is fine. You can avoid this by setting the DB user as a sysadmin on the SQL server. But you might forget to remove the permissions again, so you may as well just run it manually in SQL Management Studio (Plus your DBA might not like the idea 😉 )
Now go ahead and start the vCloud Cell service
[[email protected] ~]# service vmware-vcd start
Starting vmware-vcd-watchdog: [ OK ]
Starting vmware-vcd-cell [ OK ]
[[email protected] ~]#
And as mentioned – you do this only on the first cell. Now upgrade the second cell with the same binary but skip the DB upgrade.
At this stage your vCloud interface should be up and running. VMs should never have stopped pinging and all should be well.
As always, make sure you test everything you can think of before proceeding.
vCloud Director Upgrade Content
1. Upgrading vCloud Director Binaries / Database
2. Upgrading vShield Manager
3. Final Upgrade Touches