A Note About Migrating OpenVZ VPSs

I had two primary reasons for adopting OpenVZ to provide server virtualization:

1) I like the idea of being able to "freeze" a complete server and resume it later, especially on different hardware. This leads to some very interesting backup / restore possibilities, in that an entire "server" and the application it is responsible for are all packaged up into one independent unit. I don't actually need virtualization for consolidation reasons.

2) Full blown virtualization solutions are extremely resource intensive, and the added overhead is just not worth the benefit in my case. I would have to buy new servers! OpenVZ solves this problem nicely.

So of course I'm having a good time suspending and resuming servers (checkpointing), and have just recently been playing with migration of servers between hardware nodes. The way that OpenVZ makes this happen is actually not terribly elegant. Nothing about OpenVZ is elegant, and I find there's always a little bit of unrest in the back of my mind about whether OpenVZ isn't going to bite me in the ass sometime down the road.

On to the notes:

When performing a migration between hardware nodes, it's extremely important that the hardware node be configured in the same or similar manner as the node you're migrating from. OpenVZ is not smart enough to configure itself during a migration, so in addition to configuring the server OS itself, you must also configure OpenVZ correctly or your migrations will not happen.

The things that got me were this:

1) Disk quotas. By default an OpenVZ install will be configured with disk quotas turned on, and the default quota is not large (it seems that there is a global quota (the one I'm talking about), and individual per-container quotas). You have to either disable quotas on the new node, or crank them up high enough to accept the VPS you're bringing in. These settings for the global quotas are in vz.conf which is probably somewhere like /etc/vz/vz.conf.

2) For the life of me I could not get a migration to work via a dump file. This concerns me greatly. I'm not sure if you're supposed to create a dummy container on the new node with the same number as the node you're migrating, but the restore wouldn't work without the proper private directory existing. You also need to bring over the VPS's conf file and put it in the right spot ( maybe /etc/vz/conf ) before trying the restore.

The instructions here: http://wiki.openvz.org/Checkpointing_and_live_migration do not work. There are things that need to happen between the dump and the restore that are not addressed in the wiki article.

3) You need to configure networking properly on the new hardware node before migrating. In my case, I had to make sure that iptables was setup the same way, and also mirror settings from the old /etc/sysctl.conf file. At a minimum, make sure you set net.ipv4.ip_forward = 1.

There are more sysctl parameter suggestions in the OpenVZ Users Guide also. (Note: this guide is extremely light on the issue of migration)

4) The online migration option seems to work if you follow exactly the instructions in this wiki article.

Note to Ubuntu Server users: you have to enable the root user on the new node in order to do an online migration. sudo passwd root

Don't be too hasty to assume that online migration is the only tool you need to migrate from one hardware node to another. Consider the case where the old hardware node has caught fire and burns down your entire compound. After you've rebuilt, you'd better make sure that you can restore your backed up VPSs from dump files to your new servers. As I mentioned earlier, I can't seem to make it work.

5) When performing an online migration with default settings, your "old" VPS will be removed!! Be aware of this if you're just playing around! See the options for vzmigrate.

6) OpenVZ error messages are about as useful as a milk bucket under a bull. Turning on verbose doesn't do anything with some commands. I got error messages that do not even exist in the google index! I got error messages that just said "Error:"!

This software works pretty well, but I really don't trust it yet.

Installing JDK in OpenVZ VPS

I started playing with OpenVZ today because I'm getting terrible performance from QEMU+KQEMU and various Java applications.

I had some trouble installing the JDK inside my VPS, and the problem cost me about an hour so I figured I'd share. I'm running OpenVZ from a pretty bare Ubuntu Hardy server, and the VPS was created from ubuntu-8.04-i386-minimal.tar.gz which I downloaded from the OpenVZ site.

When I'd try to install the Sun JDK or OpenJDK via apt-get, I'd get the following error:

Setting up sun-java6-bin (6-06-0ubuntu1) ...
Aborted
dpkg: error processing sun-java6-bin (--configure):
subprocess post-installation script returned error exit status 134
dpkg: dependency problems prevent configuration of sun-java6-jre:
sun-java6-jre depends on sun-java6-bin (= 6-06-0ubuntu1) | ia32-sun-java6-bin (= 6-06-0ubuntu1); however:
Package sun-java6-bin is not configured yet.
Package ia32-sun-java6-bin is not installed.
dpkg: error processing sun-java6-jre (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of sun-java6-jdk:
sun-java6-jdk depends on sun-java6-bin (= 6-06-0ubuntu1); however:
Package sun-java6-bin is not configured yet.
dpkg: error processing sun-java6-jdk (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
sun-java6-bin
sun-java6-jre
sun-java6-jdk
E: Sub-process /usr/bin/dpkg returned an error code (1)

Turns out it's just a resource problem. If you're getting this error, check /proc/user_beancounters and see if you're getting any failures:

cat /proc/user_beancounters

101: kmemsize 625164 1720287 11055923 11377049 0
lockedpages 0 0 256 256 0
privvmpages 1173 129807 131072 139264 2

As you can see I was having problems with the limit on privvmpages. I doubled the default twice before it worked, resulting in a barrier of 262144 and a limit of 278528. You can change these values like so:

vzctl set 101 --privvmpages 262144:278528 --save