A Note About Migrating OpenVZ VPSs

I had two primary reasons for adopting OpenVZ to provide server virtualization:

1) I like the idea of being able to "freeze" a complete server and resume it later, especially on different hardware. This leads to some very interesting backup / restore possibilities, in that an entire "server" and the application it is responsible for are all packaged up into one independent unit. I don't actually need virtualization for consolidation reasons.

2) Full blown virtualization solutions are extremely resource intensive, and the added overhead is just not worth the benefit in my case. I would have to buy new servers! OpenVZ solves this problem nicely.

So of course I'm having a good time suspending and resuming servers (checkpointing), and have just recently been playing with migration of servers between hardware nodes. The way that OpenVZ makes this happen is actually not terribly elegant. Nothing about OpenVZ is elegant, and I find there's always a little bit of unrest in the back of my mind about whether OpenVZ isn't going to bite me in the ass sometime down the road.

On to the notes:

When performing a migration between hardware nodes, it's extremely important that the hardware node be configured in the same or similar manner as the node you're migrating from. OpenVZ is not smart enough to configure itself during a migration, so in addition to configuring the server OS itself, you must also configure OpenVZ correctly or your migrations will not happen.

The things that got me were this:

1) Disk quotas. By default an OpenVZ install will be configured with disk quotas turned on, and the default quota is not large (it seems that there is a global quota (the one I'm talking about), and individual per-container quotas). You have to either disable quotas on the new node, or crank them up high enough to accept the VPS you're bringing in. These settings for the global quotas are in vz.conf which is probably somewhere like /etc/vz/vz.conf.

2) For the life of me I could not get a migration to work via a dump file. This concerns me greatly. I'm not sure if you're supposed to create a dummy container on the new node with the same number as the node you're migrating, but the restore wouldn't work without the proper private directory existing. You also need to bring over the VPS's conf file and put it in the right spot ( maybe /etc/vz/conf ) before trying the restore.

The instructions here: http://wiki.openvz.org/Checkpointing_and_live_migration do not work. There are things that need to happen between the dump and the restore that are not addressed in the wiki article.

3) You need to configure networking properly on the new hardware node before migrating. In my case, I had to make sure that iptables was setup the same way, and also mirror settings from the old /etc/sysctl.conf file. At a minimum, make sure you set net.ipv4.ip_forward = 1.

There are more sysctl parameter suggestions in the OpenVZ Users Guide also. (Note: this guide is extremely light on the issue of migration)

4) The online migration option seems to work if you follow exactly the instructions in this wiki article.

Note to Ubuntu Server users: you have to enable the root user on the new node in order to do an online migration. sudo passwd root

Don't be too hasty to assume that online migration is the only tool you need to migrate from one hardware node to another. Consider the case where the old hardware node has caught fire and burns down your entire compound. After you've rebuilt, you'd better make sure that you can restore your backed up VPSs from dump files to your new servers. As I mentioned earlier, I can't seem to make it work.

5) When performing an online migration with default settings, your "old" VPS will be removed!! Be aware of this if you're just playing around! See the options for vzmigrate.

6) OpenVZ error messages are about as useful as a milk bucket under a bull. Turning on verbose doesn't do anything with some commands. I got error messages that do not even exist in the google index! I got error messages that just said "Error:"!

This software works pretty well, but I really don't trust it yet.

DanielStallmanAliceLippmann/Richard Stallman

Would everyone please stop calling DanielStallmanAliceLippmann/Richard Stallman "Richard Stallman?" After all, Daniel Stallman and Alice Lippmann were the first to start development on DanielStallmanAliceLippmann/Richard Stallman. It's only fair to give credit where credit is due.

FACTOID: Did you know that 50% of DSAL/Richard Stallman was created by Daniel Stallman, and the other 50% by Alice Lippmann? Richard Stallman is responsible for 0% of the creation of Richard Stallman, yet he still runs around calling himself Richard Stallman.

DSAL/Richard Stallman, for those no in the know, is responsible the GNU/Linux operating system. The GNU/Linux operating system is Linux with a GNU and a forward-slash in front of it.

rm -rf /opt/liferay

Sheesh, another Liferay install down the tubes. This is getting ridiculous.

Unfortunately I can't find another portal application with a half way decent document management system included. If anyone knows of anything, let me know. I'm currently downloading the just-released Liferay 5.1.1 which I'll be hating in approximately ten minutes.

Here's some advise for anyone who is thinking about using Liferay:

1) Liferay is extremely convoluted. The end-user system is organized the way a Java programmer might organize his source files. This does not lend itself to useability. Your users, if not extremely competent, won't know what the hell is going on or where they are in the system half the time.

2) Liferay is extremely customizable, but not in a meaningful way. The concept of a "personal space" and a "public" space within the portal all exist, as you might expect, but merging the two concepts into one concise "website" is very difficult.

3) The permissions system has fine-grained itself into sand. You have users, roles, groups, communities, and organizations all duking it out at once over private and public pages organized by user, community, and organization.

A prime example of how this fails is this: you can add users to an organization, and then add organizations to a community. This sounds good, especially for an extranet application. However, just because a user belongs to an organization that belongs to a community does not mean that the various portlets within the community will recognize that the user is part of the organization that belongs to the community. Whew, that's a mouthful. Here's what I mean: if you create a community-wide announcement, users who only belong to organizations that belong to the community will not see the announcement, because the system doesn't realize that they are implicitly part of the community. One way around this is to create a catch-all user group where you dump all of the users who belong to the various organizations that are part of your community. You should also then create a group for each organization in case you want to send announcements to individual organizations through the same announcement instance. Ok, so you're now managing three groups in addition to your community and organizations. Why? Who the hell knows. It's a nightmare.

4) Extending the system is a horrible thing. Just don't, unless it's going to be your full time job. Every single Java technology that has ever escaped from Java Hell is used behind the scenes. You're going to have to know them all. In depth. There are so many references to references to references that you're going to have half a dozen files open just trying to figure out how to make one silly change. Be careful, do something wrong, even in an external portlet, and you could break your install. (why do you think I'm here bitching while waiting for 5.1.1 to download!?) And of course in Java web development style, every little change you try out means restarting the application server, regardless of whether it's in an actual java class or just raw HTML in some template. Wearing a watch? Don't.

5) Documentation was good at around 4.4. Now it sucks. And the fact that liferay.ca expired and the last Wiki overhaul broke every link in the universe, don't even bother using Google. The Wiki isn't getting updated very well. Development documentation is awful. Just awful. The days of the Lifecast seem to be over.

6) The community is virtually non-existant. I would speculate that 70-90% of all questions asked in the forums go unanswered or unresolved. People posting code examples don't often check their work (because it would take about an hour to do so!). The forum software itself doesn't handle code examples at all because they're using a stupid fixed width template.

7) It's a resource hog. Most Java portals are. It's amazing at all the code that runs just to render a typical payload. They've got all their bases covered, for sure, having used every existing technology all at once, but unfortunately there's only one guy playing, and he's standing in right field with his finger in his nose looking up at some birds.

7) What's with the Christian portlets? I mean, Jesus H Christ, nobody ever mentions this. It's a bit odd. Is it a statement? "Look, we know this shit is hard... so here's a Prayer Portlet."

The sad thing is that Liferay seems to be miles ahead of other opensource portal solutions. Nothing would make me happier than a complete feature freeze and a complete examination of how the system is actually being used by, like, people 'n stuff. I really think that all of the features are there, they just need to work properly and take into consideration more than one use case.

As it stands I am very close to keeping Sharepoint as my internal portal even though I really don't like being locked into Microsoft products. The project I'm currently using Liferay for is an extranet and not entirely critical. Based on what I've learned from the experience there's almost no way that I can deploy this software internally and expect it to function well in the short or long term. (BTW, this little project of mine is supposed to be demoed tomorrow. Haha. haha ahdhhahaha haha ah ah ahha aha ah aha aaaaaahhhhhhhhhhhhhhh ha.)

Oop, my download is done. Ahh, nothing like a fresh install to reinstate my sense of hope.

Back in ten minutes...

Oh GNU You're So Awesome!

It's called "Linux", get over it.

We're all comfortable with it. We all know what it means. Tooting your own horn just makes you sound like, well, someone tooting his own horn.

Look, we're all impressed that GNU is responsible for about 30% of any "Linux" operating system. And thankfully, the remaining majority of contributors don't seem to be quite so self-obsessed.

Here's a tip: you should have picked a better name.

Really, Mr. Stallman, do you lay awake at night thinking of such petty things?

Alfresco - "Commercial Open Source"

So I don't want to sound like too much of a hater, although the purpose of this blog is to give me a place to vent, but I'm very unimpressed with Alfresco's "open source" business model.

While the Church of the Free Software Foundation would have us believe that source code is a basic human right, other companies like Alfresco have a more restrictive view when it comes to the licensing of their code. And really I'm ok with that. Just because something is trivial to duplicate and distribute does not mean that it has no monetary value. Work is work.

What I do have a problem with are various voices from the Alfresco blogs ripping on closed source vendors and patting themselves on the back as if they were on the same level as "real" open source companies like, for instance, the Apache Software Foundation. There's a big difference between commercial open source companies like Alfresco and less restrictive entities like Apache.

Alfresco maintains two open source branches of their software. One for the so-called community, and one for the Starship Enterprise (i.e. paying customers). What's wrong with that?

Well, for starters I've come across bugs in Community version of Alfresco that I know are fixed in Enterprise. Even though the Community or "Labs" version of Alfresco continues to plow along and is nearly at version 3.0, show-stopper bug fixes from Enterprise 2.1+ are still not merged into Community. Ok, it's open source, so I can fix them, right?

Let me ask you this: why should I? If I were to fix the JSR-168 authentication flaw that's holding me back from actually using Alfresco, then I would be duplicating work that has already been done. Not only that, but the severity of the flaw would almost certainly mean that my fix would be of lesser quality than the fix that already exists because I am not an Alfresco developer. Even if I did fix it, and fix it well, there's nobody to give my fix back to because Alfresco doesn't take a lot of community contributions, and (have I mentioned this?), it's already fixed.

So let's talk about the Alfresco Community for a moment. Where is it? I'm seeing an outdated Wiki and some very inactive forums where moderators spend more time moving threads around to "appropriate" forums than answering questions. The use of the word Community here is a bit of a stretch, and has no greater value than I'd expect to get from a closed-source product. The Alfresco Community is less of an "open source community" and more like a gathering of frustrated but oddly grateful beta testers.

In addition to these not-so-opensourcey qualities of Alfresco is the biggest slap in the face of all: the cute little "nag screen" at the bottom of every page rendered by the Community version. You know, the one that reads: "This software might not be any good because the company running it hasn't paid any money." While much of the look and feel of Alfresco can be easily changed, this little gem is actually buried in the Java source. On purpose. To irritate you into paying. Cute.

"Dude, why don't you just shut up and pay for the software?"

I'm not convinced that it's worth paying for yet. If Alfresco has a similar pricing structure to other open source commercial/support companies, then it's probably more expensive than its closed source competitors. Whoa, did I say that? Yeah. Despite what you might have been led to believe, products like Sharepoint from terrible companies like Microsoft are actually dirt cheap for small to medium sized companies. Sure, you don't get free support with the license fees, but at least I'm not forced to pay extra for something I haven't to this day had to use.

So why should I fork over the cash if the "demo" version I'm using doesn't even correctly implement a fairly straight-foward feature? A feature that is actually specified in a formal standard no less!

"Yeah, but it's worth the price because it's open source, which is all open and not all closed 'n' stuff."

I love open source, but in this case I'm not really sure what the benefit is. I don't use open source software because I'm afraid that some day the vendor will go out of business and I'll be able to continue modifying the software myself. I'm certainly not going to just assume that the "community" is going to pick it up and save the day. I do like open source because I can modify the behavior of the software and fix bugs if need be, but this is really something that I rarely want to do. I'm interested in the software because I want to use the software. And I certainly don't want to be fixing bugs in software that I'm paying support fees for.

If an open source application isn't fully open source then its appeal is greatly diminished. The community around it is diminished. It's really only slightly more interesting than closed source alternatives.

"Ok, so you're still saying that Alfresco is more intersting than its closed source competitors. What the hell is your problem?"

Well, frankly I'm just sick and tired of the Alfresco bloggers ripping on closed source companies. Alfresco is not a community driven open source project, and I certainly don't feel that they have any business taking the moral highground and ripping on their closed source competition. Suck it up, Alfresco, and open the the real source. Microsoft might have its head up its ass when it comes to understanding the benefits of open source, but you've only got one eye peeking out.

"Does it make you feel better to make up these insulting questions that I, the reader, am supposedly dying to ask you?"

Why, yes. Yes it does. Thanks.

DIE WIKI DIE

A Wiki is good for recording and storing all of the shit that comes out of your brain.

A Wiki is a terrible place for managing and showcasing all of the shit that comes out of your brain.

The Wiki love affair has got to stop. Please, make it stop. At least stop using Wikis as your primary technical support system.

1) Every wiki is out of date. It doesn't matter how much information you've got in your Wiki if much of it no longer applies.

2) Wikis have no good mechanism to determine what content is out of date. They also have no mechanism to convey to the reader that the information may be out of date. A Wiki relies solely on the writer to maintain the validity of the information, and the writer can't do that, as evident by his use of a Wiki to manage information. What's so bad about out-of-date information? It costs your consumers time. Lots and lots of time.

3) Wikis encourage "yada yada" explanation.

4) Wikis disrupt discussion forums, which are vastly more immediately useful. (Reply: it's in the wiki).

5) Wikis encourage bad information management through the very feature that makes a Wiki unique and useful.

6) Wikis are often difficult to navigate and search, as Wikis are often designed with the Wiki writer in mind.

In conclusion, I hate Wikis. If you need a helpdesk application, then use one.

Liferay + Alfresco = Lalfrescoray

Hey what's that, some egg on my face?

My comments about the Liferay WebDAV support in this whiny article are incorrect. As of Liferay 5.1 WebDAV does seem to work since I replaced a goofy NIC in my server. Please see the article comments for more information.

*************************************

Once again I find myself stuck between two partial solutions that I can't combine to solve what seems to me to be a rather simple problem. Here it is:

1) I need a "portal".
2) I need a document management "portlet".
3) I must access the document management system via CIFS or WebDAV.
4) The system must be somewhat seamless; single sign on at least.

After reviewing some options, I decided that Liferay should be able to meet my requirements. It's a "portal" with lots of "portlets" included, including document management. (notice my sarcastic quotation marks)

DENIED: Liferay's WebDAV doesn't work well enough to be considered well enough.

Ok ok, I've heard a lot of talk on the ol' blogosphere about integrating Alfresco and Liferay. From all the buzz it must be a pretty good solution, especially with all this "web script" stuff on the Alfresco side. Plus Alfresco has some great features that I can take advantage of, and I know that it's CIFS support works because I've used it in the past.

FAIL. Liferay and Alfresco do not mix. Like, not at all. Well, not if you want to use Alfresco 2.1 Community or better. (BTW, the latest Liferay Alfresco Content Portlet 5.0.0.1 uses Alfresco 2.0) Ok ok, I'm being a little too negative. They do mix I'm sure if you write your own authentication filter to make SSO work. No problem!

You see, both Liferay and Alfresco love talking it up about standards support, like JSR-168, but it seems that neither of them pull it off well enough to, you know, INTEROPERATE.

The guy (guys?) at Cignex have a supposed solution using CAS between the two, but Alfresco doesn't jive with CAS as easily as Liferay, and I couldn't get this working reliably for SSO. I even bought this book called "Liferay Portal Enterprise Intranets" by Jonas Yuan in anticipation of the Alfresco integration section. What a complete bust and $60 down the drain. The book isn't necessarily bad, but the devil is in the details and the book is unfortunately devil-free.

Not suprisingly, another solution is to simply purchase Alfresco Enterprise, because it uses a different code base than community. In fact, Alfresco Enterprise 2.2+ will work with Liferay while Alfresco Community 2.9B will not. Open source? Sorta.

Frankly, I'm a little bit suprised to see the world of open source portals and document mangagement systems still in such a ridiculous state. Liferay has more bugs than a volkswagon dealer and both Liferay and Alfresco are stupidly difficult to configure in any interesting way, as all Java-based web applications seem to be. XML NIGHTMARE! Here's a tip: if it takes weeks just to understand how to configure your software, then your software isn't finished. Excessive meta configuration files are not a feature. Most of your users don't give a shit about Java Beans or any other beans for that matter.

Community? Not very good in either camp. Outdated documents, wikis (DIE WIKI DIE) and very poor forum support.

Not to just pick on Liferay and Alfresco, I've also tried out Exo and Icecore (built on Liferay) with limited success. I've looked at the JBoss portal a few times also but I just can't get over them basing their forum portlet on phpBB. I know, sounds petty, but phpBB....jesus...I don't want to think about this anymore. Bye.

Oh, and one more thing (damn I'm frustrated). HTTP Basic Auth is NOT a good solution for any feature of any software intended to be used in the Enterprise. (yes, including the Starship Enterprise) Pushing it over SSL only makes transport safer, and doesn't solve the root problem. (and we're just self-signing anyhow, even if we don't like to admit to it) And entering a password into a browser is NOT a good solution, ever. Receiving a password from a web browser is NOT a good solution. This is all the more true on the Starship Enterprise where people are either using potentially weak SSO or making all of their passwords match. The last thing you want are domain passwords in your damn browser cookies. I'm looking at you, REST authentication appologist!