Liferay + Alfresco on OpenVZ

This post is similar to my last entry, as it deals with Java apps and resource settings in OpenVZ. Today I tried to fire up Liferay 5 with the Alfresco 2.1 WAR on Tomcat 5.5, and kept getting the following OutOfMemoryError:

INFO: Starting Coyote HTTP/1.1 on http-8080
Jul 10, 2008 7:50:07 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.(ThreadPool.java:648)
at org.apache.tomcat.util.threads.ThreadPool.openThreads(ThreadPool.java:520)
at org.apache.tomcat.util.threads.ThreadPool.start(ThreadPool.java:149)
at org.apache.jk.common.ChannelSocket.init(ChannelSocket.java:436)
at org.apache.jk.server.JkMain.start(JkMain.java:328)
at org.apache.jk.server.JkCoyoteHandler.start(JkCoyoteHandler.java:154)
at org.apache.catalina.connector.Connector.start(Connector.java:1090)
at org.apache.catalina.core.StandardService.start(StandardService.java:457)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:700)
at org.apache.catalina.startup.Catalina.start(Catalina.java:552)
... 6 more

This error turned out to be a bit deceptive and had me needlessly fiddling with Java and OpenVZ memory settings for a time. I should have known to check /proc/user_beancounters right away. If I had, I would have noticed that I was hitting the limit on the numproc parameter, which is set to a very low 240. I increased the limit to to 1000 (just to get an idea of how many processes I would need) using the following command:

vzctl set 101 --save --numproc 1000:1000

Turns out that Liferay with the Alfresco plugin on Tomcat 5.5 using MySQL for everything uses up 248 processes on my machine just to start.

Anyhow, the OutOfMemoryError was due to the process limit and had nothing to do with memory. Just thought I'd share in case anyone else runs into this and Googles before troubleshooting properly.

Installing JDK in OpenVZ VPS

I started playing with OpenVZ today because I'm getting terrible performance from QEMU+KQEMU and various Java applications.

I had some trouble installing the JDK inside my VPS, and the problem cost me about an hour so I figured I'd share. I'm running OpenVZ from a pretty bare Ubuntu Hardy server, and the VPS was created from ubuntu-8.04-i386-minimal.tar.gz which I downloaded from the OpenVZ site.

When I'd try to install the Sun JDK or OpenJDK via apt-get, I'd get the following error:

Setting up sun-java6-bin (6-06-0ubuntu1) ...
Aborted
dpkg: error processing sun-java6-bin (--configure):
subprocess post-installation script returned error exit status 134
dpkg: dependency problems prevent configuration of sun-java6-jre:
sun-java6-jre depends on sun-java6-bin (= 6-06-0ubuntu1) | ia32-sun-java6-bin (= 6-06-0ubuntu1); however:
Package sun-java6-bin is not configured yet.
Package ia32-sun-java6-bin is not installed.
dpkg: error processing sun-java6-jre (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of sun-java6-jdk:
sun-java6-jdk depends on sun-java6-bin (= 6-06-0ubuntu1); however:
Package sun-java6-bin is not configured yet.
dpkg: error processing sun-java6-jdk (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
sun-java6-bin
sun-java6-jre
sun-java6-jdk
E: Sub-process /usr/bin/dpkg returned an error code (1)

Turns out it's just a resource problem. If you're getting this error, check /proc/user_beancounters and see if you're getting any failures:

cat /proc/user_beancounters

101: kmemsize 625164 1720287 11055923 11377049 0
lockedpages 0 0 256 256 0
privvmpages 1173 129807 131072 139264 2

As you can see I was having problems with the limit on privvmpages. I doubled the default twice before it worked, resulting in a barrier of 262144 and a limit of 278528. You can change these values like so:

vzctl set 101 --privvmpages 262144:278528 --save

Sigh, Goodbye OpenSUSE

So I had to ditch OpenSUSE 10.3. It doesn't come as any surprise, really. I've never been able to break out of the Linux installation loop:

1) Install new distro
2) Work out various hardware kinks
3) Try to configure new distro to my liking
4) Try to use new distro for a few weeks
5) Hit a brick wall on some necessary feature; or distro simply breaks
6) Goto step 1

I had OpenSUSE 10.3 installed on both my Thinkpad R32 laptop, and on a standard desktop machine (Asus Mobo, Athlon XP, Geforce 6200 video). In both cases I ran into a dreaded KDE Freeze Bug. Unfortunately the KFB has like 80HP and does 20+2 (freeze) damage, which is a formidable enemy; even though I'm dumping all my skill points into Linux Tinkering and have several Potions of Diet Pepsi in my inventory.

Screw it. I might put more effort into battling the Freeze Bug, but OpenSUSE 10.3's package manager is absolutely atrocious. There have been so many times over the past month that I've thought about searching for software and decided that it wasn't worth it to wait for the package manager to start up. Even if I had that kind of time I'd still just end up fighting RPM dependency nightmares.

I realize that OpenSUSE 11 is just out and has a substantially better package management system. Unfortunately, they said the same thing about 10.3. If the package manager in 10.3 offered "dramatically improv[ed] speed," then I can't imagine how terrible it was previously. And even if package management is twice as fast in version 11, it would still be too damn slow.

So I installed Ubuntu 8.04 on both machines. So far so good, although I ran into the same old nVidia driver problem on the desktop and the laptop won't shut down properly with my PCMCIA wireless card installed. The nVidia driver issue was easy enough to fix (again), and when I figure out where to tell Ubuntu to eject the PCMCIA card on shutdown I'll update that blog post.

It's nice to be back on Ubuntu even though it uses the ever-ugly Gnome desktop and lacks a proper control panel. The Debian package management is really where it's at. I've decided not to try anything fancy with the laptop either - defaults all the way. We'll see how long this all lasts.

Oh, here's a cute one: when I successfully wake my Thinkpad from hibernation, the Hardy Heron informs me that the laptop was unable to hibernate. Thank goodness for the guy who put the "don't tell me this again" checkbox in that dialog.

Nagios with NSClient++ Character Flaws

Arg. It can be frustrating to pass special characters to check_nt arguments!

First, the dreaded ampersand (&):

Unfortunately, it appears as though the ampersand is the field delimiter used by NSClient++, so passing an ampersand to check_nt is absolutely not going to work. Take a look at the following code snippit from check_nt.c :

  249  case CHECK_PROCSTATE:
  250 
  251   if (value_list==NULL)
  252    output_message = strdup (_("No service/process specified"));
  253   else {
  254    preparelist(value_list);  /* replace , between services with & to send the request */
  255    asprintf(&send_buffer,"%s&%u&%s&%s", req_password,(vars_to_check==CHECK_SERVICESTATE)?5:6,
  256         (show_all==TRUE) ? "ShowAll" : "ShowFail",value_list);
  257    fetch_data (server_address, server_port, send_buffer);
  258    return_code=atoi(strtok(recv_buffer,"&"));
  259    temp_string=strtok(NULL,"&");
  260    output_message = strdup (temp_string);
  261   }
  262   break

  620 void preparelist(char *string) {
  621  /* Replace all , with & which is the delimiter for the request */
  622  int i;
  623 
  624  for (i = 0; (size_t)i < strlen(string); i++)
  625   if (string[i] == ',') {
  626    string[i]='&';
  627   }
  628 }
  

As you can see, the ampersand is hardwired into the request to the NSClient++ server, so any fix will require changes to both the check_nt plugin and NSClient++.

There is no workaround for this, except to avoid using the ampersand (escaping the ampersand with a backslash ( \ ) does not work). If, for example, you are trying to check on the status of the "Backup Exec Device & Media Service", use the service name instead of the display name -- NSClient++ can use either. In this case, the service name is "BackupExecDeviceMediaService", which you can find in the service properties.

P.S. - if you're unfortunate enough to be using Backup Exec, I feel for you.

Next, the dollar sign ($):

The dollar sign is a goofy one too, and can't be escaped with the backslash character ( \ ). Instead, you have to double it and put quotes around it (like so: "$$"). Neat, eh?

An example: let's say that you're trying to monitor the service MSSQL$BKUPEXEC. Unfortunately, this is both the display name, and the service name, so the last trick we used won't work. No worries, though, thanks to our friend the double-dollar-sign-enclosed-in-quotes. Your check_command will look like this:

check_command check_nt!SERVICESTATE!-l "MSSQL"$$"BKUPEXEC"

So awesome! Yay for annoying things!

Note: you might think you're clever and use single quotes instead of double around the entire service name. Unfortunately, that does not work reliably. It does seem to work, however, if you're only checking on one service name in the command. Anyhow, don't bother.

Next up, the backslash ( \ ):

This one is pretty easy, you just double it up ( like so: \\ ). Thus checking on a performance counter will look something like this:

check_command check_nt!COUNTER!-l "\\Network Interface(Intel[R] 82546EB Based Dual Port Network Connection - Packet Scheduler Miniport)\\Bytes Total/sec"

Phew, that counter name is a mouthfull, which is actually why I chose it. Don't try to manually type in your performance counters; copy and paste them. From a terminal to the Windows machine you're monitoring, open up Performance Monitor. Add the counter you're looking for to the graph, select the counter from the legend at the bottom of the window, and then click on the Copy Properties button (it's one of the buttons at the top of the graph). Now open up notepad or your favorite text editor and paste the performance counter data into it. Somewhere in there you should see a .path attribute that contains the entire counter reference which you can copy and paste into the specific Nagios configuration file we're working with (remove the server name and double all of the back slashes). Thankfully we can copy and paste from a terminal in Windows to local windows, if we're running Windows.

Note: I believe that much of the confusion over passing arguments to the check_nt command in Nagios has to do with this double back slash which looks like we're escaping the back slash. We're not...well, not really. Don't expect to simply use regular Bash shell notation in your arguments. Single quotes don't necessarily behave the way you'd like. Escaping doesn't work the way you might expect it to. Just don't bother trying to out-think the system, follow its conventions, make no assumptions, and you'll be fine.

OpenSuse 10.3 + Nvidia Driver for Geforce 6200 = CRAP

Sigh. So I followed the instructions you gave me for installing the nVidia driver for my 6200 with YaST. The installation process was very simple, but the after installation process not so much.

I'm curious as to why exactly you'd want to set my monitor refresh rate to 71.0KHz : 88.4Hz? Seems a little odd. Thankfully my monitor can still display this rate, but I get a big "OUT OF RANGE" box in the center of my screen. Nice.

Ok, so the obvious place to start is the "Graphics Card and Monitor" utility from the YaST2 Control Center. In other words, SaX2: X11 Configuration. (BTW, your multi-case meaningless acronyms are sweet. Sweet indeed.) Working around the giant "OUT OF RANGE" message, I'm able to limit the range of my monitor and I do so. No go. Neither changing the monitor nor reducing the frequencies has any effect. The card insists on driving the monitor at 71 : 88.4.

I check the xorg.conf file and verify that the correct frequency ranges are there. They are. I change a few things and break the config. Ok, restore the old file.

Aha! Wipe that evil grin off your smug face, because you haven't beaten me on this April morning! I found a workaround:

1. Get into the YaST Control Center (e.g. Administrator Settings), and choose the Hardware tab from the left.
   
   
2. Click Graphics Card and Monitor and click the Options button for the video card.
   
   
3. There should be a VertRefresh option. Set it to 60 (or some other safe value for your monitor, but 60 should almost always be safe).

Voila. Not the best, but at least it works.

If you can't get into X at all, then manually edit your xorg.conf file:

1. Login to your shell as root, or `su -` to become root.
   
   
2. Enter `emacs /etc/X11/xorg.conf` (Unless you have a simpler editor installed, but my install only had Emacs and vi.)
   
   
3. Hit enter once if you get the Emacs "welcome / help" screen. You should now see the contents of the xorg.conf file. Scroll down to 'Section "Device"' and add the following line before 'EndSection':
   
   Option "VertRefresh" "60"
   
   
4. CTRL+X then CTRL+C to quit Emacs. Hit 'y' on your way out to save.
   
   
5. Restart (or test by running `startx` from your shell).
   
   
6. I'm not sure if SaX will freak out on you for manually editing the file, so follow the first procedure above to make sure it sticks.

Happy Wasting-Your-Time,

Year of the Linux Desktop

xx00

Vista "Run As" Support

Thanks ever so much for changing the behavior of the "Run As" context menu item to better fit your broken User Access Control feature which broke certain vertical apps that run from GPO-mapped network drives thus forcing me to disable UAC. Very clever.

But I can handle it -- I'll just use the runas command from the shell!

Oh, I see you also "fixed" Explorer so that I can't use runas to launch it as a new process. Hm. No problem, let me try using IE to browse the file system. Dag, yo. I see you "fixed" IE in version 7 to launch an Explorer window to browse the file system.

Ok, whew, there's a registry value for Explorer called SeparateProcess which has to be set for the user that I want to run the Explorer process as. Alright, let me add that to my script and verify that it's set...ok...now run it...... .... .... .... .... .... you bastard.

Haha. Ok ok. Good one, seriously. What really gets me is that you had me all excited about the new security features in Vista, but I already had to disable UAC and now I'm looking at running as an administrator just to do my job? Yeah, it's cool that you fixed fast user switching so that we can use it on our domain machines, but who wants to completely switch to an administrator environment just to find a file for someone? Lame.

But you haven't won. I did find a solution.

To anyone who is trying to launch Explorer on Vista from an elevated account, just stop trying. It's a waste of time. Here's what you do:

Instead of trying to get Explorer to work, grab a copy of FreeCommander. FreeCommander is a file manager for Windows and, as its name implies, is free. You can launch FreeCommander with runas successfully, and you'll be able to browse your local file system or network as an administrator or domain administrator.

To make things easier, you can launch FreeCommander with runas from a simple shell script (batch file). Despite what you might think, you do not have to put your password into the runas command to make it work. Simply leave the password out and when you run the batch a command shell will appear asking for your password and once you've entered it, FreeCommander will fire up. All nice and safey safe.

To make things even easier, it's Sysinternals to the rescue once again with ShellRunas v1.01. Run `shellrunas /reg` to register it as a context menu entry, and you'll see a "Run as a different user" item on the menu when you right-click an executable. Basically, this restores in Vista the runas functionality we had in XP. You'll still see the worthless "Run as administrator" entry in the menu, but the world can't be perfect.

(Thank Blod for Mark Russinovich and Sysinternals. Note that Microsoft bought Sysinternals so you'd better grab everything you can now just to be safe.)

Wondering why I chose FreeCommander as my file manager? No? Well read on. Of all the free file managers I found in a quick search, FreeCommander is the only one that didn't puke when run under a secondary login. All of the rest either didn't work well on Vista period, or died horrible deaths when trying to browse the network. None of the programs will recognize Vista's awesome new way of handling mapped network drives, but at least FreeCommander keeps chugging along instead of freezing up and can do UNC just fine.

Here is a list of the programs that I tried on 04/10/08. None of these worked well enough:

Ac Browser Plus (ACB) (trouble on the network when started with runas)
ExplorerXP (blank context menus on Vista)
FileAnt (great program, but freezes on network when started with runas)
UltraExplorer (great looking program, but freezes too often)
muCommander (Java, very bad network support on Vista)

Off all these, only muCommander is GPL. Drag. FreeCommander wasn't my favorite of the five I tried, but it's the only one that doesn't puke when I run it this way.

Anyhow, I added a link to my batch file on my RocketDock (a MUST HAVE application for Windows) and everything is once again golden...or sort of brown anyhow....well, there's a little green in there...

*note: I can already hear the question in my own mind: well this solution works great from my own machine, but what about from my users' machines? Thankfully, my users all run XP with un"fixed" runas, explorer, and yeah, I've stuck them with IE6. When the Vista changes happens against my will, then fast user switching and runas from the shell will get most jobs done just fine.*

Settling on OpenSUSE 10.3

That's right, I've finally found the least annoying Linux distribution to run on my Thinkpad R32: OpenSUSE 10.3.

I burned through the following distros on my quest for non-suckage:

Ubuntu 7.10 & 8 Beta
PCLinuxOS 2007
Kubuntu 7.10 & 8 Beta
Fedora 8
Freespire 2.0

...and none of them were stable enough to survive initial updating and configuration. I was most disappointed by Ubuntu and Kubuntu because these are the distributions I really wanted to use. I actually use Ubuntu as a desktop OS on one of my machines, and have been for years, but I only use it for a few tasks. I'm running Ubuntu Server 6.06 LTS on a machine as well and am very happy with it (aside from its adaptec driver support being broken out of the box).

I didn't have much hope for OpenSUSE when I began installing it. I was certainly disappointed by the lack of a LiveCD with an installer. However, the "old school" installation went very smoothly and in no time I had a functional installation that ran well and looked very sharp. OpenSUSE is a very polished distribution by comparison to everything else that I've tried so far, aside perhaps from PCLinuxOS.

I was also a little bit leary of using KDE, because for years I've always been a Gnome "fan." Years ago I tried KDE and it was seriously flakey and irritating beyond belief. Plus, putting a "K" in front of all of your software names is Ketarded. Well, I realize now that Gnome hasn't changed a whole lot over the years, but KDE has really grown during my time away from it. I'm not missing Gnome at all.

OpenSUSE is not without its shortcomings. Firstly, it's an RPM-based distribution and I'm much more comfortable with debian based installs. YaST is a usable packager but is terribly, terribly slow and in typical RPM fashion (in my experience), occasionally doesn't grab necessary dependencies and vomits all over your lap. Secondly, the configuration tools are much better than the mess you get with Ubuntu, but even after a few days working with SUSE I'm still having trouble finding the right icons to click on. Too much ambiguity and poor naming. This is a common theme in all Linux distributions. Thirdly, I had to grab madwifi drivers "manually" and SUSE refuses to shut down when I have my PCMCIA wifi adapter plugged in. And lastly, I'm able to crash the entire system all over the place while trying to get Kerberos and LDAP working in a Windows domain. This isn't completely SUSE's fault, it's a pretty typical Linux kludge of half working services where getting a fully working system is akin to solving a jumping puzzle in Half Life.

(Here's a fun one: if you happened to follow recommended practice and use a .local TLD for your Windows domain, then mDNS freaks out. If you disable mDNS, then LDAP and/or Kerberos freak out. The result is some very interesting crashes, including DBUS, kpowerd, and knetworkmanager. Fug it. To give credit where credit is due, this may actually be Apple's fault. It's their spec. Anyhow, it turns out you don't need to configure Kerberos or LDAP clients if you just use the Windows Domain tool)

The last time I tried to push myself into migrating my desktops completely to Linux, about five or six years ago, I had a very similar experience. Half-baked, poorly glued together software that requires constant babysitting. I am very disappointed with the state of Linux on the desktop today. I know that it can work, especially if you limit yourself to a basic install and don't mess with anything. Otherwise, it's still a mess and its fan base is either poorly informed or in serious denial.

Shortly I'm going to be proposing my latest IT plan at the office, and a portion of the plan is an "Open Source Initiative" in which I'm going to lay out a procedure for migrating our proprietary software to open source alternatives. My primary objective is to limit or eliminate vendor lock in, which is a real sore spot for me. I also hate Microsoft and the BSA. Anyhow, the last step in my plan is going to be moving our desktops from Windows to Linux, and I'm still going ahead with it. However, I'm much less enthusiastic about it now. Despite its many shortcomings, I fail to see how Windows is inferior to any Linux distribution aside from licensing and cost. Thankfully Vista sucks eggs, because that might make the transistion a little easier for my users to swallow.