First, the one-click rails app on webfaction assumes you will run one mongrel server per application. However, in order to handle multiple instances with load balancing, we will need to make some changes. The setup is not too complicated, but we need to run our own apache2. But before you install additional software, you should first make sure your app runs on the server with one instance of mongrel. Once you do that, you can follow the instructions below.

Installing Stuff

First, we need to install apache2. You can compile your own version by downloading the source, but webfaction makes it easy since they already have a one-click install for mod_python. If you install mod_python, it will install apache as a webapp under your webapp directory. That’s what I did.

Next, we’ll install the mongrel and mongrel_cluster gems. But if you haven’t install your own version of Ruby and RubyGems, you might want to follow this knowledgebase article. Run:

gem install mongrel_cluster

to install mongrel_cluster.

Configuring Stuff

Control Panel

In order to create new instances of mongrel, you need to go into the control panel to open up a new port. All you need to do is specify the app as a custom app. The custom app will not install anything, but it does create an empty directory in your webapps directory. With a custom app, webfaction just assigns a port number to you. There’s no need to run an app within a specific directory. So go ahead and create as many apps as you need instances, and note the port numbers.

httpd.conf

We now need to setup the httpd.conf by turning off the mod_python stuff, and turning on some extra modules. If you installed apache through the one-click install, your httpd.conf file lives here:

~/webapps/apache/apache2/conf/httpd.conf

Open that up in your favorite text editor and replace the contents with this:

ServerRoot "/home/username/webapps/apache/apache2"

LoadModule authz_host_module modules/mod_authz_host.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule alias_module modules/mod_alias.so

DocumentRoot /home/username/webapps/apache/htdocs
KeepAlive Off
Listen 7708
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog logs/access_log combined
ServerLimit 2

NameVirtualHost *:7708

<VirtualHost *:7708>
  ServerName example.com
  ServerAlias example.webfactional.com www 

  DocumentRoot /home/username/webapps/website_mongrel_source/public

  RewriteEngine On

  <Proxy balancer://mongrel>
    BalancerMember http://127.0.0.1:7777
    BalancerMember http://127.0.0.1:7779
  </Proxy>

  <Directory /home/username/webapps/railsapp/public>
     Order allow,deny
     Allow from all
  </Directory>

  # Let apache serve static files
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f
  RewriteRule (.*) $1 [L]

  # Rewrite index to check for static
  RewriteRule ^/$ /index.html [QSA]

  # Rewrite to check for Rails cached page
  RewriteRule ^([^.]+)$ $1.html [QSA]

  # Redirect all non-static requests to cluster
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule ^/(.*)$ balancer://mongrel%{REQUEST_URI} [P,QSA,L]

  ProxyPass / balancer://mongrel/
  ProxyPassReverse / balancer://mongrel/
  ProxyPreserveHost on

  # Custom log file locations
  ErrorLog  /home/username/webapps/railsapp/log/error.log
  CustomLog /home/username/webapps/railsapp/log/access.log combined
</VirtualHost>

All the things in bold should be replaced with information specific to your server. This is a typical apache configuration for a mongrel_cluster on apache2. You should configure it to suite your needs. Remember to restart apache2 after you modify httpd.conf for the changes to take place. I noticed that the httpd command was somehow still bound to the root apache2 server, so to restart the server, I had to run this:

~/webapps/apache/apache2/bin/restart

mongrel_cluster

One of the problems with webfaction is that we can’t specify specific port numbers that our applications run on. Port numbers are automatically assigned every time you create a new app under their control panel. Since mongrel_cluster uses sequential port numbers (ie. if you run 3 instances and the first port number is 3000, mongrel_cluster assumes the other two instances run on 3001 and 3002.) a simple modification to mongrel_cluster is required. Follow this article and you’re all set. Once you make the modification to mongrel_cluster, and create a modified mongrel_cluster config file, you can run mongrel_cluster with:

mongrel_rails cluster::start

as usual. Once you do that, you should be all set. You will now be serving up your rails app with mongrel_cluster under your own apache2 install.

autostart.cgi

You may want to create or modify your autostart.cgi file to make sure your app comes back up if your server died for some reason. My python skill was a bit rusty, but this did the trick:

#!/bin/env python2.4
import os

# Test if the process is already running
running = False
ports = ['7777', '7779']
for port in ports:
  try:
      # Try to read pid file
      pid_file = '/home/username/webapps/railsapp/tmp/pids/mongrel.%(port)s.pid' % {'port': port}
      pid = open(pid_file).read()
      # Check if this process is up
      lines = os.popen('ps -p %s' % pid).readlines()
      if len(lines) > 1:
          running = True
      else:
          # Delete pid file
          os.remove(pid_file)
  except IOError:
      pass

print "Content-type: text/html\r\n"
if running:
    print """<html><head><META HTTP-EQUIV="Refresh" CONTENT="2; URL=."></head><body>
    Site is starting ...<a href=".">click here<a></body></html>"""
else:
    print """<html><head><META HTTP-EQUIV="Refresh" CONTENT="2; URL=."></head><body>
    Restarting site ...<a href=".">click here<a></body></html>"""
    os.setpgid(os.getpid(), 0)
    os.system('/path_to/mongrel_rails cluster::start')

and that should allow your rails app to restart properly. If you want to serve another rails app with mongrel_cluster, you just need to add a new virtual host to the httpd.conf file. Anyway, that’s all there is to getting mongrel_cluster running on a shared Webfaction account!

Today was a good day. Not only did I win Solitaire on my iPod on a train down to Providence, I fixed a mailserver issue that I’ve been struggling with for a few weeks.

I’m slowly switching over to Slicehost for all my web hosting requirements, and setting up the server (I’m running Ubuntu Hardy) so that I can get rid of all the other hosting I use (I already got rid of crappy Dreamhost). I’ve been using Pair for the last ten years and have been really happy with them, but I’m a server junkie. VPS is just too tasty. Anyway, most server stuff is pretty straight forward to set up. Pickled Onion articles and a bunch of howtoforge docs will get you started from a fresh disk image in no time. One thing that tripped me up was setting up the mail server. I wanted the standard PostfixAdmin controlled virtual user setup using MySQL to manage the users. Following this and this got everything working, except SMTP would not authenticate properly when storing passwords encrypted in the database.

Everything worked, Courier IMAP and SMTP Auth using TLS…but only if the users passwords were stored cleartext in the database. Even if connecting securely, storing passwords cleartext is a bad idea. The problem comes from Ubuntu SASL libraries not supporting password_format. I didn’t want to run the patch, since…well, I don’t really know how patching while using aptitude works (you probably have to run it every time you update), and I didn’t want to start compiling some things from source, while mostly relying on a package manager. I’ve inherited servers where some things are compiled from source and other things are managed on multiple package managers, and it is a BAD IDEA.

So, the solution. I followed the Ubuntu article and set up my database much like their example, but instead of using auxprop for pwcheck_method, I use saslauthd with PAM.

First, make sure you install libpam-mysql:

sudo aptitude install libpam-mysql


Then, in /etc/postfix/sasl/smtpd.conf:

pwcheck_method: saslauthd
mech_list: plain login


You might also want to add log_level: 7 while you are debugging. Next, create /etc/pam.d/smtp with the following:

auth required pam_mysql.so user=the_user passwd=the_password host=127.0.0.1 db=postfix table=mailbox usercolumn=username passwdcolumn=password crypt=1
account sufficient pam_mysql.so user=the_user passwd=the_password host=127.0.0.1 db=postfix table=mailbox usercolumn=username passwdcolumn=password crypt=1


Then make sure you use $CONF['encrypt'] = ’system’ for PostfixAdmin. You should observe /var/log/mail.log and /var/log/auth.log for any errors. When I first tried out PAM, I didn’t realize that I had not installed libpam-mysql, so only today when I went back and set the log_level did I actually realize I needed to install it.

Now I feel a little better not having cleartext passwords around…

ps. I am really bad at Solitaire, so it was a good day.