8-bit b.us
A secret lightning / to firefly lovers below / but they never land
Bourbon is not gold / Tonight I die by amber / Tomorrow by gold
From a Faction on the Web to a Slice of the Host
From a Faction on the Web
…
Until recently, 8bitb.us was a super-basic Django blog with no caching, running on Webfaction. ‘Twas slow, but got the job done. The only interesting bit was serving media from S3. Since the super-cheap Webfaction plan has pretty strict limits on resources, I needed to serve media outside of Apache so my two (yes, two) Apache clients could focus on serving dynamic content.
The basic setup looked like:
… to a Slice of the Host
A few nights ago I moved the whole mess over to Slicehost. As a sysadmin, it irked me that Webfaction limited the number of long-running processes I could have and didn’t give me root. I know why they do this, and it works for most people. But I had plans, big plans, and the plans involved daemons. Here’s the new setup:
So requests for both static and dynamic content come in to Varnish, which passes the requests to Nginx if the URL isn’t in Varnish‘s cache.
What the above diagram doesn’t show you is that every process is being controlled with Daemontools. I like Daemontools because I get process respawning, log rotating, and consistent signal-based control for free.
Files
in the Ointment
backend default {
.host = "127.0.0.1";
.port = "8000";
}
#!/bin/sh
exec 2>&1
exec /usr/local/sbin/varnishd -T 127.0.0.1:60001 -F -a 0.0.0.0:80 -f
./8bitbus.vcl
/service/8bitbus-nginx/nginx.conf
#!/bin/sh
exec 2>&1
exec /usr/local/sbin/nginx -c ./nginx.conf
#!/bin/sh
export PYTHONPATH=/home/alex/python
exec 2>&1
cd /home/alex/djangoprojects/eightbitbus
exec python manage.py runfcgi host=127.0.0.1 port=8080 maxchildren=5
daemonize=False --settings=settings
Questions You Didn’t
Ask
Why no Memcached?
Since every vistor to the site (except for me) sees the same content, I don’t need fine-grained caching. All I need’s a coating of Varnish over the whole thing.
Why SQLite instead of
Postgresql or Mysql?
SQLite takes very little RAM (my slice only has 256MB). Postgresql and Mysql take a lot. SQLite has good read concurrency, and I minimize the number of database writes by using file-based sessions.I’m trying to avoid them for all but admin work. And when I do need a session, do I store it in the database? No! I’d like to minimize the number of writes to SQLite, partially because SQLite doesn’t handle concurrent writes well, and partially because I’m keeping the db in git.
How fast is it?
Hard to say - this is, after all, just a slice of a machine. And we all know benchmark numbers are dumb, but … fine. Via ab, I’ve gotten ~6000 homepage requests locally. Via Siege, I’ve gotten up to 1900 homepage requests per second locally, or 400 requests per second across two remote machines. And I still have some kinks to work out with Varnish, so I expect those numbers to go up. Other people on real hardware can get 4000 requests per second with Varnish.
The Clonus3 Horror
Parts: the Clonus Horror was an abomination. Clonus3: My Backup Script isn’t too bad.
So, I’ve moved all the media for 8bitb.us (user-facing and admin) to Amazon’s S3 service, and it’s made a huge difference. The poor Apache instance hosting 8bitb.us only has a couple of child processes, so the fewer requests per page Apache has to handle, the better.
To handle syncing media to S3, I decided to modify a little S3 backup script I wrote. We use it at work to back up 800GB of media and database dumps, but keeping a private backup is different than serving public media. Our backups store the entire path of the file, but http://media.8bitb.us/home/alex/django-projects/8bitbus/media/css/global.css is a really ugly URL. So I added support for relative paths. Add a few canned ACLs into the mix, and you’ve got one script that makes private backups or syncs public media.
Clonus3 is fast, but it wasn’t always this way. To retrive metadata from an S3 object, you have to issue a HEAD request for every object, individually. This is syrup-slow and Clonus3 could be about half the size if listing a bucket retrieved said metadata, but I devolve. Jeff Triplett recommended I cache the HEAD calls, so I did. A backup of my laptop, if nothing’s changed, takes 9 minutes 25 seconds without the caching and 8.37 seconds with. To be safe, Clonus3 (by default - you can turn this off if you’re not paranoid) lists the bucket first and uncaches any entries with modified etags. With this option off, a backup of my laptop takes 1.15 seconds - the bytes move so fast, they have blisters. A caveat: Apple doesn’t appear to have included Berkeley DB libraries in OS X, so you may have to jump through fiery hoops to get the caching on a Mac.
Soon, I’ll add support for setting the Expires, Content-Encoding and Content-Type headers based on regexps. In the far future, maybe Clonus3 could gzip text files on upload. Then maybe I’ll add a restore function. And locking, so you can sync two computers over S3, and … HURK [Editor’s note: Here, the author dies of feature creep.]
Django on a Joyent Accelerator, the Hard Way
There have been a couple of excellent guides on installing Django on an Accelerator, and I thought I’d throw mine into the mix.
These are the notes I took during a recent install of Django on a pair of Joyent Accelerators. While the other guides I found used the versions of Apache, PostgreSQL et al available in Blastwave or Pkgsrc, this project required newer versions than the package managers could provide. This means, of course, that nearly everything had to be built from source.
All source was downloaded to /home/admin/src, and everything was installed to /opt/local. In the following, Server1 refers to the Accelerator running Apache, and Server2 is the Accelerator running Mysql.
/home/
admin/.bashrc
Changed the default PATH on both machines to something like:
export PATH=/opt/local/bin:/opt/local/sbin:/usr/xpg4/bin:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/ccs/bin:/opt/csw/bin:/opt/csw/sbin:/opt/csw/gnu:/opt/csw/gcc3/bin:/opt/local/apache2/sbin:/opt/local/apache2/bin:/opt/local/mysql/bin
Python
2.5 on Server1
Building:
cd Python-2.5.1/ gpatch -p1 < ../Python-2.5.1.patch ./configure --enable-shared --prefix=/opt/local make sudo make install
OpenSSL 0.9.7 on both
servers
This may not have been needed on Server2, as I used the SSL bundled with MySQL.
Building:
cd openssl-0.9.7m/ ./Configure --prefix=/opt/local --openssldir=/opt/local/openssl solaris-x86-gcc gpatch -p1 < ../openssl-0.9.7m.patch make sudo make install
MySQL 5.0.45
(client and mysqld) on both servers
Building:
cd mysql-5.0.45/ CFLAGS="-O3" CXX=gcc CXXFLAGS="-O3 -felide-constructors -fno-exceptions -fno-rtti" ./configure --prefix=/opt/local/mysql --enable-assembler --with-yassl make sudo make install sudo mysql_install_db # Only on Server2; probably not necessary for upgrades
Edited /etc/passwd on Server2:
mysql:x:100:100::/opt/local/mysql:/bin/false
SSL certs are in /opt/local/mysql/certs.
Config is at /etc/my.cnf; databases and logs are in /opt/local/mysql/var.
MySQLdb 1.2.2 on Server1
The “python setup.py build; rsync …” dance is to avoid installing eggs.
Building:
cd MySQL-python-1.2.2/ gpatch -p1 < ../MySQL-python-1.2.2.patch1 gpatch -p1 < ../MySQL-python-1.2.2.patch2 LD_LIBRARY_CONFIG=/opt/local/mysql/lib LDFLAGS="-L/opt/local/lib" python setup.py build sudo rsync -aP build/lib.solaris-2.11-i86pc-2.5/ /opt/local/lib/python2.5/site-packages/
MySQL-python-1.2.2.patch1
MySQL-python-1.2.2.patch2
Apache 2.2.6 on Server1
Pre-build, I moved the existing Blastwave Apache install out of the way, but didn’t pkgrm it:
sudo mv /opt/csw/apache /opt/csw/apache2.disabled
Building:
cd httpd-2.2.6/ ./configure --prefix=/opt/local/apache2 --enable-mods-shared=all --enable-ssl make sudo make install
Config is in /opt/local/apache2/conf/httpd.conf
mod_python 3.3.1 on
Server1
Building:
cd mod_python-3.3.1/ ./configure --with-apxs=/opt/local/apache2/bin/apxs --with-python=/opt/local/bin/python make sudo make install
libevent/memcached on
Server1
Install via Blastwave:
pkg-get -i memcached
Libevent wouldn’t build from source cleanly; the solution may be here:
http://monkeymail.org/archives/libevent-users/2007-February/000481.html http://monkeymail.org/archives/libevent-users/2007-June/000661.html http://www.mail-archive.com/libevent-users@monkey.org/msg00658.html http://lists.danga.com/pipermail/memcached/2007-July/004631.html http://monkeymail.org/archives/libevent-users/2007-May/000615.html http://monkeymail.org/archives/libevent-users/2007-May/000621.html http://monkeymail.org/archives/libevent-users/2007-June/000678.html
python-memcached 1.40 on
Server1
Building:
cd python-memcached-1.40/ python setup.py build sudo cp build/lib/memcache.py /opt/local/lib/python2.5/site-packages/
Django on
Server1
Added /opt/local/lib/python2.5/site-packages/tetriad.pth:
/home/code
Django svn checkout is at /home/code/django. /home/code/network/settings.py shows how I connected to MySQL:
DATABASE_ENGINE = 'mysql'
DATABASE_OPTIONS = {
'read_default_file': '/etc/my.cnf',
}
DATABASE_NAME = 'installer'
Manifests and Methods (both
servers)
Copies of the manifests are in /opt/local/svc/manifest. Methods called by the manifests are in /opt/local/svc/method.
