Gah.

The second time, through an AI course, I got decently fluent in very basic Emacs Lisp. Not actually interacting with Emacs that much, just using elisp as a Lisp dialect for getting the course projects done (for various reasons, elisp is a bit of a stupid choice of dialect for general-purpose programming, but never mind). Being young and stupid, I thought that I could just switch gears and pick up Common Lisp in my stride.

I tore through some random Common Lisp tutorial with ease, reached macros, and my head exploded. It wasn't its fault, it had never seen anything like that. I had a gut feeling that macros were exceedingly cool, kind of the light sabers of programming: an elegant weapon for a more civilized age, that can cut through just about anything while making a cool sound. But I wasn't ready. So retreated once more, back to the familiar pastures of C, Python and Java.

Last year, after rebuilding and consolidating my brain in various ways, I tried again. This time, I understood how macros worked, and was suitably impressed at being able to run code at compile time to produces more code. But I lost interest. It seemed to me that Common Lisp was being used for really byzantine pieces of software, which could only be described with multiple uses of the 'meta-' prefix, and just generally spent a lot of time not doing anything of practical value. Doing anything that might benefit mere mortals seemed just out of grasp.

And so, I went back to the pastures of C, C++, Python, and other old friends. To my surprise, I did sometimes find myself pining for macros once or twice, when building particularly horrid little pieces of code that were trying really hard to abstract away concepts with no help from the language.

Which brings us to here and now. I've spent two days with Seibel's book, which addresses precisely what I felt last time, that Common Lisp cannot be used for pragmatic development of everyday software. The book elegantly demolishes this illusion by digging into a cornucopia of practical programming projects in Common Lisp, including:
· A unit testing framework, using macros to define new syntactic constructs that can concisely define new tests, and abstract away the ugly book-keeping machinery
· An ID3v2 tag parser. The approach used here is to first create, using macros, a domain-specific language for describing the structure of binary formats. The code to read, write and manipulate the tags is then derived from this very beautiful and concise definition.
· A Shoutcast streaming server, demonstrating everything from network I/O to keeping an MP3 database to using our ID3v2 tag library
· A little HTML template language. This little language lets programmers describe the HTML object tree in a delightfully minimal manner, as well as mixin some basic rendering logic (conditional blocks, looping ...). Peter then goes on to write both an interpreter and a compiler for this language, showing an abstraction technique that is still something pretty exclusive to Lisp: the creation of new languages to concisely express solutions to problems.

The great thing about Lisp, as so many have said before me, is the REPL. If you've ever used Python, Perl or Ruby, you know what I'm talking about: the ability to establish a dialog with the language. You feed it something, it tells you what it thinks. You change the code slightly, build up a little more functionality, tell the language about a new abstraction you'd like it to understand... And all the time, you have at your fingertips the ability to ask Lisp "What do you think? Does this look good? Could you evaluate this for me real quick, to see if I got it right?" I don't think that anyone coming from a purely edit-compile-run language background can really grasp the basic joy that there is in having a conversation with your language to build software. It's a very pleasant way of programming, and on a fundamental level it defies description. You have to try it.

Given how much Lisp code I've written, I am but a humble grasshopper. I've barely started my journey down this path, and to paraphrase Feynman, if I think I understand Common Lisp, I probably don't understand Common Lisp. Not yet at least. But, while writing the 70 lines of Lisp code that make up my first non-trivial program (a parser for the wire format of Google Protocol Buffers, if you care), there were brief moments when the clouds parted, and I caught a glimpse of where the path leads, all the way to the horizon.

I think I may enjoy walking the path.

The Road goes ever on and on
Down from the door where it began.
Now far ahead the Road has gone,
And I must follow, if I can,
Pursuing it with eager feet,
Until it joins some larger way
Where many paths and errands meet.
And whither then? I cannot say.

The short version: My configuration is now 50% smaller, even with the addition of an extra virtual host or two. mod_macro is really kinda cool.

If you don't know, mod_macro is an Apache module that lets you define macros. Astounding and surprising, isn't it? In web programming, you'd be more correct in calling them templates: a named chunk of configuration directives, in which you can perform limited variable expansion. As a simple example, here is a macro that restricts access to a path:

<Macro RestrictAccess $url $userfile $username>
  <Location $url>
    AuthType basic
    AuthName "Restricted access"
    AuthUserFile $userfile
    Require user $username
  </Location>
</Macro>

You then invoke this macro and pass it parameters to inject into the template:

Use RestrictAccess /private /srv/www/.htpasswd dave

Using this mechanism, you can dramatically reduce the size of your apache configuration, especially with several virtual hosts with redundant statements. I used macros to do just that. Let's construct a set of macros to aid us in configuring virtual hosts.

First of all, let's standardize and fold away all the tedious boilerplate of virtual hosts: defining the server name, the logging configuration, and default access directives.

<Macro DocrootLoggingAccess $name $vhost_dir>
  ServerName $name
  ServerAdmin dave+httpd_$name@...
  DocumentRoot ${APACHE_WWWROOT}/$vhost_dir
  CustomLog ${APACHE_LOGROOT}/$vhost_dir/access.log combined
  ErrorLog ${APACHE_LOGROOT}/$vhost_dir/error.log
  LogLevel warn
  ServerSignature off
  <Directory ${APACHE_WWWROOT}/$vhost_dir>
    AllowOverride None
    Order allow,deny
    Allow from all
  </Directory>
</Macro>

Note the use of APACHE_WWWROOT and APACHE_LOGROOT variables. These are environment variables, which Debian's apache package helpfully lets us define in /etc/apache2/envvars. Debian uses this mechanism to define things like the PID file path, which gets pushed into the configuration, but also gets included by debian helper scripts, like the start-stop daemon. I subverted the mechanism to define my own global configuration variables:

export APACHE_WWWROOT=/srv/www
export APACHE_LOGROOT=/srv/logs/www

This lets me define paths relative to these global roots, which makes the configuration look much nicer. With this macro alone, a virtual host for serving static files becomes as simple as:

<VirtualHost *:80>
  Use DocrootLoggingAccess static.natulte.net natulte.net/static
</VirtualHost>

That will serve the content of /srv/www/natulte.net/static on http://static.natulte.net . Don't try it, it was just an example, I'm not actually serving anything there :-)

I won't go over the other macros I have in detail, but here is a quick overview:

# Configure a FastCGI dynamic instance to handle all .php files through PHP 5
Use PHP5

# Serve the Trac environment 'nxos' under URL path /nxos, via FastCGI.
# Logins to the environment should use authentication data for
# domain natulte.net, group 'nxos-dev'
#
# This macro makes use of an APACHE_TRACROOT envvar.
Use Trac nxos /nxos natulte.net nxos-dev

# Serve a read-only copy of the nxos mercurial repositories under /hg
#
# Uses the APACHE_HGROOT envvar.
Use MercurialReadOnly nxos /hg

# Same as above, but authenticate write access using natulte.net's
# authentication data (access to specific repositories is configured in
# each repo directly, apache needs only to identify potentially acceptable
# users).
Use Mercurial nxos /authenticated-hg natulte.net

# Configure the given URL path as a Dokuwiki instance (directory access and
# rewrite rules).
Use Dokuwiki /wiki

# Activate /server-status and /server-info on this vhost. Useful for
# debugging broken macro expansions.
Use ServerStatus
Use ServerInfo

# Small helper template: given a domain name, include configuration directives
# for HTTP Basic authentication using user/group data global to that domain.
# Reused in all the above templates that do authentication.
Use AuthData natulte.net

I won't bore you with any more gruelling details. Suffice it to say that, with this set of macros, the configuration for, say, nxt.natulte.net goes from a 50 line monstrosity of boilerplate and weird FastCGI directives, to:

# Notice how I can still add snippets of configuration. I just
# don't have boilerplate rubbish any more!
<VirtualHost *:80>
  Use DocrootLoggingAccess nxt.natulte.net natulte.net/nxt
  Use TracEnv nxos /nxos/trac natulte.net nxos
  Use MercurialReadOnly nxos /nxos/hg

  <Location /nxos/docs>
    Options Indexes
  </Location>
</VirtualHost>

Thanks to mod_macro, I'm happy with my web server configuration for the first time in a while. It's as code should be: compact, common elements refactored, no duplication, and above all, clear and easy to understand.

If, like me, you were drawn in by the prospect of being able to have a more compact configuration, but have anything but the very simplest setup, you may want to save yourself a day or so and head straight for Apache's mod_define and mod_macro instead.

Everyone and their dog uses AES, why is it safe?

Secure because of AES, not necessarily. As with many things, it's not just what you have, it's all about how you use it. I could write a password store that uses AES to encrypt all fields but the password, and say "It's secure, it uses AES!" Of course this would be ridiculous, but it's just an extreme demonstration of the core point: using a currently secure cipher doesn't magically make your password store secure.

I looked around the sites of both Password Manager and KeePass for an overview of their datastore, and all I could find was "it's encrypted, so it's secure", followed by dissertations on how to securely copy a password from the store into an application.

Am I the only one who would like an overview of how the database is structured? Is each password encrypted individually, or is the final binary blob of a database encrypted? When I access one password, is the entire database sitting decrypted in ram, or just that entry? Is chaffing used so that an attacker is unable to infer password length by observing changes to the database's size? Are codes used to detect, and maybe even correct errors, so that an attacker flipping a bit doesn't alter my passwords, or at the very least makes it detectable?

All these are questions that a password store should be answering in an "overview of the system" page, without forcing me to go down into the code to reverse engineer it all.

Consequently, I'm still on the lookout for a good cross-platform password store. Ideally, all security critical code would be constrained to a small portable library, both for auditability and to ease the creation of varied UIs. If I can read the whole source of the core "database" layer (ie. no GUI) in one afternoon and convince myself that it is secure enough for me, I will feel much less bothered at the idea of using your software. Please don't force me to make a leap of faith with password storage.

:mozilla.se.eu.dal.net 004 daive mozilla.se.eu.dal.net bahamut-1.8(04)
   aAbcdefFghiIjkKmnoOrRswxXy bceiIjklLmMnoOprRstv

This message tells the client what server he is connected to (mozilla.se.eu.dal.net), what server software is running (bahamut-1.8(04)), what user modes this network knows about (aAbcdefFghiIjkKmnoOrRswxXy, one letter per mode), and what channel modes it knows about (bceiIjklLmMnoOprRstv, again one letter per mode).

All this information helps the IRC client to be a good citizen of that network, for example by avoiding attempting to set modes that aren't known to the network (which will fail anyway, so it's just a waste of network trafic).

Another one of those very useful messages is actually so big that it's transmitted in several messages: numeric reply 005.

:mozilla.se.eu.dal.net 005 dave NETWORK=DALnet SAFELIST MAXBANS=200 MAXCHANNELS=20 
  CHANNELLEN=32 KICKLEN=307 NICKLEN=30 TOPICLEN=307 MODES=6 CHANTYPES=#
  CHANLIMIT=#:20 PREFIX=(ov)@+ STATUSMSG=@+ :are available on this server
:mozilla.se.eu.dal.net 005 dave CASEMAPPING=ascii WATCH=128 SILENCE=10 ELIST=cmntu
  EXCEPTS INVEX CHANMODES=beI,k,jl,cimMnOprRst MAXLIST=b:200,e:100,I:100
  TARGMAX=DCCALLOW:,JOIN:,KICK:4,KILL:20,NOTICE:20,PART:,PRIVMSG:20,WHOIS:,WHOWAS:
  :are available on this server

What does all that garbage mean? Well, it looks like a list of directives giving information about some capabilities of the network. Some are fairly obvious: the network is called 'DALnet', the maximum topic length on a channel is 307 characters...

But others on that list are somewhat more cryptic. What does INVEX mean? What's with the strange comma-separated syntax for CHANMODES ? Fortunately, this is what we have RFCs for! This is what RFC 2812, "Internet Relay Chat: Client Protocol", has to say about numeric reply 005:

005    RPL_BOUNCE
       "Try server <server name>, port <port number>"

       - Sent by the server to a user to suggest an alternative
         server.  This is often used when the connection is
         refused because the server is already full.

If you're thinking that this doesn't look like our message from above at all, you would be right. I've tried five different IRC networks, and all transmit numeric reply 005 in the format I showed above. What probably happened is that someone came up with this capabilities system for IRC networks, and replaced an unused numeric reply with it. And, of course, didn't document it or anything. Because that would be too easy!

I can't help but think that this change was rather stupid. There is a huge gap of unused numeric replies from 006 to 199. And yet, whoever came up with this new message chose to replace an existing one, which means that we now have conflicting specifications and reality. It seems that this is unfortunately par for the course for IRC.

And, for extra credit, figure out what the third set of flags EFnet sends out in numeric reply 004 is for! It's also not in the spec.

I want something where I edit files containing HTML fragments, or something like restructured text. When I want to publish, I run a script that puts all these files through a templating engine that would work somewhat like Django's inheritance mechanism, which injects the fragments in a given base template. Then, I can publish these pages as static content, with ubercaching configured to have good performance.

I don't care much for any cool dynamic features. Making the timestamp at which the script runs available in the template engine might be cool, but that's about it.

It's simple enough for me to write this in Python, but before I start, does anyone know of a project that does this?

Edit: Found it! It's webgen.

I must be defective.

I love this place.

Then, install xcompmgr. If you're on ubuntu, the package is in universe. In Debian, the package is only in experimental, and you have to build it yourself. Add the following to your /etc/apt/sources.list:

deb-src ftp://ftp.pl.debian.org/debian/ experimental main non-free contrib

You should also comment out all other deb-src sources temporarily. Then:

$ sudo apt-get update
$ apt-get source xcompmgr
$ cd xcompmgr-1.1.1+cvs.20051218
$ fakeroot debian/rules binary
$ dpkg -i ../*.deb

You also need to install transset-df. Ubuntu also has transset in universe, but that is not what you want: transset-df is a fork of transset, and where transset can only apply transparency settings to the window you click on, transset-df can also select the window by window-id. Transset-df is pretty easy to install, just run make && sudo make install in the extracted sources directory.

Now, we have all the components that we need. First things first, we need to get Openbox to start xcompmgr. Put the following in your ~/.config/openbox/autostart.sh:

# Call the systemwide startup script to start Gnome/KDE/DBus services
. $GLOBALAUTOSTART

# Set the wallpaper, and any other stuff you might want here
$BG -full ~/.config/openbox/wallpaper.jpg

# Start xcompmgr
xcompmgr -n &

Check out the xcompmgr manual for more flags. -n tells xcompmgr to do simple client-side compositing: transparency only, no dropshadows or eyecandy fading effects. I like my desktop simple.

Restart openbox (or start xcompmgr manually for this session), and check if the compositing works: run transset-df 0.80 and click on any window when the cursor turns to a targetting reticle. When you click, the targetted window should turn a classy shade of transparent. If it didn't, there's something wrong with xcompmgr or your X11 Composite setup.

Now, that's all very well, but how do we make all terminals transparent? We're not going to click on each one each time we open it!

The easiest way I've found to do this is to edit my .bashrc (replace with the settings script appropriate for your shell), and add the following right at the end:

# Engage transparency if we're running in xterm
if [ "$TERM" = "xterm" ]; then
  sleep 0.1; transset-df -i $WINDOWID 0.75 >/dev/null
fi

The sleep is there to give xterm enough time to actually create the window. Without it I found that on my machine, transset-df would sometimes run before the window was actually created, so it'd end up not being transparent. A 0.1s delay is imperceptible enough to not be annoying, and gives xterm enough time to fire up properly before going transparent.

And there you have it. Of course, now that you have compositing enabled, you can make any window as transparent as you like, but I like to keep it simple and classy, and only have the terminals colored a hyperintelligent shade of transparent.

That is all.