I came across this post which, quite frankly, I consider to be complete and utter FUD. The author clearly at least tried Nginx or perused the documentation since he's got at least a passing familiarity with the software, but he seems to have missed the boat overall. I'm going to address his points one-by-one (at least as much as possible).

[Nginx] behaves in inexplicable ways for different browsers.

This complete and utter FUD. Browsers behave in inexplicable and different ways given the same content, but all Nginx does is serve content. It doesn't create content, so this claim is patently ridiculous.

The primary documentation is in Russian.

More FUD. It is true that Igor (the author of Nginx) is Russian and writes his documentation in Russian (surprise!). However, every bit of that documentation has been translated to English (and has ongoing efforts in several other languages as well, but I cannot attest to their completeness). This documentation has existed for over two years (and is linked to from the "primary" documentation), so there's little excuse to not be aware of it.

Nginx does not support .htaccess files.

Absolutely true. It's considered a feature. .htaccess files are certainly convenient, but they also introduce a serious performance penalty (if you enable them, Apache must check every directory on every page request to see if it has an .htaccess file and if it has changed since the last time it read it). One of Nginx' claims to fame is that it is much faster than Apache. One of the ways it achieves this is by not doing things that add lots of overhead to every request when there are faster ways of achieving the same thing. Further, .htaccess files are a potential attack vector for hackers, although this isn't the main reason Nginx doesn't support them.

Nginx requires you to have apache support tools lying around to do stuff.

Sort of true, but ultimately false. It's true that Nginx does not ship with its own version of htpasswd. It could (and such a program is trivial to write) but for whatever reason it doesn't (perhaps the assumption is that most people have Apache installed and so it would be redundant). But of course the web is full of such tools so you don't really need to install Apache just for this one tool. Also, the author says "tools", implying that anything except htpasswd is missing, but of course this is highly misleading. There's nothing else missing.

Nginx doesn't actually do anything beyond serve static HTML and binary assets... which is to say, it doesn't run php or perl or any of the other P's that you might find in the LAMP stack. What it does is take requests and proxies them to other servers that do know how to execute that code.

Once again, sort of true but highly misleading. The author suggests that all Nginx can do is proxy requests to another HTTP server. This is patently false. Nginx can use HTTP proxying or FastCGI to talk to backend applications. What it does not do is embed languages into the webserver or support CGI. There is a project to embed Python into Nginx (mod_wsgi) but it's not widely used as most Nginx users consider this separation of concerns a good thing. With mod_php or mod_perl, it can be a real pain to debug things like memory leaks because the programming language's interpreter runs within the Apache process. Apache's mod_language tools make for easy deployment but painful debugging. Also, unless you want to restart the entire web server anytime you make a change, you must use .htaccess files (whose drawbacks were outlined above). CGI isn't supported simply because, as Igor says "if you use CGI then you don't care about performance and you should just use Apache in that case". In any case, the difference between how Nginx handles dynamic content and how Apache does it boils down to how they pass the request along to another process (binary API vs HTTP/FastCGI).

The author sums up with this:

Finally, I am left with the question why? The ostensible reason is that it's faster and can therefore handle more requests. Even if we accept that as true (grumble, grumble), it only accomplishes that speed by passing the buck off to other servers. When you find a non-responsive site it's not because the static assets like images and HTML text are being served slowly... it's because the dynamic content generated by php/perl/python/ruby/whatever and the underly database from which the data is drawn cannot keep up. Nginx suffers that same failing... while requiring just as many resources because you now have to run so many different servers for each of the languages you want to code it.

Again, the author displays his complete and utter lack of understanding of a web stack and the difference between how Apache handles a request and how Nginx does it. Apache is a process-based server (threads being a type of process), whereas Nginx is asyncronous. Threaded programs can (but usually don't) perform as well as async programs if you are only measuring, say requests per second under a small to average load. What they don't do nearly as well is called scaling. The reason threaded programs don't scale as well is because they typically launch a separate thread for each incoming request. Not only is there latency introduced as the thread is spawned (this can be addressed to some degree with thread pools, but that introduces other issues), but the worst part is that a thread consumes a significant amount of RAM. On 32 bit Linux, this amount is typically 2MB per thread (the default stack size). That's 2MB per concurrent request, even if you are serving a 2K page of static HTML or a small image. The thread can also allocate more RAM if it needs to, but let's assume it doesn't for simplicity.

Let's say your server has 1GB of RAM and we ignore the RAM used by the kernel, Apache itself, PHP/Perl/Python/etc and all the other system processes. How many concurrent requests can you handle? Pretty easy: 1024MB/2MB = 512. Now for most sites, that's a pretty decent amount of traffic. Of course, we've consumed all the memory to do this, so if we get 513 requests we're now forced into swap. In reality, due to the other software running on the system, we'd certainly be forced into swap much, much sooner and we'd have much less available RAM to start with. Realistically, a 1GB Apache server could probably only handle around 100-200 simultaneous requests (much depends on the size of the response). Note that this is not the same as requests per second, which, if you have sub-second response times, 100-200 requests per second might only amount to 10-20 simultaneous requests.

The author's claim that site slowness isn't caused by static files is probably true for 90% of the web. Of course for sites that serve large files this is absolutely false. The longer a request takes to process, the more likely it is we'll start seeing a higher number of concurrent requests (since new requests come in faster than we can respond to them). This is why sites like youtube.com and torrentspy.com have switched to Nginx for serving static content. A single Nginx server can easily handle as many concurrent connections as imposed by the operating system limits (rather than memory limits). You don't need to be youtube.com to bring an Apache server to its knees if you serve many files that are more than a a few kilobytes each. A single Nginx server can easily replace a dozen similarly configured Apache servers for handling static content.

So what if you aren't serving big files? What difference does this make to you? Well, as a result of its asynchronous approach, Nginx also has the benefit of utilizing far less CPU. It doesn't take much traffic to drive Apache up into the 70-100% CPU utilization range. If you can manage to drive Nginx past 20% CPU then you don't need to read this as you certainly already have a team of scalability experts who can tell you the pitfalls of threading already.

The claim that Nginx is faster because it "hands off" processing to some other server is simply ludicrous. Apache does the same thing. The only difference is the protocol they use to communicate with that other process. Apache uses a binary API to pass information to PHP, Perl, Python, etc. Nginx uses HTTP or FastCGI. The processing of dynamic content is still done by the other process, not Apache or Nginx. Further, because Nginx uses less CPU and RAM, there are more resources available for that other process to use to get its job done. Frankly, this assertion left me flabbergasted. It's actually really difficult for me to believe that someone could have such a poor understanding of such a simple software stack.

Finally, I'm going to address the author's grudging allowance that Nginx might be faster ("grumble, grumble") for static content by simply asserting that Nginx stomps an absolute mudhole in Apache when it comes to overall performance. We aren't discussing a few KB/s faster or a few requests/second faster, we are discussing hundreds to thousands of requests per second faster for static content (depending on your hardware), all while using a fraction of the CPU and RAM Apache does before it fails.

Anyway. I'm going to try to forget I read this pile of rubbish. If the author wants to believe that Apache is better, so be it. I just find it unfortunate that a certain percentage of clueless people will find the article informative.

Get the shirt that makes Patrick Swayze cry.

$20USD via PayPal to sales(at)develix.com. Be sure to include your shirt size and shipping info (overseas is fine).

If you'd like more information, you can email me directly cliff(at)develix.com.

Today I stumbled across Spawning, a high-performance WSGI server written by Donovan Preston. It's based on eventlet, which is a coroutine-based server framework for Python.

Initial reports deploying a Django application behind it appear promising, so I decided to give it a try.

For Pylons, the configuration required only a couple of changes to the Paste .ini file:

[server:main]
use = egg:Spawning
num_processes = 2
host = 127.0.0.1
port = 8000

I've deployed it on both this blog and on the Breve site. I'm planning on getting a proper benchmarking environment up later tonight so I can compare it to Paste#http.

First of all, let me note that I tried to install Foresight 2.0 on this laptop only to find the process too annoying to continue. Fedora, for all its shortcomings, remains one of the best distros for dealing with laptops and less-common hardware.

As expected, the Fedora install went flawlessly. I used the defaults except for filesystem layout (sorry, but I will not use ext3 on my machines).

After installing Fedora, the next item on the list was to change /etc/fstab and add noatime to all the mount options. One day the distros will wake up to what a bad idea access timestamps are. Until then this is requisite, especially on laptops, which tend to have slower disks than servers or desktop machines.

Next, replace Fedora's pathetic package management utilities with SmartPM. You must use the bazaar checkout in order to have Fedora's new mirror system supported. Smart has few dependencies and I can't recall any that aren't included with the base Fedora install.

At this point you have a functional system, but there's still a few things missing:

1. suspend, hibernate and resume
2. Vaio function keys (volume up/down, lcd brightness, etc)

Pretty much everything else should already work, so these two will be the focus for the rest of this article.

The very first thing you'll want to verify is that the sony_laptop kernel module is loaded. Most likely it already is, but you should verify this:

# lsmod | grep sony
sony_laptop            30684  0

If it isn't loaded, you'll need to load it by hand using:

# modprobe sony_laptop

and also add an appropriate line to /etc/modprobe.d/sony.conf or something similar so that it's loaded on boot.

At the time I installed this laptop, gnome-power-manager was at version 2.22. This version will not work properly on a Vaio. You can find the 2.23 version in Rawhide that should work properly.

Next, we need to enable HAL quirks for our model. You can get your exact model string with the following command:

# lshal | grep -i system.hardware.product
system.hardware.product = 'VGN-NR160E'  (string)

You'll need this string when you edit quirks files.

Next, edit /usr/share/hal/fdi/information/10freedesktop/30-keymap-module-sony-laptop.fdi and add a section like this:

<match key="/org/freedesktop/Hal/devices/computer:system.hardware.product" string_outof="VGN-NR160E">
  <append key="input.keymap.data" type="strlist">0x06:mute</append>
  <append key="input.keymap.data" type="strlist">0x07:volumedown</append>
  <append key="input.keymap.data" type="strlist">0x08:volumeup</append>
  <append key="input.keymap.data" type="strlist">0x09:brightnessdown</append>
  <append key="input.keymap.data" type="strlist">0x0a:brightnessup</append>
  <append key="input.keymap.data" type="strlist">0x0b:switchvideomode</append>
  <append key="input.keymap.data" type="strlist">0x10:suspend</append>
  <append key="info.capabilities" type="strlist">input.keymap</append>
</match>

You also need to edit /etc/X11/xorg.conf and add the following:

Section "ServerLayout"
       ...
       InputDevice    "Vaio keys" "SendCoreEvents"
EndSection

and:

Section "InputDevice"
       Identifier  "Vaio keys"
       Driver      "evdev"
       Option      "Name" "Sony Vaio Keys"
       Option      "XkbLayout" "jp"
       Option      "XkbModel" "jp106"
EndSection

Add this as a new section, do not replace the existing InputDevice section.

This will enable the Vaio function keys. It also enables the lid closed sensor (which, like the function keys, is an ACPI button). They won't actually work until you've restarted the computer, but don't do that yet.

Next up, we need to set the HAL quirks for the Intel G965 video so that the laptop will resume properly from suspend or hibernate.

Edit the file /usr/share/hal/fdi/information/10freedesktop/20-video-quirk-pm-sony.fdi and add the following section:

<match key="system.hardware.product" string_outof="VGN-NR160E">
  <merge key="power_management.quirk.s3_bios" type="bool">true</merge>
  <merge key="power_management.quirk.s3_mode" type="bool">true</merge>
</match>

Finally, you'll most likely discover that even though the brightness up/down function key invokes the little onscreen indicator in GNOME, nothing actually happens. To fix this, you must run this command:

$ xrandr --output LVDS --set BACKLIGHT_CONTROL native

Note that this must be run every time X is restarted, so you'll probably want to add it to your session (System->Preferences->Session).

You'll probably need to restart your laptop (I've had mixed results restarting hal-daemon), but at this point you should have a fully functional system.

I'd like to add that the integrated Intel G965 video provides more than adequate performance for a non-gamer and works fine with compiz (I haven't tried compiz-fusion because, well... I just don't care).

References:

1. http://www.pihhan.info/sony/sony-hotkeys.html
2. http://ubuntuforums.org/showthread.php?t=626406&page=2
3. http://www.linux.it/~malattia/wiki/index.php/Vaio_VGN-SZ72B

LinuxHater is an effective man. His blog is like a car-wreck you can't peel your horrified eyes away from despite your desire to maintain a sense of decency and the terrible knowledge that you will be haunted by disgust for days afterward. It's damn hard to stop reading his crap.

Anyway, in much the same way that watching the Special Olympics makes us ponder our own humanity, LinuxHater has compelled me to think about what makes FOSS fundamentally different than proprietary software.

First of all, ask yourself, why the hell would anyone write software for free? I mean, writing software is hard in general and you can actually get paid to do it. What kind of moron gives it away? Here's a suggestion: ask your wife/girlfriend why she doesn't charge you for sex. After all, there's clearly a market and, regardless of what you might like to believe, having sex with you isn't likely the pinnacle of sexual experience for a woman. Still, she does it. There's some inexplicable driving force that causes an otherwise sane person to want to get sweaty with your fat ass. We refer to this phenomenon as "love", probably for lack of a more rational explanation.

Free software is driven by a similar force. Software developers, like women, come in two handy flavors: the ones who do it for love of the task, and Professionals. Now, I have nothing against professionals, but there is a very, very small intersection of people who are both paid to do something and love what they are doing. This means that most of the people who write "professional" software hate their jobs. In contrast, I'd be amazed to find the open source contributor who doesn't love his job. This doesn't mean that the the open source developer doesn't face the same frustrations as the professional developer, only that they are differently motivated and one's expectations of them should be different than a professional. To illustrate this point, consider again your wife or girlfriend (or "life partner" for you fruity-rainbow Mac folks). There are certain, um "tasks" that are almost exclusively the realm of professionals. So when a free software developer tells you in no uncertain terms that he's not going to bend over for you, perhaps you should take it with the same disappointed silence you do when your wife declines to do that thing you saw on while browsing thumbnail sites or maybe, just maybe you might help taking take out the trash now and again you lazy bastard.

You might just get lucky.

Somehow I missed that this guy has a pretty interesting blog, even if it's a bit difficult for a Linux user to read due to the apparent vitrol he extols with every other word. It's hard to read because so many of his bitches about Linux and FOSS are dead-on and it hurts a bit to have the weaknesses of Linux pulled from under the rock and cooked with a magnifying glass. It's also hard to read because the truths are so insidiously intertwined with utter crap that your mind has to do somersaults to properly separate and digest it.

LinuxHater lives in a world of hypocrisy. Why? Because he simultaneously rips into FOSS and then turns and compares OSX in a positive light. Maybe he should remove all FOSS software from OSX and then tell me how much he likes it: remove Mach, BSD, Safari, Adium, bash, etc and then tell me how great it is. Apple has done a wonderful job of polishing these FOSS turds with their millions of dollars. Quite an amazing feat. Take free software, spend tons of money cleaning it up, and then call it your own. Brilliant, if your name is Montgomery Burns.

Secondly, OSX runs on a very limited amount of hardware. I don't necessarily think this is a bad thing, but I expect running OSX on, say, a cheap Dell laptop would not be a much happier experience than with Linux (in fact, probably worse). Would suspend work? How about the sound card or 3D acceleration? Maybe, but your odds would be probably much better with Linux. He complains that Ubuntu crashes on him frequently. As anyone will tell you, this is almost certainly caused by hardware problems (either broken hardware or poorly-supported hardware). Any computer with unsupported or broken hardware is almost guaranteed to cause serious stability issues regardless of the OS you use. Next time, buy a Linux-certified PC (they do exist!).

Of course, any time I mention Apple's limited hardware selection, I'm forced (by the buildup of bile) to point out that Apple's computers tend to be among the most expensive. Sure, you can argue that the price difference will be made up by time save by not futzing around with cheaper alternatives. We could also just solve the world's energy crisis by insisting everyone buy a new electric vehicle. I know that in our precious sense of self-entitlement we often forget that some people actually have more time than money, but let us take a moment to remind ourselves that not everyone is skinny because they are rich anorexics, or that they buy clothes at second-hand stores simply because they have no sense of fashion. Linux fills a niche for those not rich enough to get on the proprietary-software wagon.

And yes, I know many of you will point to the $1200 wonders that Apple sells as low-end models. $1200 isn't horrible for a laptop, but those particular laptops tend to be end-of-life models that would normally appear on overstock.com or in the vendor's closeout section if they were PC's. I just purchased a Sony Vaio for $620 from overstock.com (core2 duo, 15" widescreen). I decided to check what a similar Mac would cost, but I couldn't. You either pay $1100 for a 13" Mac, or jump straight to $2000 for the Pro series. Inexpensive 14" and 15" Macs are a thing of the past, apparently. I guess sometimes choice is a good thing.

Anyway, back to the main topic.

Windows has the same problem Linux has in regard to hardware (it must support a wide variety of hardware), but it gets around the lion's share of the problem by getting support from hardware vendors. They don't have to write 435,485,308 drivers, just charge vendors to have the driver certified. Frankly, there is no way to force vendors to do this from the Linux world. Linux must rely on good will from vendors (e.g. nVidia, Intel, AMD) and the work of volunteers (many of whom have other jobs that actually pay money). Of course, even this actual advantage has failed to make Windows any more stable. After 5 or 6 years, XP was actually usable (if mind-numbingly boring), but Vista is the most god-awful piece of crap I've ever had to pinch my nose over while downloading a Fedora DVD to replace it. Microsoft has had exactly two successful operating systems (from a non-suck perspective): NT 3.1 and 2000. Neither of them were very interesting but at least didn't remind you every 3.5 seconds that there was an OS there who's primary function was to prevent you from working.

At the end of the day, I can't help but wonder what the hell LinuxHater's problem is. If he hates Linux and has the money to buy a Mac, then fucking do it. No one is forcing him to accept what amounts to a free gift and certainly no one is asking for his whining about it (even if he's right). If he's forced to use it at work, then maybe I can sympathize a bit (I bitch endlessly when forced to use Windows or Mac for more than a few minutes), but I doubt this is the case. It sounds to me like a case of someone with too much time on their hands (and too much disposable income as well). Nothing like being rich and bored to get that endearing sense of entitlement and superiority going. At the end of the day, it sounds like LinuxHater is probably technically capable enough (and certainly loud-mouthed enough) to make splash in the FOSS world and help address some of the issues he finds so annoying, so the fact that he doesn't simply leaves me believing he needs his diaper changed.

Some people seem to believe that he's making a valuable contribution by pointing out the shortcomings in Linux. If you also consider punching someone in the mouth when they buy you socks for Christmas as valid criticism, then I guess you might agree.

LinuxHater: we get the message. Please take your ball and go the fuck home.

I've been outed.

I've been keeping half an eye on web2py for a while, since it seems to fill a previously untapped niche in the web framework world (TTW development without a lot of complexity).

I decided to check up tonight and I have to say I'm amazed. I'm mildly leery of the fact that it uses a custom ORM rather than SQLAlchemy (SA seems sufficient and is rapidly becoming the de facto Python ORM), but overall, web2py seems to offer a rather appealing way to manage web applications.

I was especially impressed by the number and quality of appliances built on top of it (and the fact that many of them are from third parties is encouraging).

I was also pleased to see that it supports alternative template engines (so I could use Breve with it if I wanted to [and I do]).

SQL doesn't naturally lend itself to nested data. However, there are several clever solutions to managing tree-like data in SQL, but most of them involve one of the following:

• adding columns to your tables that explicitly map the current nesting (and triggers to keep them sync'd)
• a huge brain (google for "sql nested sets")

If you use PostgreSQL, you can avoid both of these drawbacks (well, a huge brain isn't necessarily a drawback if you have one available - unfortunately for me, most of the huge brains I know belong to evil robots). Actually, any database with a decent procedural language will work, but I only know of one in the open source world: PostgreSQL.

Let's assume we have a table that was created like this:

CREATE TABLE accounts (
  id SERIAL PRIMARY KEY NOT NULL,
  accounts_id FOREIGN KEY (accounts),
  account_name TEXT
);

And it's got records like this:

    account_name     | id  | accounts_id
---------------------+-----+-------------
 account             |  78 |
 sub account         | 102 |          78
 sub sub account     | 151 |         102

As you can see, this table represents a tree graph, where an account can have one or more subaccounts.

This is a really simple and easy-to-maintain system. Unfortunately, it's not so easy to ask questions like "given an account, what are all the subaccounts of that account?" Obviously if there's only one level of subaccounts, the answer is easy, but if you have multiple levels, then it isn't quite so simple.

Here's a fairly simple PL/PGSQL function that can answer that particular question for you. Given the id of an account, it returns a list of rows informing you of all the children of that account.

CREATE FUNCTION get_subaccounts ( parent_id INTEGER )
RETURNS SETOF accounts AS $$
  DECLARE
    child accounts;
    schild accounts;

  BEGIN
    FOR child IN SELECT * FROM accounts WHERE accounts_id=parent_id LOOP
      RETURN NEXT child;
      FOR schild IN SELECT * FROM get_subaccounts ( child.id ) LOOP
        RETURN NEXT schild;
      END LOOP;
    END LOOP;
    RETURN;
  END;
$$ LANGUAGE 'plpgsql';

You can now do a query like this:

db=# select id, accounts_id from get_subaccounts ( 78 );
 id  | accounts_id
-----+-------------
 102 |          78
 151 |         102
(2 rows)

I wanted it back, so there you have it.

After reading this article about how the OLPC initiative is going to embrace Windows on the XO, I was struck about how sometimes being focused on a single solution can be detrimental to your goals.

Negroponte said he was mainly concerned with putting as many laptops as possible in children's hands.

On the surface, this is laudable. However this is a silly goal unless there's a corollary. In the OLPC project, education is the corollary. Without the education aspect, we may as well expect the XO's to be used mainly for browsing MySpace and internet porn. So simply "getting laptops... in children's hands" isn't sufficient. They don't need laptops so much as they need learning tools. If the laptop isn't a learning tool then it's just more strain on the environment and a distraction from real solutions.

This is where Negroponte's narrow vision begins to fail his goal.

Proprietary software, such as Windows, OSX, or Flash, isn't viable as a learning environment unless you are able to spend money to make it into one. The entire point of OLPC is to get learning tools into the hands of children who have no money. Providing them with proprietary software is providing them with a dead-end solution. Will Microsoft continue to provide them with free upgrades that are available over low-bandwidth connections? What about development tools and API's? Can we expect that in ten years the XO will be as up-to-date as Windows 98 is today? Microsoft doesn't have a good track record of supporting legacy software and interfaces nor for providing an inexpensive upgrade path.

I consider providing Windows-based XO's the equivalent of selling African farmers crop seeds that contain the so-called "death gene". It solves their immediate problem today, but gives them no way to become self-sufficient. In fact, all it guarantees is that they will become locked into the corporate food-chain and another revenue source for Microsoft (or more likely, another abandoned project, once the PR has faded).

Exactly what I was thinking.

Continuing my tradition of abusing whatever language is under my keyboard (and in the interest of stress-testing Breve), here's a template that traverses its own DOM and extracts out CSS selectors:

assign ( 'selectors', [ ] ),

macro ( 'get_selectors', lambda tag, is_tag: (
    macro ( 'css_sep', lambda attr:
        '.' if attr == 'class' else '#'
    ),
    selectors.extend ( [
        "%s%s%s { }" % ( tag.name, css_sep ( _k.strip ( '_' ) ), _v )
        for _k, _v in tag.attrs.items ( )
        if _k.strip ( '_' ) in ( 'id', 'class' )
    ] )
) ),
macro ( 'walk_dom', lambda tag:
    tag.walk ( get_selectors, True ) and tag
),
macro ( 'walk_results', lambda selectors:
    pre [ '\n'.join ( selectors ) ]
),

html [
    head [ title [ 'macro madness' ] ],
    body [ walk_dom (
        div ( class_='text', id='main-content' ) [
            img ( src='/images/breve-logo.png', alt='breve logo' ),
            br,
            span ( class_='bold' ) [ "Hello from Breve!" ]
        ]
    ), walk_results ( selectors ) ]
]

This outputs the original page with the CSS selectors at the end like so:

<html>
  <head>
    <title>macro madness</title>
  </head>
  <body>
    <div class="text" id="main-content">
      <img src="/images/breve-logo.png" alt="breve logo"></img>
      <br />
      <span class="bold">Hello from Breve!</span>
    </div>
    <pre>
      div.text { }
      div#main-content { }
      span.bold { }
    </pre>
  </body>
</html>

Don't try this at home, professional driver on closed-course, etc.

Also, this requires Breve 1.2.1 or greater (it required Tag.walk to be changed to return self rather than None).

I was going to wait until later today (it's 2:30a.m.), but I couldn't see a reason to withhold the release.

Here's a short summary of what's new:

• Tag multiplication - you can multiply a tag by a list of dictionaries and have the dictionaries values propagated as attributes and data.
• Macros - Define a function inside a template.
• Assign - Create new variables inside a template.
• Tag.walk - traverse a tag and all its children, performing an action on each tag.
• Improved include function - included templates are now evaluated within the scope of the caller and several former limitations are removed.
• Template code cleanup.
• Unit tests for major API features.
• Improved documentation.
• Tools (soup2breve, xsd2breve, html2breve) are now properly installed.

Be sure to check out the documentation and review the tests to get an idea of what Breve is about.

The evil spirit of Python? I knew it.

I'm thinking of stealing satanpythonface as an IRC handle.

My unit testing paid off (found one bug and a lot of confidence) so 1.2 is slated for release tomorrow.

This blog and the Breve site are already running on it =)

I've been resisting testing frameworks for a long time. I tend to test by using (and apparently being adept at writing relatively bug-free code), but one place tests provide true value is when modifying code.

I'm reworking the internals of aspects of Breve and I realized that I needed to do more than run visually inspecting the output of the included examples (I also test by deploying trunk onto the breve site and this blog prior to releases, but since I don't necessarily exercise all aspects of Breve, this is only a feel-good test).

Anyway, part of the daunting task of testing a template engine is that you must test against literal chunks of HTML. I decide to see what Genshi does, since whether I like that engine or not, it is the 900lb gorilla of Python template engines, and I know Chris Lenz and his cohorts have put a lot of effort into having a solid development process.

I took a look at Genshi's test suite and it was enough to get me started, although I wasn't thrilled about the tests having the expected output embedded directly in the tests. It seemed a bit cluttered and I knew it would get more cluttered as I implemented tests for corner cases (nesting of directives, inheritance, and combining features to check for side-effects).

I hacked about for a bit and came up with a system that seems to be working pretty well.

If you've been avoiding Breve because you're loathe to use software that doesn't include a test suite, you are now encouraged to look again.

My work on getting macros to work in Breve forced me to figure something out that's going to drastically improve Breve.

The fundamental problem is that when Python calls a function, that function gets a local namespace. This is usually a good thing, but it's not always what you'd want in a specialized environment such as a template engine. For example, Breve has an include ( template ) feature that lets you include fragments of Breve code into another template. This worked okay, except that there were all sorts of limitations on what you could or couldn't do in an include file. Further, it required building a dictionary to use as the included file's namespace to make sure that user-supplied variables would be available to it.

Enter sys._getframe. This little-utilized function accepts an integer and retrieves the frame that many levels up the execution stack. For example sys._getframe ( 1 ) returns your current frame of execution. This can be used, for example, to retrieve the name of the function you're currently in (useful for debuggers and the like).

What's interesting to me, however, is that sys._getframe ( 2 ) returns the frame of the function that called the current function. This allows you to retrieve the global namespace of that function and put things back into it. In short, you can make the current function change or create variables in the outer scope:

import sys

def caller ( ):
    return sys._getframe ( 2 )

def f ( ):
    g ( )
    print v

def g ( ):
    frame = caller ( )
    print frame.f_code.co_name
    frame.f_globals [ 'v' ] = 2

f ( )

Running this code will output:

f
2

What this means to Breve is that now when a template calls include(), now the included file will be executed in the caller's namespace. It's actually even simpler than what's outlined above because Breve templates are passed to eval, so all that I need to do is pass caller().f_globals as the globals parameter to eval and the magic is done:

co = compile ( 'fragment.b', 'fragment.b', 'eval' )
return eval ( co, caller().f_globals, { } )

This will provide a much smoother interface. I also expect to apply this to template inheritance.

You can find out more about this topic here.

As it turns out, in Python 2.4, eval ( ) was changed to allow any sort of mapping object (not just a plain Python dictionary) as the locals namespace. What this means is that we are free to pass in an object that can pretend it has all the variables in the world:

class Proxy ( dict ):
    def __getattr__ ( self, attr ):
        return Proxy ( )
    def __getitem__ ( self, key ):
        return Proxy ( )
    def __call__ ( self, *args, **kw ):
        return Proxy ( )
    def __str__ ( self ):
        return ''

eval ( 'x.y ( ).z [ 3 ]', { }, Proxy ( ) )

Despite the fact that none of the variables (x, y, z) exist in the code fragment we're evaluating, we get no exceptions. In fact we don't get a peep of output.

Clearly you'd need to flesh this class out a bit (like handling mathematical operations) and you'd probably want to actually have some values in it, so you'd need to define __setitem__ and __setattr__ behaviour, but that's left as an exercise for someone else.

Once I've made a maintenance 1.1.8 release this weekend, Breve 2.0 will be next on stage.

The big version jump is due to the fact that Breve 2.0 will have some backwards-incompatible changes (although I'll try to make a legacy API available). These changes won't affect Breve templates, only how they are invoked.

Join us on the list and discuss it! Your ideas will be welcomed with the arms of condescension and your fears gently patronized.