Blooki

Nightly builds

Tue, 19 Oct 2004 05:24:26 GMT

As Blooki now has nightly builds, I'm one step closer to getting tolerable score on The Joel Test:

Do you use source control? yes
Can you make a build in one step? yes (now)
Do you make daily builds? yes (now)
Do you have a bug database? yes/no (I know, I know, it's either yes or no; it's not publicly available, but it exists ;))
Do you fix bugs before writing new code? yes
Do you have an up-to-date schedule? no (only plans)
Do you have a spec? no (not written at least)
Do programmers have quiet working conditions? yes
Do you use the best tools money can buy? yes
Do you have testers? yes, one tester and one developer
Do new candidates write code during their interview? n/a (we are not hiring ;))
Do you do hallway usability testing? yes

Nightly builds are available here.

Content negotiation

Wed, 13 Oct 2004 21:46:00 GMT

As Ian Hickson explained, Sending XHTML as text/html considered harmful, but it's fine "send XHTML as application/xhtml+xml to UAs that support it, and as text/html to legacy UAs."

To implement this I've extended the content-type parameter in template, so that instead of text/html it is now possible to write application/xhtml+xml,text/html. Server will then use Accept header to identify what content type should be served; it will serve application/xhtml+xml to Firefox and text/html to InternetExplorer. It also applies to RSS content types: Atom template has application/atom+xml,application/xml as the content type.

All this is done by the following lines of code (thanks to Gisle Aas for his HTTP::Negotiate):

 my @types = split /,/, $template->content_type;
 my $pos = 0; # ::Negotiate sorts results by quality and then by size (asc)
 my $variants = [map {[$_, 1, $_, (undef) x 3, ++$pos]} reverse @types];
 my $headers = new HTTP::Headers Accept => $request->cgi->http('accept');
 my $type = HTTP::Negotiate::choose($variants, $headers) || pop @types;

Dynamic Publishing: Bake or Fry?

Tue, 28 Sep 2004 20:54:00 GMT

I was reading Tim's MT 3.1 Dynamic Publishing Blues and it reminded me a topic I read almost two years ago -- Half baked and little fried -- about dynamic vs. static generation. The topic of Timothy's post is about controversy of recently introduced dynamic publishing in MovableType that uses PHP as its engine. While Ben Trott explained their reasoning behind the decision I'm not sure I agree with him.

In his analysis Ben describes several options and dismisses most of Perl related options as inefficient. Without considering speed of interpreters, I guess he is talking about startup penalty of perl interpreter that is called on every request. This cost is definitely there, but the solution reminds me advice to buy more hardware to solve a performance problem without looking at optimizing an algorithm that is being used. Let's look at it.

The stated problem of reducing latency and server load can be solved in several ways:

1. Expires and Last-Modified headers. The script can return these headers (along with ETag) to make the page cacheable. The tricky part is to set proper Expires header. It should be long enough to minimize calls to the server and short enough to allow those calls when the content changes. This can be achieved by giving different values to different pages or even doing something similar to what google bot does visiting more often pages that change more frequently; +10% on expiration and -50% on modification may be a good start. Expires/Last-Modified also works well with static content (images, client-side scripts, and stylesheets). Expires header can be set to a fairly large value; if a file is updated, it can be served using a new URL.

2. Handling of If-Modified-Since and If-None-Match. The script should return 304 if content hasn't been modified. It doesn't even need to have a copy of the page; all it needs to know that it hasn't changed. It may be as simple as one LastModified time per blog/site for rarely updated sites that invalidates all caches; or as complex as dependency tracking to know exactly what information was used to generate a page.

3. Local cache of generated pages. The dependency check still needs to be done, but the page may already be generated and served from a local cache (likely file system). At this point most people would point out that all this can be done by saving generated pages as static pages and have them to be served by a webserver using little bit of mod_rewrite-like magic. While that's true, there are still several things that need to be addressed:

Expires values need to be configured and they likely to be static
Any custom headers need to be configured
No personalization is possible (new since the last visit and other similar things)
Authentication requests may need to be handled separately
No parametrized request: pagination, searches and the like.

Pages can be cached (they can even be compressed) along with their headers and served when necessary. While this may be a viable option in many cases, there is still a question of how this cache should be in/validated: dependencies can be checked on every request, or they can be checked when pages are added/updated/deleted.

4. Template fragment caching. While the script may not cache the entire page, it still may be feasible to cache some of page fragments, especially the most time consuming or most frequently used, as recently updated items or list of subcategories that are likely to be used across many pages. This requires tracking of what fragment uses what information, so they can, again, be properly invalidated, but this may not be as complex as it seems.

5. File/memory cache. While template fragments may not be cached, the information that is necessary for page generation can be cached in memory (applicable to mod_perl, daemon and similar server solutions) or in files (this works well for filesystem-based solutions like Blooki, Blosxom, and other file I/O hungry solutions). It is not necessary to cache all the information; in most cases modification date/time, title, and some meta information is enough.

6. Access optimization. This probably doesn't apply to MT, but it definitely applies to Blooki (which uses filesystem to store its information). Even when information is not available in a cache, it's still possible to optimize a process of getting this information. Blooki is super-lazy about getting the stuff it needs. First, it's driven by templates; if it's not requested by a templates it probably won't be processed. Second, it only read directories first without even stat'ing files in them. Then it only stat files if you ask for their modification times. And then it only reads their content if you ask for title, meta, or other information.

7. Direct access. If nothing else helps, then information has to be read and page has to be regenerated from scratch.

Now, back to the original question: was it worth it? Unfortunately, it's not clear from the description if Perl is being used at all when PHP-based rendering is used, but as far as I understand it is (please correct me if I'm wrong). Switching from Perl to PHP only addresses items 2 and 3; doing everything else still requires Perl interpreter (and hence startup penalty). Now, both 2 and 3 can be quite effectively achieved by using a local proxy/cache, which some users may already have and for those that don't it is a one-time deal and is much easier than PHP engine integration. In my opinion the asnwer is clear.

Download

Tue, 28 Sep 2004 05:49:51 GMT

Blooki 0.48 released 28 Sep 2004. Available as .zip and .tar.gz archives. Downloaded package can be installed in just a few clicks.

Changes in the latest version (0.48 Tue Sep 28 01:13:32 2004)
- fixed daemon detection in IfModule directive
- added Markdown filter
- added Excerpt filter (to generate 'see more...')
- added pagination for list of entries
- added next/previous for entry pages
- added better diagnostics on template error; now includes tag/variable names and parameters

Nightly builds can be downloaded here (.zip and .tar.gz).

Licensed under Creative Commons Attribution-NonCommercial-ShareAlike 2.0.

Installation

Tue, 28 Sep 2004 05:12:41 GMT

Blooki runs just anywhere where Perl runs. If you plan to run Blooki as a standalone server, there is nothing to install. Assuming blooki-app is a current directory, the server can be started by running the following command:

  > perl blooki.pl --port 81

Alternatively, if you have perl associated with .pl extension you should be able to just double click on blooki.pl file to start blooki daemon server. The server will use port 80 by default.

To install Blooki to use it with your webserver as CGI/mod_perl script type the following:

  > perl Makefile.PL
  > make
  > make test
  > make install

This will:

Detect whether you have a webserver available to set proper default values
Create Blooki configuration file populated with the values you enter
Create .htaccess file(s) that match your configuration, and
Copy all application and configuration files into proper locations.

Users on Windows platform may try nmake instead of make. If no make is available, then Makefile.PL can be run with --nomake option that will also install necessary files:

  > perl Makefile.PL --nomake

You can still run tests using perl test.pl command.

The script will also detect whether you have mod_rewrite and mod_perl extensions available and will generate necessary configuration files based on information you provide.

Although webserver detection in the installation script currently works only for Apache server, the way package and configuration files are organized should allow you to run Blooki with any other webserver with minimum or no changes.

Pagination in Blooki

Sun, 26 Sep 2004 19:11:00 GMT

Shelley has been writing about upcoming Wordpress 1.3 release and one of the features that will be included in the release -- pagination. I got curious about how difficult it would be to add something like this to Blooki.

First, I added two template variables (entries::prev and entries::next) that calculate number of entries to display and URL to use.

The rest was easy: update template to provide default parameters for number of entries listed and starting position:

TemplateVar page::entries <<.
  $entries{
    start => $request{args}{start} || 1,      # start from 1 by default
    display => $request{args}{display} || 20, # display no more than 20 entries
    header => $request{isentrypage} ? $entry::prevnext : '', # show entry prev/next
    footer => q[$entries::prevnext], # show entries prev/next if necessary
  }
.

This code checks if start and display parameters from query string available or uses 1 and 20 by default; it also sets footer to the value of $entries::prevnext template variable, which is defined as follows:

TemplateVar entries::prevnext <<.
<div class="prevnext">$get{join ' | ', grep {length}
  $entries::prev ? qq[<a href="$entries::prev{url}">&lt;&lt; Prev $entries::prev{display}</a>] : '',
  $entries::next ? qq[<a href="$entries::next{url}">Next $entries::next{display} &gt;&gt;</a>] : '', 
}</div>
.

This code generates prev/next links. As parameters are not hardcoded anywhere, users can check start and display values as they need (start can even be negative; in this case it will display entries from the end of the list, just like @foo[-2..-1] does).

Excited by the ease of the change I also tried to do prev/next links on entries. This didn't required any coding at all; only a small template change:

TemplateVar entry::prevnext <<.
<div class="prevnext">$get{join ' | ', grep {length}
  $entries{display => 1, sort => "-modified_on", modified_on => "<$entry{modified_on}", q[<a href="$entry::permalink">&lt;&lt; $entry{title}</a>]},
  $entries{display => 1, sort => "+modified_on", modified_on => ">$entry{modified_on}", q[<a href="$entry::permalink">$entry{title} &gt;&gt;</a>]},
}</div>
.

This code displays prev/next links with titles of entries (if there is previous/next entry). It's also easy to make it display prev/next entries in a particular category, just add a category filter category => $entry{category}:

TemplateVar entry::prevnext <<.
<div class="prevnext">$get{join ' | ', grep {length}
  $entries{display => 1, category => $entry{category}, sort => "-modified_on", modified_on => "<$entry{modified_on}", q[<a href="$entry::permalink">&lt;&lt; $entry{title}</a>]},
  $entries{display => 1, category => $entry{category}, sort => "+modified_on", modified_on => ">$entry{modified_on}", q[<a href="$entry::permalink">$entry{title} &gt;&gt;</a>]},
}</div>
.

Markdown Integration

Sat, 25 Sep 2004 19:11:00 GMT

Even though Blooki currently provides Textile, Tiki and ConvertBreaks filters, I've been thinking about adding Markdown as several people asked me about it. Markdown is a perl module, so I didn't expect any troubles integrating it with Blooki. There were few challenges though both related to the fact that Markdown is not distributed as a Perl module; instead it's distributed as a Perl script.

First, since it can run as a script and as a plugin for MovableType and Blosxom it checks if either of those is avaiable and if not, it waits for input to be processed. This is easily fixable by changing defined($blosxom::version) (line 184 in Markdown 1.0) to something like (caller(0))[7], which returns true if the script was called using use/require/do.

Second, the old IfModule directive in a config file only processesed "real" modules/packages, and not files. So I extended it to accept files and Perl versions (as Markdown requires Perl 5.6 and later). Now the fragment that loads Markdown looks like the following:

  <IfModule 5.006 Markdown.pl> # Markdown requires Perl 5.006
    AddModule Markdown
  </IfModule>

This goes into Blooki 0.48.

The Cat is Out of The Bag

Sat, 25 Sep 2004 14:23:00 GMT

I've been quitely working on this application for some time, but it looks like a good time to make it public. The package seems to be good enough to be useful to other people (one of the things that is missing is documentation). I'm dog-fooding blooki to run this site: it's used to run a development weblog and I also plan on using it to power support forums and a project documentation website.

Contact

Fri, 24 Sep 2004 06:25:13 GMT

Emails with your comments, suggestions, feature requests, bug reports and complaints should be sent directly to the author (Paul Kulchenko).

If you are interested in helping with the project you can send an email to contribute@blooki.org.

Welcome

Wed, 22 Sep 2004 05:09:36 GMT

Blooki is a simple yet powerful tool that combines features of weblogs and Wikis. It brings together simplicity of Blosxom and richness of MovableType allowing you to store and share things you know and to slice and dice that information almost any way you want.

Blooki provides readable and search engine friendly URLs, flexible and lightweight hook-based plugin architecture, simple and extensible template engine with custom tags/variables, config-driven customization, file-based data storage, easy setup, several execution options (standalone server, CGI, or mod_perl), and more.

You can read about Blooki features and philosophy or download, install and have fun with it.

Philosophy

Wed, 22 Sep 2004 05:09:35 GMT

Most of the good ideas implemented in Blooki have been borrowed from other tools and languages: simplicity and file-based data storage from Blosxom, hook-based extensible architecture and config-driven customization from Apache, template engine with custom tags and variables from Perl itself, super-easy install and several execution options from Instiki, rich object-oriented model from MovableType, free links from Wiki, and site design from Stopdesign.

Blooki has been design with several goals in mind: simplicity, succinctness, hackability, efficiency, and learnability.

Simplicity -- Provides small core with powerful, highly orthogonal modules on top of it.
Succinctness -- Gives you ability to solve common problems with little or no coding.
Hackability -- Allows to change (almost) everything, including the syntax.
Efficiency -- Driven by a template engine with multi-level lazy initialization and caching.
Learnability -- Requires very limited time and effort to learn.

Features

Wed, 22 Sep 2004 05:09:34 GMT

Dynamic publishing
Any changes made to pages, templates, or stylesheets are immediately visible.

Powerful plug-in architecture
While config-driven customization and extensible template engine minimizes the need to write custom plugins, Blooki features powerful plugin architecture that allows you to extend the system in many ways and easily change the existing functionality.

Template-based rendering
All pages that Blooki generates are defined using a simple Perl-based template language. Existing set of tags can be easily changed or extended with custom tags. Multiple templates per entry and different templates for each category are also supported. You can even associate a template with a particular URL (for example, to provide custom "file not found" and "not authorized" pages).

Super-easy installation
Blooki provides zero-step (for servers that use built-in webserver) or one-step (for servers that run using external webserver) install.

Multiple execution options
Blooki gives you a choice of single-threaded, forking, or multi-threaded daemon server (using built-in webserver), CGI, or mod_perl execution options.

Flexible archiving
Blooki supports archiving by day, month, year, category, or a combination of those; all this can also be applied to select posts from a particular category.

Config-driven customization
Almost all aspects of the system (templates, modules/plugins to run, and their configuration) can be customized by changing configuration files on a category-by-category basis.

Multiple categories per post
Blooki allows you to have an entry linked to multiple categories.

Web standard compliance
Everything that Blooki generates -- XHTML, CSS, and RSS code -- is in compliance with web standards out of the box.

RSS syndication
Blooki includes support for RSS 1.0 (RDF), RSS 2.0, and Atom syndication formats.

Several markup choices
Not only does Blooki support several markup filters (Textile, Tiki, and simple ConvertBreaks), but it also gives you several ways to select what filters to apply (config file, template, and entry) and allows you to apply multiple filters.

Multiple sub-categories and sub-sites
Blooki allows you to have unlimited number of sub-categories and sub-sites. You can limit their visibility to/from other parts of the system by applying Hide and Root plugins.

Minimum requirements and dependencies
Blooki runs out of the box and only requires Perl 5.005 or later. Some plugins may have additional requirements.

Free for taking
Blooki is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 2.0.