Template Engines

2.1   Model-View separation

So, with a template engine the programmer should separate the presentation (=view) from the logic (=model). It is very important to strictly separate these two parts! Otherwise these parts won't be independent anymore, and you will lose all advantages of a template engine!

Unfortunately, the template engine itself cannot completely enforce this separation without some serious restrictions on the view (see below). And -- at least in my opinion -- the template-engine should not restrict the design of the view or even make some designs impossible.

Since the separation cannot be done automatically, here are some rules:

• Always cleanly separate model and view.

  Don't put parts of the view into the model, don't put parts of the model into the view. Not even temporarily.
• Always push the data from the model to the view.

  Accessing/modifying the model from the view is forbidden. Pulling data is forbidden.

  So, the model may only provide a bunch of values, and the view then uses these values.

• All data from the model must be display-/layout-independent. Display/layout-information in the data is forbidden.

• Make all computations of the model in the model.

• The view must only perform computations for view-specific and model-independent "things".

2.2   Different approaches

But in reality, there are different kinds of template-engines, with different approaches. I'll group them into 4 categories (although many of the existing template-engines are somewhere between these categories):

1. substitution only:

   The template only contains placeholders which are replaced by data. No loops, no conditions etc. are possible in the template. This is the simplest but also least powerful kind of templates.

   Examples: printf-formatstring, python's string.Template, Templayer.

   It should be clear that this is insufficient for most cases.

2. substitution + if-defined + loop-over-data + recursive macros:

   The template can contain:

   • placeholders (for string-substitution)
   • if-defined-conditionals (to test presence/absence of some data)
   • foreach-data-loops (for multi-valued data/lists)
   • recursive macros (to walk recursive data-structures)

   This is powerful enough for some websites. And it cuts down the power of the template, to prevent that anything which belongs to the model can be done in the template/view.

   This approach is described in detail in an interesting paper, named Enforcing Strict Model-View Separation in Template Engines. Although I don't agree to many conclusions of the author (and think that these templates are definitely not powerful enough), it may be worth reading.

   BUT: This approach puts some serious restrictions on the view: Some designs simply can't be done (see Why if/for/...) -- or can only be done if you put parts of the presentation into the model code, which is a no-go. This often makes this approach unusable.

   Examples: StringTemplate (the template of the above paper), ...

3. substitution + conditionals + loops + macros + "embedded expressions" with restricted access:

   In addition to the above, the template may also contain "free" loops, tests of variable-values, and has "expressions of a embedded language" (e.g. for calculations, formatting etc.). But this "embedded language" can only access a restricted set of variables/functions; it especially cannot access or modify the model.

   This is powerful enough for all cases. And it prevents direct access and modification of the model in the template. It can't prevent that tests, calculations etc. which belong to the model are done in the template -- this is in the responsibility of the programmer/designer. But that's the price for a full flexible, unrestricted view.

   Examples: unfortunately not many, but e.g. my pyratemp

4. unrestricted templates, i.e. unrestricted embedded code:

   Many template-engines simply include a full-powered programming language without any restrictions into the template. But this has the drawback that even the model can be modified in the template, which can completely countermine the model-view-separation.

   Examples: Mako, Cheetah, TemplateToolkit, EmPy, ...

2.3   Why if/for/calculations etc. are necessary in the View

There are many people who think that the "category 2" above is sufficient. Unfortunately, this is not the case. There are a lot of things which clearly belong to the view, but are not possible without variable-testing, for-loops, calculations, formatting etc.

This can be demonstrated best with a few examples. Note that these examples are not constructed, but really occurred in existing templates:

• "free" for-loops:

  • Imagine a fixed form, with e.g. a table which should always have 10 rows, no matter how much data exists. If there is less data, the table will have some empty rows. If there is more data, there will be 2 (or more) tables.

    This probably isn't needed very often in websites, but definitely occurs in fixed formats, e.g. in LaTeX.

  • Or simply take an enumerated list like this:

    	Value
    1	one
    2	two
    3	three
    4	four

    The enumeration and enumeration-style here belongs to the view.

    (Note: This is possible in some "category-2-template-engines", but only due to some "magic-variables" like loop-counters. But it then only works for "1,2,3,..." and not for "10,20,30,..." or "I,II,III,...".)

• formatting:

  • How should a date look like? "CCYY-MM-DD" or "DD. Month CCYY" or "MM/DD/YY"?
  • How many digits of a float should be displayed? One output may e.g. require 2, an other 3. Some other needs a "+" in front of a positive number...

• calculations:

  Imagine a bill (e.g. in LaTeX) with a variable number of items, which may be broken into several sheets of paper. The calculation of a subtotal at the end of each page clearly belongs to the view.

All these examples clearly belong to the view, but are not possible with the pure "category 2" approach. And these are only some examples -- there are a lot more...

This is why such "category 2"-templates often get extended bit by bit, until they end in "category 3 (or 4)", but then have an inelegant concept.

2.4   What a template-engine needs

Here's a short description of how a template-engine should look like in my opinion:

• stand-alone (i.e. work without a webserver)

• result-independent: able to create any kind of textual data, e.g. HTML/XML (both static and dynamic), LaTeX, PostScript, email, ...

  (Contrary to most XML-template-engines, which can only create XML-documents.)

• simple, small, lightweight, easy to use, well-documented and extensible

• clear template syntax ("There should be one -- and preferably only one -- obvious way to do it." [2])

• completely Unicode

• good syntax-error-checking and good error-messages, to make the writing and debugging of templates easy

• fast

• secure, so an "untrusted template" cannot compromise your system.

  (think of a sandbox, prevention of infinite loops, etc.)

• basic functionality from "category 3" above):

  • placeholders/string-substitution
  • conditionals
  • loops
  • macros (with parameters)
  • restricted "embedded expressions" (formatting, arithmetics etc.)

• additional functionality:

  • inclusion of other templates (for reusability and "things" which are used in several templates)
  • escaping-mechanism (e.g. for html: escape < > & ' ")
  • (MAYBE) definition of the result-encoding

• complain about undefined/non-existing variables in the template by default

  (Most template-engines silently ignore undefined variables in the template. But silently ignoring errors is bad, and wrong data is much worse than no data.)

Since unfortunately I didn't find any existing template-engine which fitted these needs, I created an own template-engine which accomplishes most of the above: pyratemp

3.1   Benchsimple

"Benchsimple" creates a simple html-page with a navigation-bar and a table with 50 entries (5 rows * 10 columns). It only uses a few template-features that are supported by all templates (escaped substitution, conditionals, loops), and so there's no Unicode, no included templates, no arithmetics, no template-defined formatting etc.

I've tested pyratemp, cubictemp, Jinja, Cheetah, Mako, EmPy, evoque and SimpleTAL, and also compared them to manually written code.

But note that the following "results" can only give you a hint how fast a specific template-engine might be.

Results:

Notes:

• run on a AMD Athlon 64 3200+, 2.0 GHz, Debian Etch
• Python 2.4 (except evoque and cubictemp-2.0, which require Python 2.5)
• all times are in "ms"
• the times are pure "template-execution"-times without the time needed to start the Python-interpreter, load the files and compile the code
• "import": time to import the module
• "complete": complete time to parse and render the template
• "parse only": time to parse (+maybe compile) a template
• "render only": time to re-render an already parsed/compiled template after it has been rendered the first time

I also wrote some python-code, which also creates the same page, but without a template-engine. This shows how fast the template-engines are compared to manually written Python-code. Additionally, I modified the code to optionally don't escape special characters, to use the variables in the substitutions directly without eval() and to use psyco. This shows which part takes how much time.

4.1   Overview

• http://en.wikipedia.org/wiki/Template_engine_(web)
• http://en.wikipedia.org/wiki/Template_processor
• http://wiki.python.org/moin/Templating
• http://de.wikipedia.org/wiki/Template_Engine (in German)
• http://wiki.python.de/Template-Engines (in German)

Additionally, I've created the same html-page using different template-engines in Benchsimple, so look there to get a first impression how these template-engines look like.

Since I haven't tested every template-engine in detail, some of the fields below are empty, and I probably haven't found some hidden features of some template-engines. So, please tell me if anything below is wrong.

lines-of-code:

Number of "code"-lines as reported by pylint, without docstrings, comments and "empty" lines.

Unicode:

A template-engine should accept both templates and data in Unicode, but some of them don't and some only do optionally.

error-msg.:

How good are the error-messages for template-syntax-errors etc.? Good error-messages are extremely useful when developing or "debugging" a template, and save a lot of time.

+: good error-messages, including line- and column-number of the error and the names of undefined variables

o: only partially the position of an error or name of an undefined variable is reported

--: error-messages do not tell where the error occurred

escaping:

Ideally, escaping should be easy to use in the template (so you don't accidently forget it somewhere), should escape <>&"' for HTML/XML, and be configurable for other escaping.

+: escapes <>&"'

o: only escapes <>&

--: no escaping

undef. vars:

What should be done if the template uses a variable (or other object) which does not exist? Some of the template-engines ignore undefined variables, some complain (and exit), and some are configurable.

embedded code:

Can (Python-)statements be included in the template?

In contrast to expressions (see below), statements do not have a value but "do something" (e.g. if/for, print, raise, return, import etc.). In my opinion, templates should not directly include statements from a programming-language but define a small set of needed "statements" themselves.

(see also: http://en.wikipedia.org/wiki/Statement_%28programming%29)

embedded expr.:

Can (Python-)expressions be included in the template?

An expression is everything which evaluates to a value, e.g. variables, arithmetics, comparisons, boolean expressions, function/method calls, list comprehensions etc.

(see also: http://en.wikipedia.org/wiki/Expression_%28programming%29)

sandbox:

Are the embedded expressions/statements sandboxed/restricted, or can they do anything? "Anything" of course includes things like open("/etc/passwd") and worse.

Note that these sandboxes do not limit the memory- and CPU-usage nor prevent infinite loops; if you want to limit these, use ulimit.

+: sandbox, where I don't know any way to break out (if you know any, please let me know!)

--: no sandbox

4.2   pyratemp

pyratemp is my own template-engine. It's probably (one of) the smallest complete template-engines [7], and it uses a very small set of special syntax in the templates. Both is good, because it reduces complexity and the probability of bugs and leads to a easy-to-use and intuitive user-interface.

Additionally, it is quite fast (although not optimized for speed), uses a (pseudo-)sandbox and produces exceptional good error-messages (e.g. pyratemp.TemplateSyntaxError: line 18, col 1: invalid keyword 'fo'), which is extremely useful.

pyratemp was inspired by cubictemp. Since there were some essential features missing in cubictemp (see below), after trying to extend cubictemp, I decided to write my own template-engine from scratch, which is nearly as small and simple as cubictemp, but without its weaknesses.

More details can be found on my pyratemp-page.

4.3   cubictemp

cubictemp is probably the smallest of all template-engines. I completely agree to a statement on its homepage:

There are many large, over-designed Python templating systems out there.

Unfortunately, there are some essential features missing in cubictemp, like Unicode-support, inclusion of other templates, good/helpful error-messages, and some kind of a sandbox. Additionally, its conditionals (if/else) are weak, since the don't have a "elif" and can't contain template-syntax in its branches.

4.4   Jinja

Jinja 0.7 was probably the best template-engine without embedded Python-expressions. And it included a sandbox, so a template could not do "bad things". But its embedded expressions were not powerful enough for many purposes, and so embedded Python-expressions were added in later releases. (Note that this is exactly what I predicted at the end of Why if/for/calculations etc. are necessary in the View.)

But although I haven't tested Jinja 2.0 yet and don't know how good its sandbox is, Jinja may be worth a look.

4.5   evoque

I recently found evoque, and was surprised that it has many things in common with pyratemp, and nearly has everything I described a template-engine should have (see What a template-engine needs).

But after reading parts of its documentation and a short test, I also found a few things I don't like -- especially in comparison with pyratemp:

• It's really counterintuitive to generate a document with an exact amount of whitespace. You can find an example in my benchsimple-code, where you can compare the whitespace in the template with the ones in the result. (Note that in the benchsimple-example, the whitespace before "</ul>" is ignored, and instead the whitespace before "$rof" of the for-loop above is used for "</ul>"!)

  Although that's probably not a problem when creating HTML/XML, it is when creating other documents.

• "sandbox": By default, the sandbox is disabled, and I don't know if this sandbox is secure or not.

  But a look into the sourcecode showed that the programmer forbids known-unsafe functions instead of explicitly allowing only known-safe functions, which is definitely the wrong way and leaves a bad impression.

• And it's code base is larger and worse documented than pyratemp. ;)