1.1   Features

My template-engine has everything I think a template-engine needs:

• variable-substitution, incl. special-character-escaping (e.g. HTML, mail_header, LaTeX)
• conditionals (if/elif/else), loops (for/else)
• template-defined functions/macros/variables
• inclusion of other templates
• very powerful expressions (due to Python)
• pseudo-sandbox: the Python-expressions are evaluated inside a "pseudo-sandbox" which prevents that "bad things" are done by accident; and even a "real" sandbox could be added
• clear template syntax ("There should be one -- and preferably only one -- obvious way to do it." [2])
• non-XML, so it can be used for any kind of documents
• good error-handling and good error-messages, incl. the exact position of an error in the template-file
• completely uses Unicode
• fast and lightweight
• modular: pyratemp consists of several parts, which can be used separately, or even be replaced by an alternative implementation
• extensible: additional functions can be easily added to the templates, and due to its modularity, the parts can be modified or even replaced
• small code base, which is well documented and should be easy to read and understand (about 500 lines-of-code + 500 lines of docstrings and comments)

3.1   Substitution

The template can contain placeholders, which are replaced by the evaluated result of their contents. pyratemp has two different placeholders:

• @!EXPR!@ escaped substitution: special characters are escaped
• $!EXPR!$ unescaped/raw substitution

Normally, you should always use @!...!@, since this escapes special characters. Currently, the following formats are supported:

• HTML (default): & < > " ' are escaped to &amp;, &lt;, &gt;, &quot;, &#39.
• MAIL_HEADER: encode non-ASCII mail-headers
• LATEX: \ # $ % & { } _ ~ ^ are escaped.
• NONE: no characters are replaced

EXPR can be any expression (e.g. a variable-name, an arithmetic calculation, a function-/macro-call etc.), and is evaluated when the template is rendered. Whitespace after @!/$! and before !@/!$ is ignored.

Examples:

• hello @!name!@. + name="marvin"

  -> hello marvin.
• hello escaped: @!name!@, unescaped: $!name!$ + name="<>&'""

  -> hello escaped: &lt;&gt;&amp;&#39;&quot;, unescaped: <>&\'"
• formatted: @! "%8.5f" % value !@ + value=3.141592653

  -> formatted:  3.14159
• hello --@!name.upper().center(20)!@-- + name="world"

  -> hello --       WORLD        --
• calculate @!var*5+7!@ + var=7

  -> calculate 42

3.2   Comments

Comments can be used in a template, and they do not appear in the result. They are especially useful for (a) documenting the template, (b) temporarily disabling parts of the template or (c) suppressing whitespace.

• #!...!# single-line comment with start- and end-tag
• #!... single-line-comment until end-of-line, incl. newline

The second version also comments out the newline, and so can be used at the end of a line to remove that newline in the result (like a backslash in Python-strings).

Comments can contain anything, but comments may not appear inside substitutions or block-tags.

3.3   Blocks

The control-structures, macros etc. have a special syntax, which consists of a start-tag (which is named according to the block), optional additional tags, and an end-tag:

<!--(...)-->        #! start tag
   ..
   ..
<!--(...)-->        #! optional additional tags (e.g. for elif)
   ..
   ..
<!--(end)-->        #! end tag

All tags must stand on their own line, and no code (except a #!...-comment) is allowed after the tag.

All tags which belong to the same block must have the same indent! The contents of the block does not need to be indented, although it might improve the readability (e.g. in HTML) to indent the contents as well.

Nesting blocks is possible, but the tags of nested blocks must have a different indent than the tags of the enclosing blocks. [4]

There's also a single-line version of a block, which does not need to stand on its own line, but can be inserted anywhere in the template. But note that this version does not support nesting [5]:

...<!--(...)-->...<!--(...)-->...<!--(end)-->...

<!--(if EXPR)-->
...
<!--(elif EXPR)-->
...
<!--(else)-->
...
<!--(end)-->

...<!--(if EXPR)-->...<!--(elif EXPR)-->...<!--(else)-->...<!--(end)-->...

<!--(for VARS in EXPR)-->
...
<!--(else)-->
...
<!--(end)-->

...<!--(for VARS in EXPR)-->...<!--(else)-->...<!--(end)-->...

<!--(macro MACRONAME)-->
...
<!--(end)-->

...<!--(macro MACRONAME)-->...<!--(end)-->...

MACRONAME

MACRONAME(KEYWORD_ARGs)

<!--(macro header)-->
<html>
<head>
  <title>A simple example: @!title!@</title>
</head>
<body>
<!--(end)-->

<!--(macro myfunc)-->
<li><!--(if exists("link"))--><a href="@!link!@">@!default("item",link)!@</a><!--(else)-->@!item!@<!--(end)--></li>
<!--(end)-->


@!header!@
<ul>
@!myfunc(item="Macros are fun!")!@
@!myfunc(item="Simple is better", link="http://www.simple-is-better.org")!@
@!myfunc(link="http://www.wikipedia.org")!@
</ul>

<!--(raw)-->
...
<!--(end)-->

...<!--(raw)-->...<!--(end)-->...

<!--(include)-->
  FILENAME
<!--(end)-->

...<!--(include)-->FILENAME<!--(end)-->...

<!--(set_escape)-->
  FORMAT
<!--(end)-->

...<!--(set_escape)-->FORMAT<!--(end)-->...

3.4   Expressions

A template needs some kind of "programming-language" to access variables, calculate things, format data, check conditions etc. inside the template. So, you could invent and implement an own language therefore -- or you could use an already existing and well designed language: Python.

pyratemp uses embedded Python-expressions. An expression is everything which evaluates to a value, e.g. variables, arithmetics, comparisons, boolean expressions, function/method calls, list comprehensions etc. [7]. And such Python expressions can be directly used in the template -- this makes pyratemp very powerful. But since it would be a bad idea to directly embed unrestricted Python-code in a template, these expressions are restricted by a pseudo-sandbox.

Examples:

• numbers, strings, lists, tuples, dictionaries, ...: 12.34, "hello", [1,2,3], ...
• variable-access: var, mylist[i], mydict["key"], myclass.attr
• function/method call: myfunc(...), "foobar".upper()
• comparison: (i < 0  or  j > 1024)
• arithmetics: 1+2

For details, please read the Python-documentation about Python expressions.

Note that accessing undefined variables in the template is considered to be an error. I chose this behaviour in contrast to many other template-engines (which often ignore undefined variables), since ignoring undefined variables would silently hide some errors, which is a bad idea. For ignoring or using a default-value for undefined variables, see default() below.

The following Python-built-in values/functions are available by default in the template:

True
False
None

abs()
chr()
cmp()       [removed in 0.3.0]
divmod()
hash()
hex()
isinstance() [new in 0.3.1 / 0.2.4]
len()
max()
min()
oct()
ord()
pow()
range()
round()
sorted()
sum()
unichr()
zip()

bool()
bytes()     [new in 0.3.0]
complex()
dict()
enumerate()
float()
int()
list()
long()
reversed()
set()       [new in 0.3.1 / 0.2.4]
str()
tuple()
unicode()
xrange()    [removed in 0.3.0]

dir()       [new in 0.3.1 / 0.2.4]

Additionally, the functions exists(), default(), setvar() and escape() are defined as follows:

exists("varname"):

Test if a variable (or any other object) with the name varname exists (in the current locals-namespace). Note that the name of the variable has to be quoted, and that this only works for single variable names, e.g. exists("mylist"). If you want to test more complicated expressions, use the default()-function.

It's especially useful in if-conditions to check if some (optional) variable exists, and then to branch accordingly.

Example:

<!--(if exists("foo"))-->YES<!--(else)-->NO<!--(end)-->

default("expr", default=None):

default() tries to evaluate the expression expr. If the evaluation succeeds and the result is not None, its value is returned; otherwise, if the expression contains undefined variables/attributes, the default-value is returned instead. Note that expr has to be quoted.

Since it is considered an error when the template tries to evaluate an undefined variable, this can be used to use default-values for optional variables, e.g. Name: @!default("myvar", "No name given.")!@.

Examples:

hi @!default("optional","anyone")!@

@!default("5*var1+var2","missing variable")!@

<!--(if default("opt1+opt2",0) > 0)-->yes<!--(else)-->no<!--(end)-->

<!--(for i in default("optional_list",[]))-->@!i!@<!--(end)-->

setvar("name", "expr"):

Although there is often a more elegant way, sometimes it is useful or necessary to set variables in the template. setvar() can also be used to capture the output of e.g. an evaluated macro.

Note that variables, which are set inside of a macro, can not be accessed outside of the macro.

Example:

$!setvar("i", "i+1")!$

escape("string", format=HTML):

Escape special characters. This is the same function, pyratemp uses internally for @!...!@, with a configurable format.

Supported formats: "None", "HTML", "mail_header", "LaTeX"

Example:

$!escape(subject, "MAIL_HEADER")!$

Note that you can use any expression evaluating to the desired string/value for all parameters (varname, expr, default, name) above.

Please look into the pyratemp-docstrings for more examples of exists(), default() and setvar().

More/user-defined functions can be added to the template as "data" (see User-interface), or by extending the evaluator.

4.1   User-interface

The user-interface of the template-engine consists of a single class: pyratemp.Template. This loads a template, checks its syntax, parses it, and can render it with your data.

class Template(string|filename|parsetree, encoding="utf-8", data=None, escape=HTML):

Load (and parse) a template. The template can either be directly given as string, or loaded from a file, or an already parsed template can be used.

encoding sets the charset-encoding of the loaded template (default: UTF-8).

data can be a dictionary containing some data which should be filled into the template by default (=if the variables etc. are not given when calling/rendering the template).

escape defines which special-characters should be escaped in substitutions by default (currently supported: NONE, HTML, MAIL_HEADER and LATEX). The escaping can also be set directly in the template (see set_escape), and setting the escape-format in the template overrides the one set here.

Note that you have to use keyword-parameters here!

To render the template, simply call the template with your data as (keyword-)parameters. This returns the result in Unicode, and you should encode it depending on your needs. Of course, the same template can be rendered several times with different data.

Example:

>>> import pyratemp
>>> t = pyratemp.Template(filename="test.html", data={"number": 1}, escape=pyratemp.HTML)
>>> result1 = t(person="Monty")
>>> result2 = t(person="Adams", number=42)
>>> print result1.encode("utf-8")
>>> print result2.encode("ascii", 'xmlcharrefreplace')

Note that data (and the keyword-parameters when calling the template) can contain nearly anything: single variables, lists, other dictionaries, nested structures, functions and even classes. So be careful what you pass to the template. If you e.g. pass the Python-built-in-function open to the template, your template will be able to open (and write) arbitrary files!

In addition, version 0.3.1/0.2.4 added a new file tools.py, which eases the creation of html-files and mails.

Example:

>>> from pyratemp.tools import html, mail

>>> result = html(template="test.tmpl", data={"number": 1}, xmlreplace=True)
>>> print result

>>> html(template="test.tmpl", data={"number": 1}, xmlreplace=True, filename="result.html")

>>> mail(maildir="outbox/", template="mail.tmpl", data={"from": "me@example.org"})

4.2   Internal parts

As said before, pyratemp consists of several parts, which are independent:

• an escaping-function (escape())
• a template-loader (class LoaderString or class LoaderFile)
• a parser (class Parser)
• a pseudo-sandboxed evaluator (class EvalPseudoSandbox)
• a renderer (class Renderer)
• a basic template-structure, e.g. used for macros/subtemplates (class TemplateBase)

All parts could even be used on their own, or modified (e.g. by creating subclasses) or replaced by an other implementation. Modified parts can then be used by setting the parameters loader_class, parser_class, renderer_class, eval_class or escape_func of class Template, or by creating a modified template-user-interface-class.

Read the pyratemp-docstrings for details.

5.1   syntax errors

To check a pyratemp-template for syntax-errors, simply let pyratemp parse the template. You can do this e.g. with pyratemp_tool with -s and without any data:

$ pyratemp_tool.py -s TEMPLATEFILE(s)

If there are syntax-errors, pyratemp raises a TemplateSyntaxError, and pyratemp_tool displays a detailed error-message:

$ pyratemp_tool.py -s TEMPLATEFILE
file 'TEMPLATEFILE':
  TemplateSyntaxError: line ##, col ##: ...

5.2   fillout/render test

For a complete test, you have to render the template. Of course you can do this in your application, with real data, but probably it's easier to test it outside of the application with some "dummy data".

This again can be done by pyratemp_tool. Simply create a JSON-file (or a YAML-file) with your dummy data, and invoke pyratemp_tool. If the JSON-/YAML-file contains all necessary data, the rendered template will be written to stdout, e.g.:

$ pyratemp_tool.py -f example.json example.html
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
          "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
...

Otherwise, if some data is missing (or invalid) or some expressions are invalid, pyratemp raises a TemplateRenderError and tells you what data is missing, e.g.:

$ pyratemp_tool.py -f dummydata.json example.html
file 'example.html':
  TemplateRenderError: Cannot eval expression 'title' (NameError: name 'title' is not defined)

5.3   whitespace handling

If you need to remove some whitespace (e.g. a newline), you can put it into comments, e.g. a #! at the end of a line removes the newline in the result.

5.4   setting variables

There's sometimes the wish to set variables in the template. But there's often a more elegant way to solve the problem without setting variables.

For example, instead of:

<!--(for obj in arr)-->
    @!i!@. @!obj!@
    $!setvar("i", "i+1")!$ #!
<!--(end)-->

better use:

<!--(for i,obj in enumerate(arr))-->
    @!i!@. @!obj!@
<!--(end)-->

But if you really need to set variables, you can use setvar() (see Expressions).

5.5   capture output

To capture the output of a macro into a variable, you can also use setvar():

<!--(macro mymacro)-->
...
<!--(end)-->
$!setvar("myvar", "mymacro()")!$#!

5.6   filters

Some template-engines have so-called "filters", which are functions which process the contents of a whole block. This behaviour can be easily achieved with pyratemp, too:

<!--(macro myblock)-->
...
<!--(end)-->
$!filter(myblock())!$

5.7   select macro / overwrite macro

Macros can be overwritten, even based on a conditional. This may be useful if you want to globally change the output depending on some options.

Example:

<!--(macro field)-->show_field $!param!$<!--(end)-->
<!--(macro input_field)-->input_field $!param!$<!--(end)-->

<!--(if x=="overwrite")-->
  <!--(macro field)-->
@!input_field(param=param)!@
  <!--(end)-->
<!--(end)-->

@!field(p=1)!@

5.8   evaluation order / caches

The expressions, macros etc. are evaluated at rendering-time, not at define-/parse-time.

So, e.g. a macro is evaluated each time it is called/used. If you want something to be evaluated only once, no matter how often it is used, you may:

• store the evaluation-result in a variable with setvar() and then use the variable
• write a cache-function in Python and use it in your template

5.9   using functions which import other modules

Some Python-functions, which may be useful in the template, try to import other modules, e.g. datetime.strftime imports time. But importing modules is of course not allowed in the (pseudo-)sandbox, so this fails.

The best solution would be to avoid these functions and e.g. use time.strftime instead of datetime.strftime.

5.10   shortcuts / template-syntax-extension

Sometimes shortcuts for often-used things might be useful. Therefore, you can:

• Define macros.

  This is probably the most common way.
• Define a Python-function and pass it to the template.

  (But take care that you don't break the model-view-separation!)
• Extend the template-syntax. [advanced]

  This essentially modifies the template-engine, and you should only do this if you are really sure that you want it, and that the alternatives do not work in your case!

  The simplest way to introduce new syntax probably is to load the template-file, map your new syntax (e.g. with regexps) to normal pyratemp-syntax, and let pyratemp do the rest.

  Example: The new syntax @x!...!x@ should be added, which should do the same as @!myfunc(...)!@:

  import re
  import pyratemp
  
  class MyLoader(pyratemp.LoaderFile):
      my_replacement = re.compile(r'@x!\s*(.*?)\s*!x@')
  
      load(self, filename):
          u = pyratemp.LoaderFile(self, filename)
          u = my_replacement.sub(r'@! myfunc("\1") !@', u)
          return u
  
  t = pyratemp.Template(..., loader_class=MyLoader)
  

5.11   return data from the template to the caller

In some rare cases, it might be useful to return data (e.g. a returncode) from the template to the caller. This is possible with pyratemp. But note that this can easily break the model-view-separation, so think twice about it before using it, and use it with care!

Example:

import pyratemp

retdict = {}
def setreturn(name, value):
    """Quick-hack to export data from the template to the code.
    """
    retdict[name] = value
    return ""

t = pyratemp.Template("""Return myreturn=True to the caller: @!setreturn("myreturn", True)!@""")
t(setreturn=setreturn)
print("retdict: %s" % retdict)

6.1   pyratemp_tool

pyratemp_tool.py is a simple command-line-interface to pyratemp which can (a) syntax-check the templates and (b) fill/render templates with the data from JSON- or YAML-files or from key-value-pairs from the command-line. Errors/messages are print to stderr, rendered templates are written to stdout in UTF-8.

By default, HTML-escaping is used for *.html and *.htm, LaTeX-escaping is used for *.tex and no escaping is used for other files. Use set_escape if you need a different encoding.

Two additional variables are defined: date and mtime_CCYYMMDD, both containing the current date in the "YYYY-MM-DD"-format.

Usage (see also pyratemp_tool.py --help and pydoc pyratemp_tool):

pyratemp_tool.py [-s] <-d NAME=VALUE> <-f DATAFILE [-N NAME] [-n NR_OF_ENTRY]> [--xml] TEMPLATEFILES
    -s      syntax-check only (don't render the template)
    -d      define variables (these also override the values from files)
    -f      use variables from a JSON/YAML file
    -n      use nth entry of the JSON/YAML file
            (JSON: n-th element of the root-array, YAML: n-th entry)
    -N      namespace for variables from the JSON/YAML file
    --xml   encode output as ASCII+xmlcharrefreplace (instead of utf-8)

For the 2nd example of Quickstart, a JSON-file might look like:

{
  "title" : "filling JSON into pyratemp",
  "special_chars" : "µ<߀",
  "number" : 13,
  "mylist" : [ "JSON", "YAML", "manually-defined variables" ]
}

To fill the template, with using a different value for number than in the JSON-file, invoke pyratemp_tool as follows:

$ pyratemp_tool.py -d number=42 -f example.json example.html > filled.html

Now, the result is in filled.html