tree-sitter-django

A tree-sitter grammar for the Django Template Language (DTL) — the {% %} tags and {{ }} variables used in Django templates.

Development

npm install
npm run parser-generate  # regenerate the parser from grammar.js
npm run parser-test      # run the corpus tests in test/corpus
npm run parser-build     # build the WASM binary
npm run playground       # interactively explore the grammar

Django libraries

Five of Django's template libraries are not builtins: their names have to be {% load %}ed, or preloaded through the engine's OPTIONS: {"builtins": [...]}. Every tag and filter all five register is modelled, with the same argument and arity rules as a builtin:

library	tags	filters
`i18n`	`trans`, `translate`, `blocktrans`, `blocktranslate`, `language`, `get_available_languages`, `get_current_language`, `get_current_language_bidi`, `get_language_info`, `get_language_info_list`	`language_bidi`, `language_name`, `language_name_local`, `language_name_translated`
`l10n`	`localize`	`localize`, `unlocalize`
`static`	`static`, `get_static_prefix`, `get_media_prefix`	—
`cache`	`cache`	—
`tz`	`localtime`, `timezone`, `get_current_timezone`	`localtime`, `timezone`, `utc`

The {% load %} is never required, because an engine may have preloaded the library. Whether one is present is a question about the order of nodes rather than about how the template parses, so it is left to a tool: queries/libraries.scm marks each name with the library it came from and each {% load %} with what it brings in, and the tool joins the two. Such a tool needs to know that loading is positional ({% trans "x" %}{% load i18n %} is an error), that it is per file rather than inherited through {% extends %} or {% include %}, and that {% load trans from i18n %} brings in only the names it lists.

Because these names are modelled, they win over the fallbacks below: a project that registers its own tag under one of them with a different signature gets a parse error. They are common enough that shadowing them is treated as the mistake.

Custom tags and filters

A library's tags and filters are registered in Python, so the grammar cannot know their names, their arity, or whether a tag opens a body. Two fallback rules accept them:

Tags. Any tag the grammar does not recognise parses as custom_tag, with the shape simple_tag and inclusion_tag give: positional filter expressions, then name=value keyword arguments, then an optional as name. So {% mytag a b|upper "s" k=1 as out %} parses, with the name on the tag: field. A keyword argument written twice is refused, as parse_bits refuses it.
Filters. An unknown filter parses as (filter name: (identifier)) with at most one :argument, which is all Django's expression regex allows. No arity check is possible.

Both are captured as @function.call rather than @function, so an editor can distinguish a name the grammar knows from one it is only tolerating.

What the fallbacks cannot do:

A block tag is not nested. {% mytag %}…{% endmytag %} is two sibling custom_tags; pairing them would mean guessing that an unknown tag opens a body.
A tag registered with @register.tag may parse anything, since it receives the raw token, so one whose arguments are not the simple_tag shape — {% mytag <<>> %} — is an error here and not in Django.
A misspelled name cannot be caught. {{ x|lenght }} is indistinguishable from a filter some library registered. Cross-referencing names against {% load %} needs the project's Python, so it belongs in a linter; queries/libraries.scm and the load rule supply what one needs.
Names that only exist inside a tag group are reserved, so {% endif %} or {% empty %} standing on its own is an error rather than a call to a custom tag of that name.

Adding rules for your own tags and filters

For a project whose custom tags deserve real rules rather than the fallbacks, use this grammar as a base grammar. grammar(base, {...}) merges your rules into it, and a rule that redefines one the base already has receives it as previous. A tag is an alternative of template_block_groups; a filter is an alternative of filter.

import django from "tree-sitter-django/grammar";

const SEP = /[ \t\r\n]+/;

export default grammar(django, {
  name: "djangox",

  rules: {
    // {% map items over rows %}
    template_block_groups: ($, previous) => choice(previous, $.map_tag),
    map_tag: ($) =>
      seq(
        "{%",
        optional(SEP),
        "map",
        SEP,
        $.filtered_value,
        SEP,
        "over",
        SEP,
        $.filtered_value,
        optional(SEP),
        "%}",
      ),

    // {{ value|shout:"!" }}
    filter: ($, previous) =>
      choice(
        previous,
        seq(field("name", "shout"), ":", field("argument", $.value)),
      ),
  },
});

Four things to know:

Whitespace is explicit. extras is empty, because Django splits a tag's contents before parsing any argument, so every separator has to be written out. The part/block/simpleTag helpers the base grammar is built from are private to it, hence the SEP above.
A new name wins over the fallback. shout becomes a token of its own, so it commits to your alternative and your arity applies — {{ x|shout }} is now an error, while names you have not modelled still fall through to (filter name: (identifier)).
A tag with a body needs a conflicts entry, as every clause in the base grammar does: conflicts: ($, previous) => [...previous, [$.map_clause]]. The generator offers an associativity instead; taking it silently discards the parse in which the body continues. Its end tag is not reserved either, so a stray {% endmap %} parses as a custom_tag rather than an error — reserved takes no previous, so the tag_name list cannot be added to a name at a time.
The external scanner has to be re-exported under your grammar's name, or block-name matching and the repeated-argument guards will not link. exports cannot deliver it — that is Node's resolver, and the scanner is C — so give your grammar a src/scanner.c that renames the base's five entry points to the ones your generated parser.c calls, and includes it:

#define tree_sitter_django_external_scanner_create tree_sitter_djangox_external_scanner_create
#define tree_sitter_django_external_scanner_destroy tree_sitter_djangox_external_scanner_destroy
#define tree_sitter_django_external_scanner_scan tree_sitter_djangox_external_scanner_scan
#define tree_sitter_django_external_scanner_serialize tree_sitter_djangox_external_scanner_serialize
#define tree_sitter_django_external_scanner_deserialize tree_sitter_djangox_external_scanner_deserialize

#include "../node_modules/tree-sitter-django/src/scanner.c"

The base's tree_sitter/parser.h then resolves to your own generated copy, and the externals keep the order they have here, so the scanner's token enum stays valid. tree-sitter build takes no include path, hence the relative #include; the package also exports tree-sitter-django/src/scanner.c, so a build step can hand the compiler an absolute path from import.meta.resolve where a package manager's layout makes the relative one unreliable.

Divergences from Django

The grammar aims to accept what Django accepts, with one deliberate exception. Where Django takes an input without complaint but the input is almost certainly a mistake — it parses, and then silently renders nothing or quietly ignores half of what was written — the grammar reports an error instead. Every case below is input Django itself accepts:

Number-shaped names. Django has no number token: a word is a literal if int() or float() parses it and a variable lookup otherwise, so 0x1f, 1a, 1., 1.2.3 and 1__0 look up a variable of that name and render nothing.
An attribute of True, False or None. Every Django context is seeded with those three names, so {{ True.x }} is an ordinary two-part lookup that can never resolve.
A keyword argument written twice. {% with a=1 a=2 %}, and the same in {% include %} and {% blocktranslate %}, where Django builds a dict and lets the last value win.
Anything after a {% static %} path. StaticNode.handle_token looks for as two words from the end and ignores the rest, so {% static "a" junk %} and {% static "a" as url junk %} render as if the extra words were not there.
A {% cache %} argument meant as something other than the fragment name. Django takes the raw word in that position, so {% cache 500 using="c" %} names a fragment called using="c" rather than choosing a cache, and {% cache 500 sidebar.name %} names one with a dot in it.

In the other direction the grammar is more permissive only where it cannot know better, which is what the fallbacks above are for.

10 KiB Raw Blame History