danhanson/tree-sitter-django
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5183f6aa5287df830af43da9ed2be7c14e73b550
tree-sitter-django/README.md
T
danhanson 5183f6aa52 proofread readme
2026-09-16 14:59:07 -05:00

338 lines
21 KiB
Markdown
Raw Blame History

# tree-sitter-django
A [tree-sitter](https://tree-sitter.github.io/tree-sitter/) grammar for the [Django Template Language (DTL)](https://docs.djangoproject.com/en/6.1/ref/templates/language/#the-django-template-language) — the `{% %}` tags, `{{ }}` variables and `{# #}` comments used in Django templates.
## Coverage
Every tag and filter Django ships as a builtin is modelled — 28 tags and 57 filters as of Django 6.1 —
and so is every tag and filter of the five libraries below. Arguments are part of the grammar rather than
something a linter checks later, so a builtin written with the wrong ones is a parse error: `{{ x|length:2 }}`
and `{% now %}` do not parse, because `length` takes no argument and `now` requires one.
The target is Django 6.1, including the builtins Django 6 added (`{% partial %}`, `{% partialdef %}`,
`{% querystring %}`, `{% csp_nonce_attr %}`). Rather than being written from the documentation, the grammar
is checked against a real Django — `Pipfile.lock` pins the version — by enumerating `Engine.default_builtins`
for the tag and filter lists and their arities, and by running candidate snippets through both
`django.template.Template()` and this parser to compare what each accepts. Where the two deliberately
disagree is listed under [Divergences](#divergences-from-django).
## Using it
There is no published package yet, but the language bindings are in place. What is here today:
- **`src/parser.c` and `src/scanner.c` are generated and committed**, so any tree-sitter host can compile
them directly without running the CLI. The scanner is required: block-name matching
(`{% block a %}…{% endblock a %}`), the repeated-argument guards and the missing-tag markers live in it.
- **`npm run parser-build`** produces `tree-sitter-django.wasm` for a `web-tree-sitter` host or an editor
that loads Wasm grammars. It is a build artifact rather than a checked-in one, so it has to be built
(or shipped by a release) before anything can load it.
- **`queries/`** holds the highlight, locals, tags, injection and error queries an editor loads, plus three files
meant for a linter. See [Queries](#queries).
- **`tree-sitter.json`** names the grammar `django`, scopes it `source.django`, and points at the query
files, which is what a tree-sitter host reads to wire all of the above together.
- **`bindings/`** holds a binding per language, generated by `tree-sitter init` from the `bindings` and
`metadata` sections of `tree-sitter.json`. See [Bindings](#bindings).
## Bindings
| Language | Build from a checkout | Entry point |
| -------- | -------------------------------------------------------- | ------------------------------------------------------ |
| Node | `npm install` builds the addon through `node-gyp-build` | `bindings/node/index.js` (ESM), typed by `index.d.ts` |
| Rust | `cargo build` | `bindings/rust/lib.rs`, `Cargo.toml` |
| Python | `pip install .` | `bindings/python/tree_sitter_django`, `pyproject.toml` |
| Go | `go test ./bindings/go` | `bindings/go/binding.go`, `go.mod` |
| Swift | `swift build` | `bindings/swift/TreeSitterDjango`, `Package.swift` |
| C / C++ | `make` (static, shared and a pkg-config file) or `cmake` | `bindings/c/tree_sitter/tree-sitter-django.h` |
```js
import Parser from "tree-sitter";
import django from "tree-sitter-django";
const parser = new Parser();
parser.setLanguage(django);
```
```python
import tree_sitter, tree_sitter_django
language = tree_sitter.Language(tree_sitter_django.language())
tree = tree_sitter.Parser(language).parse(b"{% if x %}hi {{ y|upper }}{% endif %}")
```
The binding files are generated, so they are not edited by hand: change `tree-sitter.json` and re-run
`tree-sitter init`. `package.json` still carries `"private": true`, so nothing is published from this
repository yet.
## Development
```sh
npm install
npm run parser-generate # regenerate the parser from grammar.js
npm run parser-test # run the corpus tests in test/corpus
npm run parser-build # build the WASM binary
npm run playground # interactively explore the grammar
```
`src/parser.c`, `src/grammar.json` and `src/node-types.json` are generated **and committed**, so a change
to `grammar.js` or `src/scanner.c` is only half made until `parser-generate` has run and the result is
committed alongside it. `tree-sitter test` compiles the parser itself, so a stale `src/` shows up as tests
that pass against the wrong grammar.
## Queries
Five of the files in `queries/` are query files an editor loads:
- **`highlights.scm`** — syntax highlighting. Tag and filter names are listed one by one, so a name the
grammar knows is captured as `@function` and a name it is only tolerating as `@function.call`.
- **`locals.scm`** — scopes, definitions and references, so an editor can resolve a variable to the tag
that bound it. A definition is a `variable:` field; the scopes are the blocks whose bindings do not
outlive them (`{% with %}`, `{% for %}`, `{% empty %}`, `{% block %}`, `{% partialdef %}`,
`{% blocktranslate %}`).
- **`tags.scm`** — the symbol index behind `tree-sitter tag`. A `{% partialdef %}` and the names a
`{% load %}` brings in are definitions; every tag and filter used is a reference.
- **`injections.scm`** — marks template text and `{% verbatim %}` bodies as another language's content, so
the HTML around the tags can be parsed by its own grammar. It names no language, leaving the editor to
supply one.
- **`errors.scm`** — what to report as a problem: `(ERROR)` as `@error.syntax`, tree-sitter's own
`(MISSING)` as `@error.missing`, and `@error.missing_tag` for the marker placed where a group's required
tag is missing. An unterminated `{% if a %}x` parses as an `if_group` ending in a zero-width
`missing_endif_tag` rather than as an `ERROR`, so completion and the scope queries keep working while
a template is being written — and the tree reports no error of its own, so this query is how to find
it. A marker is placed only where the scanner can tell: at the end of input, before an end tag that
belongs to an enclosing group, or before an `{% endblock %}` naming a block further out.
A group's tag written where no open group can hold it, such as `{% else %}` inside a `{% for %}`, is an
`unexpected_tag`, captured as `@error.unexpected_tag`. Words a tag does not take, such as the `x` in
`{% else x %}`, are an `unexpected_argument` inside that tag, captured as `@error.unexpected_argument`. A `{{` whose `}}` is missing closes with a zero-width
`missing_variable_close`, captured as `@error.missing_variable_close`, so the rest of the template still parses.
The other three are data for a tool rather than queries an editor runs. Each captures the two halves of a
relation a tree-sitter query cannot express, and the tool does the join:
- **`libraries.scm`** — which library each tag or filter is registered in, and what each `{% load %}`
brings in. The grammar never requires the load, so reporting a missing one is a linter's job; a query
has no ordering predicate to compare the positions itself.
- **`exports.scm`** — a name a tag binds in the scope _around_ it rather than the scope it opens. Only
`{% blocktranslate %}`'s `asvar` does that, and a locals query cannot say so: it places a definition in
the innermost scope containing it and offers no way out again.
- **`conditionals.scm`** — which bodies render on only some passes, as `@conditional.group` /
`@conditional.branch` / `@conditional.default`, so a checker can tell that
`{% if a %}{% now "Y" as n %}{% endif %}{{ n }}` may reach `{{ n }}` with nothing bound. A group is
exhaustive exactly when one of its branches is a default. A locals query has no notion of a path not
taken.
`tree-sitter test` loads every file in the directory, so a pattern naming a node that no longer exists
fails the suite rather than quietly matching nothing.
## Django libraries
Five of Django's template libraries are not builtins: their names have to be `{% load %}`ed, or preloaded
through the engine's `OPTIONS: {"builtins": [...]}`. Every tag and filter all five register is modelled,
with the same argument and arity rules as a builtin:
| library | tags | filters |
| -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| `i18n` | `trans`, `translate`, `blocktrans`, `blocktranslate`, `language`, `get_available_languages`, `get_current_language`, `get_current_language_bidi`, `get_language_info`, `get_language_info_list` | `language_bidi`, `language_name`, `language_name_local`, `language_name_translated` |
| `l10n` | `localize` | `localize`, `unlocalize` |
| `static` | `static`, `get_static_prefix`, `get_media_prefix` | — |
| `cache` | `cache` | — |
| `tz` | `localtime`, `timezone`, `get_current_timezone` | `localtime`, `timezone`, `utc` |
**The `{% load %}` is never required**, because an engine may have preloaded the library. Whether one is
present is a question about the order of nodes rather than about how the template parses, so it is left to
a tool: `queries/libraries.scm` marks each name with the library it came from and each `{% load %}` with
what it brings in, and the tool joins the two. Such a tool needs to know that loading is positional
(`{% trans "x" %}{% load i18n %}` is an error), that it is per file rather than inherited through
`{% extends %}` or `{% include %}`, and that `{% load trans from i18n %}` brings in only the names it lists.
Because these names are modelled, they win over the fallbacks below: a project that registers its own tag
under one of them with a different signature gets a parse error. They are common enough that shadowing
them is treated as the mistake.
If a project insists on shadowing one of these tags, it may use a sub-grammar that replaces the rule with
their own. See [Adding rules for your own tags and filters](###adding-rules-for-your-own-tags-and-filters).
## Custom tags and filters
A library's tags and filters are registered in Python, so the grammar cannot know their names, their
arity, or whether a tag opens a body. Two fallback rules accept them:
- **Tags.** Any tag the grammar does not recognise parses as `custom_tag`, with the shape `simple_tag` and
`inclusion_tag` give: positional filter expressions, then `name=value` keyword arguments, then an
optional `as name`. So `{% mytag a b|upper "s" k=1 as out %}` parses, with the name on the `tag_name:` field.
A keyword argument written twice is refused.
- **Filters.** An unknown filter parses as `(filter filter_name: (identifier))` with at most one `:argument`. No arity check for unknown filters is possible.
Both are captured as `@function.call` rather than `@function`, so an editor can distinguish a name the
grammar knows from one it is only tolerating.
What the fallbacks cannot do:
- **A block tag is not nested.** `{% mytag %}…{% endmytag %}` is two sibling `custom_tag`s; pairing them
would mean guessing that an unknown tag opens a body.
- **A tag registered with `@register.tag` may parse anything**, since it receives the raw token, so one
whose arguments are not the `simple_tag` shape — `{% mytag <<>> %}` — is an error here and not in Django.
- **A misspelled name cannot be caught.** `{{ x|lenght }}` is indistinguishable from a filter some library
registered. Cross-referencing names against `{% load %}` needs the project's Python, so it belongs in a
linter; `queries/libraries.scm` and the `load_tag` rule supply what one needs.
- **Names that only exist inside a tag group are reserved**, so `{% endif %}` or `{% empty %}` standing on
its own is an `unexpected_tag`, reported by `errors.scm`, rather than a call to a custom tag of that name.
### Adding rules for your own tags and filters
For a project whose custom tags deserve real rules rather than the fallbacks, use this grammar as a base
grammar. `grammar(base, {...})` merges your rules into it, and a rule that redefines one the base already
has receives it as `previous`. A tag is an alternative of `template_tag`; a filter is an
alternative of `filter`.
```js
import django from "tree-sitter-django/grammar";
export default grammar(django, {
name: "djangox",
conflicts: ($, previous) => [...previous, [$.loop_block]],
rules: {
template_tag: ($, previous) => choice(previous, $.map_tag, $.loop_group),
// {% map items over rows %}
map_tag: ($) =>
seq(
$.simpleTagOpen,
field("tag_name", "map"),
$._sep,
$.filtered_value,
$._sep,
"over",
$._sep,
$.filtered_value,
$.simpleTagClose,
),
// {% loop x %}…{% endloop %}
loop_tag: ($) =>
seq(
$.startTagOpen,
field("tag_name", "loop"),
$._sep,
$.identifier,
$.startTagClose,
),
loop_block: ($) => seq($.loop_tag, optional($.template)),
endloop_tag: ($) =>
seq($.endTagOpen, field("tag_name", "endloop"), $.endTagClose),
loop_group: ($) =>
seq(
$.loop_block,
choice($.endloop_tag, alias($._missing_tag, $.missing_endloop_tag)),
),
// {{ value|shout:"!" }}
filter: ($, previous) =>
choice(
previous,
seq(field("filter_name", "shout"), ":", field("argument", $.value)),
),
},
});
```
Seven things to know:
- **Whitespace is explicit.** `extras` is empty, because Django splits a tag's contents before parsing any
argument, so every separator has to be written out. The `part`/`tag`/`simpleTag` helpers the base
grammar is built from are private to it, but the separator is the hidden rule `$._sep`, and it and the
tag utilities (`simpleTagOpen` and the rest) are rules, so an extending grammar uses them as they are.
- **Every tag begins with one of the opens**: `$.simpleTagOpen`, with `$.simpleTagClose` at its end, for a
tag outside any group such as `map_tag`, and `startTagOpen`, `followTagOpen`, `repeatTagOpen` or
`endTagOpen` for the tags of a group. Its
zero-width token hands the tag's name to the scanner and is taken wherever a tag can start, so a tag
written from a bare `"{%"` never matches: `{% map items over rows %}` would quietly parse as a
`custom_tag` instead.
- **A new name wins over the fallback.** `shout` becomes a token of its own, so it commits to your
alternative and your arity applies — `{{ x|shout }}` is now an error, while names you have not modelled
still fall through to `(filter filter_name: (identifier))`.
- **A tag with a body is split the way the base grammar's are**, so every `{% %}` is a node of its own:
a `loop_tag` ending in `$.startTagClose`, a `loop_block` of that tag and the body, an
`endloop_tag` built from `$.endTagOpen` and `$.endTagClose`, and a `loop_group` of the block and the
end tag. A middle tag uses `followTagOpen`/`followTagClose` if the group holds it at most once, and
`repeatTagOpen`/`repeatTagClose` if it may repeat; the scanner trusts that choice. The block needs a
`conflicts` entry, as every block in the base grammar does. The generator offers an associativity
instead; taking it silently discards the parse in which the body continues.
- **End tags are not reserved**, a stray `{% endmap %}` parses as a `custom_tag` rather than an error — `reserved` takes no
`previous`, so the `tag_name` list cannot be added to a name at a time. Instead you may drop the `custom_tag` rule to avoid parsing end tags as custom tags. While the fallback
is there, stray words in one of your end tags (`{% endloop extra %}`) read as a `custom_tag` too, which
leaves your group with a missing-tag marker at the end of input instead of an `unexpected_argument`.
- **Your groups get missing-tag markers.** The scanner knows no group by name — a group is closed by `end`
followed by its opening tag's name — so the `alias($._missing_tag, $.missing_endloop_tag)` choice
above is all it takes: an unclosed `{% loop x %}`, or one left open inside an `{% if %}` that
`{% endif %}` closes, holds a `missing_endloop_tag`. Add the name to your copy of `errors.scm`.
- **The external scanner has to be re-exported under your grammar's name**, or block-name matching and the
repeated-argument guards will not link. Give your grammar a `src/scanner.c` that renames the base's five entry points to the
ones your generated `parser.c` calls, and includes it:
```c
#define tree_sitter_django_external_scanner_create tree_sitter_djangox_external_scanner_create
#define tree_sitter_django_external_scanner_destroy tree_sitter_djangox_external_scanner_destroy
#define tree_sitter_django_external_scanner_scan tree_sitter_djangox_external_scanner_scan
#define tree_sitter_django_external_scanner_serialize tree_sitter_djangox_external_scanner_serialize
#define tree_sitter_django_external_scanner_deserialize tree_sitter_djangox_external_scanner_deserialize
#include "../node_modules/tree-sitter-django/src/scanner.c"
```
The base's `tree_sitter/parser.h` then resolves to your own generated copy, and the externals keep the
order they have here, so the scanner's token enum stays valid. `tree-sitter build` takes no include
path, hence the relative `#include`; the package also exports `tree-sitter-django/src/scanner.c`, so a
build step can hand the compiler an absolute path from `import.meta.resolve` where a package manager's
layout makes the relative one unreliable.
**Turning the fallbacks off.** A grammar that models every tag and filter its project uses probably does
not want `custom_tag` and `_custom_filter` accepting unrecognized names as they allow typos to parse. `previous` is the base rule whose `members` are filterable. We can filter out these rules to make the grammar more strict:
```js
template_tag: ($, previous) =>
choice(
...previous.members.filter((m) => m.name !== "custom_tag"),
$.map_tag,
),
filter: ($, previous) =>
choice(
...previous.members.filter((m) => m.name !== "_custom_filter"),
seq(field("name", "shout"), ":", field("argument", $.value)),
),
```
`{% mytag %}` and `{{ x|unknown }}` are then errors, while the builtins, the libraries and your own rules
carry on as before.
## Divergences from Django
The grammar aims to accept what Django accepts, with one deliberate exception: where Django takes an
input without complaint but the input is almost certainly a mistake. It reads the mistake and silently
renders nothing or quietly ignores half of what was written. This grammar tries to report such mistakes
as errors instead. Every case below is input Django itself accepts:
- **Number-shaped names.** Django has no number token: a word is a literal if `int()` or `float()` parses
it and a variable lookup otherwise, so `0x1f`, `1a`, `1.`, `1.2.3` and `1__0` look up a variable of that
name and render nothing. (`1.` really is a lookup there and not a float — `Variable` rejects a trailing
dot after parsing it — while `1.e5` is a float, and both are treated that way here.)
- **An attribute of `True`, `False` or `None`.** Every Django context is seeded with those three names, so
`{{ True.x }}` is an ordinary two-part lookup that can never resolve.
- **A keyword argument written twice.** `{% with a=1 a=2 %}`, and the same in `{% include %}` and
`{% blocktranslate %}`, where Django builds a dict and lets the last value win.
- **Anything after a `{% static %}` path.** `StaticNode.handle_token` looks for `as` two words from the end
and ignores the rest, so `{% static "a" junk %}` and `{% static "a" as url junk %}` render as if the
extra words were not there.
- **A `{% cache %}` argument meant as something other than the fragment name.** Django takes the raw word
in that position, so `{% cache 500 using="c" %}` names a fragment called `using="c"` rather than choosing
a cache, and `{% cache 500 sidebar.name %}` names one with a dot in it.
In the other direction the grammar is more permissive only where it cannot know better, which is what the
fallbacks above are for.
## License
MIT — see [LICENSE](LICENSE).
Reference in new issue View Git Blame Copy Permalink