392 lines
29 KiB
Markdown
392 lines
29 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
A tree-sitter grammar for the Django Template Language: `{% %}` tags and `{{ }}` variables.
|
|
|
|
## Commands
|
|
|
|
```sh
|
|
npm run parser-generate # regenerate src/parser.c, grammar.json, node-types.json from grammar.js
|
|
npm run parser-test # run the corpus tests
|
|
npm run parser-build # build tree-sitter-django.wasm
|
|
npm run playground # explore the grammar interactively
|
|
|
|
npx tree-sitter test -i "Cycle With As" # one test, by name regex
|
|
npx tree-sitter test --file-name cycle.txt # one corpus file
|
|
npx tree-sitter test -r # force a rebuild of the parser library
|
|
npx tree-sitter parse file.html # parse one file
|
|
npx tree-sitter parse --debug normal f.html # lex/shift/reduce trace
|
|
npx prettier --write grammar.js queries test CLAUDE.md # not `.` — see below
|
|
```
|
|
|
|
`src/parser.c`, `src/grammar.json` and `src/node-types.json` are generated **and committed**, so run
|
|
`parser-generate` after every change to `grammar.js` or `src/scanner.c`.
|
|
|
|
Format with prettier over specific paths rather than `npm run format` (`prettier --write .`): that also
|
|
rewrites the generated `src/grammar.json` and `src/node-types.json`, which `parser-generate` then writes
|
|
back, producing churn in both directions.
|
|
|
|
`npm run lint` does not work: `eslint.config.mjs` was copied from a Next.js project and enables
|
|
`@typescript-eslint` and `react-hooks` rules whose plugins aren't installed. Formatting is prettier's job.
|
|
|
|
### Stale parser library
|
|
|
|
tree-sitter caches the compiled parser in `~/.cache/tree-sitter/lib/django.*`, and it has served a stale
|
|
build after edits — producing parses that contradict the current grammar (debug traces naming tokens that
|
|
no longer exist). **If a parse result disagrees with `grammar.js`, suspect this first**: rerun with
|
|
`npx tree-sitter test -r`, or delete `~/.cache/tree-sitter/lib/django*`.
|
|
|
|
## Architecture
|
|
|
|
### Whitespace is syntax, not `extras`
|
|
|
|
`extras: []`. Django's `Token.split_contents()` splits a tag's contents on whitespace _before_ parsing any
|
|
argument, so `{% cycle 1as x %}` is the single token `1as` — an error, not `1` followed by `as`. `extras`
|
|
can never be made mandatory, so the separator is an explicit token and every rule is built from helpers
|
|
that place it:
|
|
|
|
- `SEP` — `/[ \t\r\n]+/`. Newlines included: Django matches tags with `re.DOTALL`, so tags may span lines.
|
|
- `part(...)` — one tag part, taking the whitespace that separates it from the part before.
|
|
- `joined(first, ...rest)` — parts separated from _each other_, with no leading separator (`not in`).
|
|
- `block(tag, ...args)` — `{%`, optional separator, tag name, args, optional separator, `%}`; tags may
|
|
hug their delimiters (`{%cycle 1%}`).
|
|
- `asVariable($)` — the `as name` clause. Every tag that takes one names the target `field("variable", …)`;
|
|
do not invent a second name for it.
|
|
- `simpleTag($, tag, args, kwargs)` — a tag registered with Django's `simple_tag`, which always accepts
|
|
positional filter expressions, then `name=value` keywords, then `as name`. `args`: `true` (`*args`), a
|
|
count (at most N), or `false`. `kwargs`: `true` (`**kwargs`), `false`, or `{name: rule}` for a known
|
|
signature. Put the Python signature in a comment above the call site.
|
|
- `arrangements(items)` — every ordered subset of `items`, so that a tag's options may be written in any
|
|
order but never twice. Used by `translate_block` and `include_block`; keyword arguments are guarded by the scanner
|
|
instead. Grows as `Σ C(n,k)·k!` (65 for four names), which is
|
|
why it is only used for signatures that name their arguments.
|
|
- `wordAsValueRules(word)` / `wordAsValue($, word, continued)` — a tag's own keyword written where a value
|
|
goes. Django tells its flag from a variable by comparing the whole bit, so `reversed` is the flag of
|
|
`{% for %}` while `reversed.x` and `reversed|last` are lookups; `continued` selects that second group.
|
|
Needed because keyword extraction hands the parser the keyword wherever both it and `identifier` are
|
|
valid, so the reading in which the word names a variable has to be built back out of it. The rules are
|
|
hidden and aliased at the use site, one per level (`variable_attribute`, `value`, `filtered_value`),
|
|
because an alias renames the node its own rule makes and **cannot nest one inside another** — writing
|
|
it as nested inline `alias(alias(...))` calls silently yields a flat, partly zero-width tree that still
|
|
generates and still parses. Check the tree, not just the accept.
|
|
|
|
Consequences worth knowing before editing a rule:
|
|
|
|
- **No `token.immediate` anywhere** — without extras, adjacency is already the default. Use it only to
|
|
forbid whitespace somewhere a separator is otherwise allowed.
|
|
- **Every optional trailing part needs somewhere for its separator to go.** Getting this wrong does not
|
|
fail at generate time; it shows up as `MISSING` nodes in a passing-looking parse.
|
|
- **`conflicts` entries are the normal fix, not precedence.** With explicit separators the lookahead at a
|
|
part boundary is `SEP` instead of the meaningful token, so LR(1) loses its discriminator. That is why
|
|
`predicate`, `binaryOperator` and `[$.library, $.load]` are listed there. Where two readings both stay
|
|
valid to the end — `is not` as one operator or as `is` and a prefix `not` — the conflict needs a
|
|
`prec.dynamic` alongside it to settle which wins; a static `prec` resolves the shift/reduce while the
|
|
table is built and discards the other parse outright, which is how a bare `{% if x is y %}` came to be
|
|
an error.
|
|
|
|
### A tag group is a sequence of clauses
|
|
|
|
Every group is split so that each tag that opens a body owns a node of its own:
|
|
|
|
```js
|
|
if_clause: ($) => seq(block("if", part($.predicate)), optional($.template)),
|
|
if_group: ($) =>
|
|
seq($.if_clause, repeat($.elif_clause), optional($.else_clause), block("endif")),
|
|
```
|
|
|
|
**A clause is complete without the closing tag, and that is the point.** While a template is being edited
|
|
the closer usually is not there yet, and a group rule cannot be reduced without it — so `{% if a %}{% x %}`
|
|
used to produce a top-level `ERROR` with the body's `template` parented to nothing that says which tag
|
|
opened it. A clause reduces anyway, so it survives error recovery and tooling can walk up from the cursor
|
|
to find the enclosing tag. That is what makes completion of `{% elif %}`, `{% else %}`, `{% empty %}` and
|
|
the end tags possible.
|
|
|
|
**Every `{% %}` is a node of its own, named `<tag>_block`** after the tag as Django spells it: `if_block`,
|
|
`elif_block`, `endif_block`, `block_block`, `templatetag_block`, `custom_tag_block`. A clause is its
|
|
opening block and a body; a group is its clauses and its end block. A block ends at `%}` and nothing can
|
|
extend it, so it reduces on whatever token follows — the end of input included — and a half-written
|
|
template still holds the last `{% for … %}` or `{% endif %}` typed as a whole node, not as loose tokens.
|
|
Stripping `_block` gives the name; a new tag, or a tag group's new part, follows the same scheme.
|
|
|
|
A group's last child is its end block or, where the end tag is missing, a zero-width marker named for it
|
|
(`missing_endif_block`; see the scanner section). So an unterminated group still reduces to a group node,
|
|
and only a stray tag or a body the grammar cannot read produces an `ERROR`.
|
|
|
|
The names themselves need no hand-maintained list either: in the generated `src/node-types.json`, a group's
|
|
`children` are its clause types and its end block, and a clause's children include its opening block, so
|
|
the candidates for a group are its end block plus the opening block of each clause it can hold. For
|
|
`if_group` that is `endif_block` from the group and `if_block`, `elif_block`, `else_block` from its three
|
|
clauses. Going the other way — from a clause the cursor sits in to the
|
|
groups that can hold it — is the same table read backwards, which is what a clause needs when error
|
|
recovery has left it without its group.
|
|
|
|
Do not reach for lookahead to solve that instead: `ts_language_next_state` follows only the shift on `{%`,
|
|
which lands in the "a tag starts here" state no matter how the tree is shaped.
|
|
|
|
`else_clause` is shared by `if_group` and `ifchanged_group`. `comment_clause` and `verbatim_clause` hold raw
|
|
scanner text that admits no tags. `blocktranslate_group` has one clause per spelling, `blocktrans_clause` and
|
|
`blocktranslate_clause`, because each must be closed by its own end tag; the counted form nests a
|
|
`plural_clause` inside it, since `{% plural %}` is only valid after `count`. The clause has to know which
|
|
form its opening block was, so `blocktransRules` builds each as a hidden rule (`_blocktrans_block`,
|
|
`_blocktrans_count_block`) and aliases both to the one visible `blocktrans_block` where the clause uses
|
|
it. Their bodies are
|
|
`_translate_body`, not `template`, which is why these four clauses need no `conflicts` entry — the rule
|
|
below is about a body that can itself start with `{%`.
|
|
|
|
**Every clause needs a `conflicts` entry.** The generator offers a left associativity instead; taking it is
|
|
what broke `{% for %}` in `06aa870`, where `prec.left` resolved the shift/reduce on `{%` statically and
|
|
silently discarded the parse in which the body continues, so every tag inside a loop became an `ERROR`.
|
|
|
|
Sharing the clauses also made the table markedly smaller — 7,200 states and a 515KB library became 5,150
|
|
and 383KB — for the same reason hoisting `blocktranslate`'s pieces into hidden rules did: the automaton
|
|
stops duplicating the body states in every group's inline context. Giving every `{% %}` its own `_block`
|
|
rule shrank it again, from 1,999 states to 1,818, by the same mechanism.
|
|
|
|
### Filter pipes differ by context
|
|
|
|
Django parses everything between `{{` and `}}` — and the entire remainder of `{% filter %}` — as one
|
|
`FilterExpression`, whose regex allows `\s*` on both sides of `|`. Every other tag splits its arguments on
|
|
whitespace first, so a pipe there must be tight. Hence `_filtered_value_spaced` /
|
|
`_filter_expression_spaced`, aliased back to the plain node names so the tree shape stays identical.
|
|
|
|
### Names a `{% load %}` brought in
|
|
|
|
A library's tags and filters are registered in Python, so the grammar cannot know their names, their
|
|
arity, or whether a tag is a block tag. Two fallback rules accept them:
|
|
|
|
- `custom_tag_block` — `simpleTag($, $.identifier)`, i.e. what `simple_tag`/`inclusion_tag` accept, which is how
|
|
a tag is registered unless it needs the parser itself.
|
|
- `_custom_filter`, the last alternative of `filter`, whose name is `$.identifier` rather than one of the
|
|
builtin literals. Hidden, so the tree is what an inline alternative would give, but named so that a
|
|
grammar extending this one can filter it out of `filter`'s `members` the way it can `custom_tag_block` — see
|
|
the extension section in `README.md`.
|
|
|
|
**A name that only exists inside a tag group has to be reserved instead.** `endif`, `else`, `empty` and
|
|
the rest are tokens only in the state their group opens, so where a tag is named the lexer reads them as an
|
|
identifier and `custom_tag_block` takes them: `{% endif %}` on its own parsed as a tag from some library. They are
|
|
listed in `NAMES_INSIDE_A_TAG_GROUP` and applied through the `tag_name` reserved context, which wraps
|
|
`custom_tag_block`'s name **and nothing else** — a reserved context replaces the global one inside whatever it
|
|
wraps, so wrapping the whole rule would un-reserve `as` and reject `{% mytag endif %}`. Adding a tag group
|
|
with a new part or end tag means adding its name there too.
|
|
|
|
**Builtin names must stay keyword-extractable, or the fallbacks swallow them.** `word: $.identifier` turns
|
|
each builtin's name into its own token, which the lexer prefers wherever it is valid, so a builtin commits
|
|
to its own alternative and its argument rules still apply. This is why a filter's name and its `:` are
|
|
separate tokens: while the name was spelled `"add:"` it was not word-shaped, so a bare `add` lexed as an
|
|
identifier and reached the fallback, losing the arity check for all 20 filters that require an argument.
|
|
`FILTERS_WITHOUT_ARGUMENT` / `FILTERS_WITH_ARGUMENT` / `FILTERS_WITH_OPTIONAL_ARGUMENT` carry that arity,
|
|
which Django itself enforces while it parses the template ("add requires 2 arguments, 1 provided").
|
|
|
|
### First-party libraries
|
|
|
|
Django ships `i18n`, `l10n`, `static`, `cache` and `tz` as libraries rather than builtins. Their tags are
|
|
modelled as ordinary tags (so far: `static`, `l10n`, `tz`, `cache` and `i18n`), and the grammar **never requires the `{% load %}`**, because
|
|
an engine can preload a library through `OPTIONS: {"builtins": [...]}`, which makes `{% static "a" %}` valid
|
|
with no load at all.
|
|
|
|
Whether a load is present is a question about the order of nodes, not about how the template parses, so it
|
|
belongs in a tool. `queries/libraries.scm` supplies the halves — `@library.<name>` on each tag, and
|
|
`@load.library` / `@load.name` on the loads — and the tool does the join, because tree-sitter queries have
|
|
no ordering predicate to express it. What such a tool needs to know about Django's behaviour:
|
|
|
|
- Loading is **positional**: `{% trans "x" %}{% load i18n %}` is an error, because the parser adds to its
|
|
tag dict as it walks tokens.
|
|
- Loading is **per file**: a `{% load %}` does not reach a template that `{% extends %}` or `{% include %}`
|
|
this one.
|
|
- `{% load trans from i18n %}` makes only `trans` available, not the rest of the library.
|
|
|
|
A library's filters go in their own arity tables (`L10N_FILTERS_WITHOUT_ARGUMENT`,
|
|
`TZ_FILTERS_WITH_ARGUMENT`, …) rather than into the builtin ones, so that the builtin parity check still has an exact list to compare against.
|
|
|
|
Modelling a library name makes it keyword-extracted, so it no longer reaches `custom_tag_block`. A project that
|
|
registers its own tag under one of these names and a different signature therefore gets a parse error. That
|
|
is accepted: these names are common enough that shadowing them is the mistake.
|
|
|
|
**Adding a library tag or filter touches four files**: `grammar.js`, `queries/libraries.scm` (which library
|
|
it came from), `queries/highlights.scm` (the name, spelled out) and the table in `README.md`, which lists
|
|
every library name the grammar models. All four are hand-maintained, and **no test catches a name left out
|
|
of any of them**: `tree-sitter test` loads every `.scm` under `queries/`, so a pattern naming a node that
|
|
no longer exists aborts the whole run, but a name that was never added passes 304/304 in silence, and the
|
|
README nothing checks at all. The library-contents differential below, run against
|
|
`django.templatetags.<name>.register`, is what finds a gap.
|
|
|
|
### External scanner (`src/scanner.c`)
|
|
|
|
Fifteen external tokens, for the constraints a context-free grammar cannot express:
|
|
|
|
- `push_block`/`push_partial`/`push_verbatim` and the matching `pop_*` — name matching for
|
|
`{% block a %}…{% endblock a %}`. They share **one** stack of `{kind, name}` entries, not one stack per
|
|
kind: the grammar already guarantees nesting, so only the innermost open tag can be closed, and there is
|
|
never a second candidate name to try. Serialized as `<kind><name> ` per entry (`tag_kind_chars`).
|
|
- `verbatim_content`, `comment_content` — raw text up to the matching close tag.
|
|
- `_tag_open`, `_bt_option`, `_kwarg_name` — what a tag has already been given, so that a repeated
|
|
argument is refused: `_bt_option` for `{% blocktranslate %}`'s five options, `_kwarg_name` for `name=value`
|
|
arguments — those of a tag registered with `simple_tag`, and the `with` bindings of `{% with %}`,
|
|
`{% include %}` and `{% blocktranslate %}` — whose names are open-ended and so are kept as names rather
|
|
than as a mask. `_kwarg_name` returns nothing unless the name is followed by `=`, which
|
|
is also what tells an argument apart from the `as name` clause or the end of the tag. `_tag_open` clears
|
|
both, and every rule that uses either has one. Both are zero width and hidden, so no tree changes. The state is one byte of
|
|
flags, not a stack, because the tag cannot nest: its body admits no tags. It lives outside the
|
|
`stack`/`error` union, is cleared in both branches of `reset_scanner`, and is serialized after the
|
|
status character, so a GLR stack split copies it like everything else.
|
|
- `_group_open`, `_group_close` — zero width, before the name of a tag that opens or closes a tag group
|
|
(`groupBlock`). The scanner reads the name ahead of the grammar and pushes or pops a nameless entry, so
|
|
the stack holds every open group and not only the three whose names it matches; `block`, `partialdef`
|
|
and `verbatim` still push and pop through their own tokens, and are the only kinds with a pop token.
|
|
`{% plural %}` goes through `_group_close` too: reading the `count` option (`_bt_option`) turns a
|
|
blocktranslate entry into its counted kind, which waits for `plural` before its end tag. `tag_groups`
|
|
holds each kind's opening name and the names that belong to it, and `group_names` **must stay in sync
|
|
with `NAMES_INSIDE_A_TAG_GROUP`**.
|
|
- `_missing_tag` — the marker where a group's required tag is missing, aliased at each use to
|
|
`missing_endX_block` or `missing_plural_block` (`MISSING_BLOCKS`). It is placed only at the end of
|
|
input; before a tag whose name is reserved to some other group, which therefore cannot belong to this
|
|
body; or before an `endblock`/`endpartialdef` naming a block further out than the innermost. It closes
|
|
the innermost group on the stack as the end tag would, the group reduces with no `ERROR`, and
|
|
`queries/errors.scm` is what reports it. **It is one token aliased per group, never a token per
|
|
group**: tree-sitter does not merge parse states whose valid external tokens differ, so a token per
|
|
group cloned every body state for each kind of group and took the table from 1,818 states to 17,664
|
|
(it is 3,057 as built). That trade is also why the scanner keeps a stack of every group instead of
|
|
being told by the token which group it is in. A supertype would be the natural way to match every
|
|
marker, but the generator silently drops one whose members are aliases, so `errors.scm` lists them.
|
|
- `matcher_error` — used by no rule; returned only if the scanner reaches an error state.
|
|
|
|
`check_space()` defines whitespace for the scanner and **must stay in sync with `SEP`**, because the
|
|
scanner skips whitespace itself while matching names.
|
|
|
|
Prefer the grammar over the scanner. A useful test: does the constraint change how the input _parses_?
|
|
Block-name matching does — it decides which close tag closes what.
|
|
|
|
"This keyword may not repeat" normally does not, and belongs in a linter. `blocktranslate` is the exception,
|
|
and the reason is worth keeping straight: its five options are a fixed set, so uniqueness _is_ expressible —
|
|
`arrangements` expresses exactly that — and the scanner is standing in for a grammar that would be correct
|
|
but takes minutes to build. Where the set is open, as with `{% querystring %}`'s `**kwargs`, no grammar
|
|
can express it at all, and the scanner remembers the names the tag has been given instead of a mask of
|
|
known ones. Only `parse_bits` actually refuses a repeat: the `with` bindings of
|
|
`{% with %}`, `{% include %}` and `{% blocktranslate %}` all go through `token_kwargs`, which builds a dict
|
|
and lets the later value win. They are guarded here anyway, as a deliberate divergence — writing a name
|
|
twice has no use and is a mistake worth reporting. `{% blocktranslate %}`'s `count` is not part of that,
|
|
because Django keeps it in a dict of its own, so `{% blocktranslate with a=1 count a=2 %}` is accepted here
|
|
as it is there.
|
|
|
|
Zero-width guards carry one hazard: `recover_with_missing` can supply one without the scanner running. The
|
|
`if (valid_symbols[MatcherError]) return false;` near the top of `scan` is what stops a guard being scanned
|
|
during tree-sitter's mark-everything-valid recovery pass, so new guards go **below** it — and above the
|
|
whitespace loop, which skips the separator the grammar still has to match. A guard must also never be valid
|
|
where `content` is: a zero-width token at a content boundary preempts the internal lexer and `content` stops
|
|
matching. `_missing_tag` is the one exception, and is safe only because it returns a token solely at the end
|
|
of input or at `{%`, where `content` cannot match. It is scanned above the raw-text scanners, which find
|
|
nothing at the end of an empty body.
|
|
|
|
### Queries
|
|
|
|
`queries/highlights.scm` lists tag and filter names explicitly, so adding either to the grammar means
|
|
adding it there too; filter names appear there without their colon. A name the grammar does not know is
|
|
captured through its field instead (`(filter name: (identifier))`, `(custom_tag_block tag: (identifier))`).
|
|
|
|
`queries/libraries.scm` is not a query editors run themselves; it is data for a linter, and every tag or
|
|
filter added from a library belongs in it. `queries/locals.scm` relies on the `variable:` field to tell a
|
|
binding from a reference — another reason to keep `asVariable` uniform. `{% blocktranslate %}`'s `asvar`
|
|
target carries an `asvar:` field instead, because it is written inside the tag but assigned outside it:
|
|
the scope on `blocktrans_clause` / `blocktranslate_clause` confines the `with` and `count` bindings, and would swallow `asvar`
|
|
too if it were spelled the same way. `queries/exports.scm` names it for a tool to
|
|
bind in the scope around the block, which a locals query cannot do: it places a definition in the innermost
|
|
scope containing it and offers no way out again.
|
|
|
|
`queries/conditionals.scm` is data for a tool too: `@conditional.group` / `@conditional.branch` /
|
|
`@conditional.default` say which bodies render on only some passes, so a checker can tell that
|
|
`{% if a %}{% now "Y" as n %}{% endif %}{{ n }}` may reach `{{ n }}` with nothing bound. A locals query
|
|
cannot: it matches a reference to the definition whose scope contains it and has no notion of a path not
|
|
taken. A group is exhaustive iff it has a `@conditional.default`, so each pattern captures the group
|
|
alongside one of its parts and a tool needs no list of which clause belongs to which group. Adding a tag
|
|
group whose body may be skipped means adding it there — `{% cache %}` is the non-obvious one, since a
|
|
cache hit skips the body outright and `CacheNode` pushes no context.
|
|
|
|
A branch that is also a `@local.scope` confines its bindings whichever way the branch goes, which is why
|
|
`{% for %}` is inert for that check: `ForNode` renders `nodelist_empty` **inside** the same
|
|
`context.push()` as the loop body, so `empty_clause` is a scope alongside `for_clause`.
|
|
|
|
`queries/errors.scm` collects what an editor should report: `(ERROR)` as `@error.syntax`, `(MISSING)` as
|
|
`@error.missing`, and every missing-tag marker as `@error.missing_tag`. A tree holding only markers has
|
|
no `has_error`, so a tool that checks that flag alone misses them. Adding a tag group means adding its
|
|
marker there, since no supertype matches them all.
|
|
|
|
## Testing
|
|
|
|
Corpus tests live in `test/corpus/*.txt`, one file per tag, and `missing.txt` for the missing-tag markers.
|
|
|
|
- A test whose header carries `:error` asserts only that the parse fails. Prefer it for syntax-error cases
|
|
over pinning an error-recovery tree, which is brittle.
|
|
- An omitted field annotation in an expected tree is treated as "don't care"; `--show-fields` forces fields
|
|
into diffs.
|
|
- `test/highlights/` is **not** run by `tree-sitter test` (which looks for `test/highlight/`), and its
|
|
`example.html` uses a custom `{% greet %}` tag the grammar does not model, so it parses with errors by
|
|
design.
|
|
|
|
### Checking against Django itself
|
|
|
|
`Pipfile.lock` pins Django (`Pipfile` itself says `django = "*"`) so the grammar can be diffed against the
|
|
real implementation instead of against documentation or memory. Both checks are worth repeating whenever tags, filters or literals change:
|
|
|
|
- **Builtin parity** — enumerate `Engine.default_builtins`, then compare `register.tags` and
|
|
`register.filters` against `grammar.js`. For filters also compare argument arity via
|
|
`inspect.signature`, ignoring the framework-injected `autoescape` parameter, which is never written in a
|
|
template. Note `querystring` and `csp_nonce_attr` register as `simple_tag`, not `@register.tag`.
|
|
- **Library contents** — `django.templatetags.<name>.register` gives a library's `tags` and `filters`.
|
|
All of them are registered with `@register.tag`, so `inspect.signature` only reports `(parser, token)`
|
|
and a signature has to come from the differential below or from reading the compile function.
|
|
- **Syntax differential** — run candidate snippets through `django.template.Template()` and through
|
|
`tree-sitter parse`, and compare accept/reject. This is how the `simple_tag` argument rules, the `_()`
|
|
literal, and the filter-pipe spacing were pinned down.
|
|
|
|
`pipenv run` has resolved to a _different_ project's virtualenv from this directory; confirm with
|
|
`python -c "import django; print(django.get_version())"` and fall back to the explicit venv path.
|
|
|
|
### Deliberate divergences from Django
|
|
|
|
`README.md` carries a shorter list of its own, and the two are not the same set on purpose: the README
|
|
lists only the divergences where Django accepts an input that is almost certainly a mistake, which is the
|
|
rule it states. Anything added here that fits that rule belongs there too; anything that does not — an
|
|
unavoidable limitation, or a case where this grammar is the more permissive one — stays here.
|
|
|
|
- Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal,
|
|
so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here. `1.` is the
|
|
one that needs care: `Variable` calls `float()` and then rejects what it parsed if the last character is
|
|
a dot (`# "2." is invalid`), so a trailing dot is a number only with an exponent after it — `1.e5` is a
|
|
float and `1.` is not. The `number` regex is spelled to say that, and `test/corpus/template_variable.txt`
|
|
pins all three.
|
|
- Django has no boolean or `None` token either. `True`, `False` and `None` are lookups, which resolve
|
|
because `BaseContext._reset_dicts` seeds every context with `{"True": True, "False": False, "None": None}`.
|
|
They are `boolean` and `none` literals here, so that a tool reading `queries/locals.scm` does not count
|
|
them as references to something the template never defines. Shadowing still parses — `{% with True=1 %}`
|
|
binds an `identifier`, because the keyword is not valid where a binding name is — but `{{ True.x }}` is
|
|
an error here and an ordinary two-part lookup in Django, which renders `string_if_invalid`.
|
|
- A `with` binding written twice (`{% with a=1 a=2 %}`, and the same in `{% include %}` and
|
|
`{% blocktranslate %}`) is an error here and not in Django, where `token_kwargs` builds a dict and the
|
|
later value wins. It has no use, so it is treated as the mistake it almost certainly is.
|
|
- `{% csp_nonce_attr "a" media="b" %}` is accepted, matching Django, which only rejects it at render time.
|
|
- A misspelled tag or filter cannot be caught: `{{ x|lenght }}` is indistinguishable from a filter some
|
|
library registered. Cross-referencing names against `{% load %}` needs the project's Python, so it
|
|
belongs in a linter; the `load_block` rule already parses the library and `from` names for one to use.
|
|
- `{% mytag %}…{% endmytag %}` parses as two sibling `custom_tag_block`s. Nesting them would mean guessing that
|
|
an unknown tag is a block tag.
|
|
- A custom tag's contents are only assumed to be the `simple_tag` shape. `@register.tag` receives the raw
|
|
token and may parse anything, so `{% mytag <<>> %}` is rejected here and not by Django.
|
|
- A library may override a builtin (`Parser.add_library` is `self.filters.update(...)`), so a project that
|
|
redefines `length` with a different arity gets a false error here.
|
|
- Django's `static` tags ignore what follows their argument: `StaticNode.handle_token` looks for `as` two
|
|
bits from the end, so `{% static "a" junk %}` and `{% static "a" as u v %}` are accepted there and
|
|
rejected here, and `PrefixNode.handle_token` raises `IndexError` rather than `TemplateSyntaxError` on
|
|
`{% get_static_prefix as %}`.
|
|
- Django's `{% cache %}` takes its fragment name as the raw word in that position and finds the cache to
|
|
use by testing whether the last word starts with `using=`, and only when something precedes it. So
|
|
`{% cache 500 sidebar.name %}` names a fragment there, `{% cache 500 using="c" %}` names a fragment
|
|
called `using="c"` rather than choosing a cache, and `{% cache 500 sidebar using= %}` takes an empty
|
|
name. All three are errors here.
|
|
- `partial`/`partialdef` are Django builtins as of Django 6; `elif`/`else`/`empty` are modelled as parts of
|
|
their enclosing tag rather than as separate tags.
|
|
- An unterminated group parses here and raises `TemplateSyntaxError` in Django: `{% if a %}x`, or a
|
|
`{% for %}` closed while an `{% if %}` inside it is still open, holds a `missing_endif_block` marker
|
|
where the end tag belongs. The marker is the error, reported through `queries/errors.scm`; a check
|
|
for `ERROR` nodes alone accepts the tree.
|