tree-sitter-django/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

A tree-sitter grammar for the Django Template Language: `{% %}` tags and `{{ }}` variables.

## Commands

```sh
npm run parser-generate   # regenerate src/parser.c, grammar.json, node-types.json from grammar.js
npm run parser-test       # run the corpus tests
npm run parser-build      # build tree-sitter-django.wasm
npm run playground        # explore the grammar interactively
npm install               # also builds the Node binding, through node-gyp-build
npm run binding-test      # the Node binding's own test
cargo test                # the Rust binding
make                      # the C library, and make clean to remove it

npx tree-sitter test -i "Cycle With As"      # one test, by name regex
npx tree-sitter test --file-name cycle.txt   # one corpus file
npx tree-sitter test -r                      # force a rebuild of the parser library
npx tree-sitter parse file.html              # parse one file
npx tree-sitter parse --debug normal f.html  # lex/shift/reduce trace
npx prettier --write grammar.js queries test CLAUDE.md   # not `.` — see below
```

`src/parser.c`, `src/grammar.json` and `src/node-types.json` are generated **and committed**, so run
`parser-generate` after every change to `grammar.js` or `src/scanner.c`.

Format with prettier over specific paths rather than `npm run format` (`prettier --write .`): that also
rewrites the generated `src/grammar.json` and `src/node-types.json`, which `parser-generate` then writes
back, producing churn in both directions.

`npm run lint` does not work: `eslint.config.mjs` was copied from a Next.js project and enables
`@typescript-eslint` and `react-hooks` rules whose plugins aren't installed. Formatting is prettier's job.

### Bindings

`bindings/` and the manifests beside it — `binding.gyp`, `Cargo.toml`, `pyproject.toml`, `setup.py`,
`go.mod`, `Package.swift`, `Makefile`, `CMakeLists.txt` — are generated by `tree-sitter init` from the
`bindings` and `metadata` sections of `tree-sitter.json`, and `.gitattributes` marks them
`linguist-generated`. Do not edit them by hand: change `tree-sitter.json` and regenerate.

`tree-sitter init --update` only rewrites what it considers outdated, and a changed `version` or
`description` does not count, so the manifests keep the old values until they are deleted and `init` is run
again. It leaves `README.md`, `package.json` and `.gitignore` alone, which is why the Node binding's
`package.json` entries — `main`/`types`, the `.` export, `files`, `node-addon-api`, `node-gyp-build`, the
`install` script and the optional `tree-sitter` peer — are maintained here rather than generated.

Every binding was built and run on macOS: `npm install` plus `npm run binding-test`, `cargo test`,
`go test ./bindings/go/`, `pip install .` with a parse, `swift build`, and `make`. The Go binding needs a
`go.sum`, which `go mod tidy` writes and which is committed with the rest.

`src/scanner.c` compiles without warnings, and should stay that way: cc and clang warn about `&&` within
`||` (`-Wlogical-op-parentheses`), so a comparison chain written with both needs its `&&` groups
parenthesized.

### CI and releasing

`.github/workflows/ci.yml` runs on pull requests, and the release workflow calls it: the corpus tests, a
check that `npm run parser-generate` leaves `src/` unchanged, the Node binding on Linux, macOS and Windows,
and the Rust, Python, Go and Swift bindings.

`.github/workflows/release.yml` runs on every push to `main` but only releases when `package.json`'s version
differs from the previous commit's and no `v<version>` tag exists yet. Then it runs CI, `prebuildify` on
five runners (Linux x64 and arm64, macOS x64 and arm64, Windows x64), and a last job that tags `v<version>`,
creates the GitHub release with `tree-sitter-django.wasm` attached, and runs
`npm stage publish --provenance`. **A release is a version-bump commit on `main`**; no other push starts one.

**The workflow stages the publish; it does not publish.** `npm stage publish` hands npm the tarball and
holds it there, unavailable to install, until a maintainer runs `npm stage approve <stage-id>` and answers a
2FA prompt — proof of presence deferred rather than dropped, which is npm's answer to deprecating the tokens
that let a workflow publish unattended. The job summary prints the `view`/`download`/`approve`/`reject`
commands for the stage it made. Approving needs npm 12 (`npm install -g npm@^12`), which is also why the job
installs it: no runner image ships an npm that has the command.

Running the workflow by hand (`workflow_dispatch`) is the retry: it stages whatever version `package.json`
holds as long as `npm view` does not find it on the registry already, and it skips the tag and the release
when a previous attempt left them behind, replacing only the Wasm asset. So a run that tagged and then
failed to stage is finished by running it again, and a run against an already published version stops at the
version job. A version that is _staged_ and not yet approved is invisible to `npm view` while still
occupying npm's version index, so a second stage of it is an error rather than a no-op; the stage job asks
`npm stage list` first and skips straight to the approval summary when it finds one. That listing is
best-effort — a short-lived OIDC token may not run `npm stage` subcommands at all — and a failure there just
means the stage is attempted.

Releasing needs three things to line up:

- credentials to stage with: either a trusted publisher configured for this workflow
  (`npm trust github --allow-stage-publish`), which needs no secret at all because npm exchanges the
  workflow's OIDC token for one of its own, or the `NPM_TOKEN` secret as the fallback. Staging works with any
  token type, so that token no longer needs — and should no longer have — the 2FA bypass an automation token
  carries. Both the exchange and provenance need `id-token: write`, which the publish job already declares.
- **the version in `package.json` and in `tree-sitter.json`'s `metadata`**. The release trigger reads the
  first; the second is compiled into `src/parser.c` as the language's version, so bumping one alone leaves
  the parser disagreeing with the manifest.
- `npm run parser-generate` after that bump, committed with it, or CI's up-to-date check fails on a two-line
  diff in `src/parser.c`.

The arm64 Linux runner (`ubuntu-24.04-arm`) is free for public repositories only. `tree-sitter build --wasm`
runs Emscripten through Docker, which the ubuntu runners provide.

**Windows needs a newer npm than its runner ships.** `windows-latest` is Windows Server 2025 **with Visual
Studio 2026**, and the node-gyp bundled with that image's npm (11.5.0) reads VS 18 as
`unknown version "undefined"` and then reports "Could not find any Visual Studio installation to use", so
`npm ci` fails in the install script. node-gyp learned VS 2026 in 12.1.0 and npm 12 bundles 13, which is why
both jobs that compile the addon there — CI's Node binding and the release's `prebuild` — install `npm@^12`
first. Pinning `windows-2022` instead would work, at the price of testing against a toolchain the image no
longer has. Installs of the published package never meet this: they unpack a prebuild and never run node-gyp.

**The npm package must carry its prebuilds**, or `npm install` runs `node-gyp-build`, finds nothing and
compiles, which fails on an image without `python3`, `make` and a compiler — `node:20-slim`, for instance.
0.1.0 was published by hand before the workflow existed and shipped none, which is what that image hit. The
`prebuild` jobs are what produce them, and `files` carries `prebuilds/**`.

`npm run prebuildify` passes `--napi --strip --tag-libc`. `--napi` makes one binary per platform that every
Node ABI can load, so the file needs no ABI tag and is named after the package. `--tag-libc` adds `glibc` or
`musl`: `node-gyp-build` treats a file with no libc tag as matching anything, so an Alpine install would
otherwise load a glibc binary and fail at run time; with the tag it compiles instead. Only glibc is built,
so musl installs need a toolchain.

`package.json` exports the Wasm build as `tree-sitter-django/tree-sitter-django.wasm`. An `exports` map
hides everything it does not name, so each published path — the queries, `grammar.js`, `src/scanner.c` —
needs its own entry.

### Stale parser library

tree-sitter caches the compiled parser in `~/.cache/tree-sitter/lib/django.*`, and it has served a stale
build after edits — producing parses that contradict the current grammar (debug traces naming tokens that
no longer exist). **If a parse result disagrees with `grammar.js`, suspect this first**: rerun with
`npx tree-sitter test -r`, or delete `~/.cache/tree-sitter/lib/django*`.

## Architecture

### Whitespace is syntax, not `extras`

`extras: []`. Django's `Token.split_contents()` splits a tag's contents on whitespace _before_ parsing any
argument, so `{% cycle 1as x %}` is the single token `1as` — an error, not `1` followed by `as`. `extras`
can never be made mandatory, so the separator is an explicit token and every rule is built from helpers
that place it:

- `SEP` — `sym("_sep")`, the hidden rule `_sep: /[ \t\r\n]+/`, which a grammar extending this one writes as
  `$._sep`. Newlines included: Django matches tags with `re.DOTALL`, so tags may span lines.
- `part(...)` — one tag part, taking the whitespace that separates it from the part before.
- `joined(first, ...rest)` — parts separated from _each other_, with no leading separator (`not in`).
- `tag(name, ...args)` — `{%`, optional separator, tag name, args, optional separator, `%}`; tags may
  hug their delimiters (`{%cycle 1%}`).
- `asVariable($)` — the `as name` clause. Every tag that takes one names the target `field("variable", …)`;
  do not invent a second name for it.
- `simpleTag($, name, args, kwargs)` — a tag registered with Django's `simple_tag`, which always accepts
  positional filter expressions, then `name=value` keywords, then `as name`. `args`: `true` (`*args`), a
  count (at most N), or `false`. `kwargs`: `true` (`**kwargs`), `false`, or `{name: rule}` for a known
  signature. Put the Python signature in a comment above the call site.
- `arrangements(items)` — every ordered subset of `items`, so that a tag's options may be written in any
  order but never twice. Used by `translate_tag` and `include_tag`; keyword arguments are guarded by the scanner
  instead. Grows as `Σ C(n,k)·k!` (65 for four names), which is
  why it is only used for signatures that name their arguments.
- `wordAsValueRules(word)` / `wordAsValue($, word, continued)` — a tag's own keyword written where a value
  goes. Django tells its flag from a variable by comparing the whole bit, so `reversed` is the flag of
  `{% for %}` while `reversed.x` and `reversed|last` are lookups; `continued` selects that second group.
  Needed because keyword extraction hands the parser the keyword wherever both it and `identifier` are
  valid, so the reading in which the word names a variable has to be built back out of it. The rules are
  hidden and aliased at the use site, one per level (`variable_attribute`, `value`, `filtered_value`),
  because an alias renames the node its own rule makes and **cannot nest one inside another** — writing
  it as nested inline `alias(alias(...))` calls silently yields a flat, partly zero-width tree that still
  generates and still parses. Check the tree, not just the accept.

Consequences worth knowing before editing a rule:

- **No `token.immediate` anywhere** — without extras, adjacency is already the default. Use it only to
  forbid whitespace somewhere a separator is otherwise allowed.
- **Every optional trailing part needs somewhere for its separator to go.** Getting this wrong does not
  fail at generate time; it shows up as `MISSING` nodes in a passing-looking parse.
- **`conflicts` entries are the normal fix, not precedence.** With explicit separators the lookahead at a
  part boundary is `SEP` instead of the meaningful token, so LR(1) loses its discriminator. That is why
  `predicate`, `binaryOperator` and `[$.library, $.load]` are listed there. Where two readings both stay
  valid to the end — `is not` as one operator or as `is` and a prefix `not` — the conflict needs a
  `prec.dynamic` alongside it to settle which wins; a static `prec` resolves the shift/reduce while the
  table is built and discards the other parse outright, which is how a bare `{% if x is y %}` came to be
  an error.

### A tag group is a sequence of blocks

Every group is split so that each tag that opens a body owns a node of its own:

```js
if_tag: ($) => startTag("if", part($.predicate)),
if_block: ($) => seq($.if_tag, optional($.template)),
endif_tag: ($) => endTag("endif"),
if_group: ($) =>
  seq(
    $.if_block,
    repeat($.elif_block),
    optional($.else_block),
    choice($.endif_tag, alias($._missing_tag, $.missing_endif_tag)),
  ),
```

**A block is complete without the closing tag, and that is the point.** While a template is being edited
the closer usually is not there yet, and a group rule cannot be reduced without it — so `{% if a %}{% x %}`
used to produce a top-level `ERROR` with the body's `template` parented to nothing that says which tag
opened it. A block reduces anyway, so it survives error recovery and tooling can walk up from the cursor
to find the enclosing tag. That is what makes completion of `{% elif %}`, `{% else %}`, `{% empty %}` and
the end tags possible.

**Every `{% %}` is a node of its own, named `<name>_tag`** after the tag's name as Django spells it: `if_tag`,
`elif_tag`, `endif_tag`, `block_tag`, `templatetag_tag`, `custom_tag`. A block is its
opening tag and a body; a group is its blocks and its end tag. A tag ends at `%}` and nothing can
extend it, so it reduces on whatever token follows — the end of input included — and a half-written
template still holds the last `{% for … %}` or `{% endif %}` typed as a whole node, not as loose tokens.
Stripping `_tag` gives the tag name; a new tag, or a tag group's new part, follows the same scheme.

A group's last child is its end block or, where the end tag is missing, a zero-width marker named for it
(`missing_endif_tag`; see the scanner section). So an unterminated group still reduces to a group node,
and a group's tag written where no open group can hold it is an `unexpected_tag` (see the load section
below), so only a body the grammar cannot read produces an `ERROR`.

Every tag is built from ten inline utility rules — `simpleTagOpen`/`simpleTagClose`,
`startTagOpen`/`startTagClose`,
`followTagOpen`/`followTagClose`, `repeatTagOpen`/`repeatTagClose`, `endTagOpen`/`endTagClose` —
through the helpers `tag` (a tag outside any group, from `simpleTagOpen`/`simpleTagClose`), `startTag`, `followTag` (a middle tag a group
holds at most once: `else`, `empty`, `plural`), `repeatTag` (one it may repeat: `elif`) and `endTag`.
They are inline rules rather than JS helpers so that a grammar extending this one can use them. **Every
tag must begin with one of the opens**, one an extending grammar adds included: the token they read the
tag's name with is valid wherever a tag's name is and the scanner always returns it, so a tag written from a bare `"{%"` never matches: its name is read as a `custom_tag` instead,
with no error to show for it. Choose `followTag` or `repeatTag` by what the grammar allows, because the
scanner trusts it: a tag marked once-only that can in fact repeat is an `unexpected_tag` from its
second use on, and one marked repeatable that cannot is never flagged when repeated.

The names themselves need no hand-maintained list either: in the generated `src/node-types.json`, a group's
`children` are its block types and its end tag, and a block's children include its opening tag, so
the candidates for a group are its end tag plus the opening tag of each block it can hold. For
`if_group` that is `endif_tag` from the group and `if_tag`, `elif_tag`, `else_tag` from its three
blocks. Going the other way — from a block the cursor sits in to the
groups that can hold it — is the same table read backwards, which is what a block needs when error
recovery has left it without its group.

Do not reach for lookahead to solve that instead: `ts_language_next_state` follows only the shift on `{%`,
which lands in the "a tag starts here" state no matter how the tree is shaped.

`else_block` is shared by `if_group` and `ifchanged_group`. `comment_block` and `verbatim_block` hold raw
scanner text that admits no tags. `blocktranslate_group` has one block per spelling, `blocktrans_block` and
`blocktranslate_block`, because each must be closed by its own end tag; the counted form nests a
`plural_block` inside it, since `{% plural %}` is only valid after `count`. The block has to know which
form its opening tag was, so `blocktransRules` builds each as a hidden rule (`_blocktrans_tag`,
`_blocktrans_count_tag`) and aliases both to the one visible `blocktrans_tag` where the block uses
it. Their bodies are
`_translate_body`, not `template`, which is why these four blocks need no `conflicts` entry — the rule
below is about a body that can itself start with `{%`.

**Every block needs a `conflicts` entry.** The generator offers a left associativity instead; taking it is
what broke `{% for %}` in `06aa870`, where `prec.left` resolved the shift/reduce on `{%` statically and
silently discarded the parse in which the body continues, so every tag inside a loop became an `ERROR`.

Sharing the blocks also made the table markedly smaller — 7,200 states and a 515KB library became 5,150
and 383KB — for the same reason hoisting `blocktranslate`'s pieces into hidden rules did: the automaton
stops duplicating the body states in every group's inline context. Giving every `{% %}` its own `_tag`
rule shrank it again, from 1,999 states to 1,818, by the same mechanism. The group stack's tokens (see the
scanner section) cost 247, to 3,304.

### Filter pipes differ by context

Django parses everything between `{{` and `}}` — and the entire remainder of `{% filter %}` — as one
`FilterExpression`, whose regex allows `\s*` on both sides of `|`. Every other tag splits its arguments on
whitespace first, so a pipe there must be tight. Hence `_filtered_value_spaced` /
`_filter_expression_spaced`, aliased back to the plain node names so the tree shape stays identical.

### Names a `{% load %}` brought in

A library's tags and filters are registered in Python, so the grammar cannot know their names, their
arity, or whether a tag is a block tag. Two fallback rules accept them:

- `custom_tag` — `simpleTag($, $.identifier)`, i.e. what `simple_tag`/`inclusion_tag` accept, which is how
  a tag is registered unless it needs the parser itself.
- `_custom_filter`, the last alternative of `filter`, whose name is `$.identifier` rather than one of the
  builtin literals. Hidden, so the tree is what an inline alternative would give, but named so that a
  grammar extending this one can filter it out of `filter`'s `members` the way it can `custom_tag` — see
  the extension section in `README.md`.

**A name that only exists inside a tag group has to be reserved instead.** `endif`, `else`, `empty` and
the rest are tokens only in the state their group opens, so where a tag is named the lexer reads them as an
identifier and `custom_tag` takes them: `{% endif %}` on its own parsed as a tag from some library. They are
listed in `NAMES_INSIDE_A_TAG_GROUP` and applied through the `tag_name` reserved context, which wraps
`custom_tag`'s name **and nothing else** — a reserved context replaces the global one inside whatever it
wraps, so wrapping the whole rule would un-reserve `as` and reject `{% mytag endif %}`. Adding a tag group
with a new part or end tag means adding its name there too.

**The same names are what `unexpected_tag` accepts.** Once reserved, a group's tag written where no open
group can hold it — `{% endif %}` with nothing open, `{% else %}` in a `{% for %}` — used to be an `ERROR`,
and error recovery placed it badly: every tag starts with `{%` and the read token, which the parser has
already accepted when the name turns out to be wrong, so recovery rewinds to just after them and folds the
stray tag into whichever tag comes next, which then holds the `ERROR`. `unexpected_tag` takes any of
`NAMES_INSIDE_A_TAG_GROUP` with loose arguments, so there is nothing to recover; `prec.dynamic(-2)`, or `-4`, keeps it
from displacing a tag the group does take, and `queries/errors.scm` reports it as
`@error.unexpected_tag`. It added 31 states and no `conflicts` entry.

**Words a tag does not take are an `unexpected_argument` inside it.** Every close utility accepts one
before its scanner token, so `{% else x %}` stays the `else` of its group, `{% endif junk %}` still closes
its `if` (it used to leave the whole file an `ERROR`), and `{% static "a.css" "b.css" %}` is a `static_tag`.
The node is a run of hidden `_unexpected_word`s, and three details make it work:

- **Lexical precedence -1.** A loose word is `token(prec(-1, /(?:[^\s%}]|\}[^\s%}])+/))`, which never holds `}}`:
  lexer states are shared between parse states, so the token is tried inside `{{ }}` too, and after
  `{{ x| ` it once swallowed the closing braces and left recovery nothing to close the variable with. Tree-sitter weighs lexical
  precedence before match length, so a token an argument really takes wins wherever one is valid
  (`b|upper` stays a filtered value) and a loose word only matches where nothing else can.
- **Identifiers are words too.** Wherever a keyword is valid — after `{% for a in b`, where `reversed` may
  follow, or after `as`, `only`, `noop`, `silent` — tree-sitter lexes every word as an `identifier` to
  check for the keyword, and the loose-word token never gets a turn. So a word may also be an
  `identifier`, aliased to an anonymous `unexpected_word` so `queries/locals.scm` does not count it as a
  reference, with whatever touches it (`a=2`, `x.y|upper`, `"z"|lower:"q"`) as an immediate tail.
- **-1 of dynamic precedence per word**, not per node, so a reading that flags more words than it must
  always loses: `{% cycle a b as x silent zqx %}` flags `zqx`, not everything after `a`. `unexpected_tag`
  costs -2 or -4 so that a misused tag its group does take beats reading it as a stray.

A tag with a known keyword signature (`simpleTag` given names, so far only `csp_nonce_attr`) needs one more
piece: the scanner's `_kwarg_name` accepts any name followed by `=` and commits the parser to a keyword
argument before the name is checked, so `{% csp_nonce_attr other="b" %}` would be an `ERROR`. `simpleTag`
offers `_unexpected_keyword` — an identifier and its tail, aliased to `unexpected_argument` — beside the
names it knows.

A separator after any argument can now also start stray words, so this needs `conflicts` entries for
`unexpected_argument`, `predicate`, `_translate_option`, and `_unexpected_word` with `variable_attribute`,
`resetcycle_tag` and `library`. It took the table from 3,363 states to 4,913 and the Wasm build from 251KB to
323KB. Two consequences: a repeat the scanner refuses — `{% with a=1 a=2 %}`, `{% blocktrans trimmed trimmed %}`
— is reported as an unexpected argument rather than an `ERROR`; and a stray **reserved** word is still an
`ERROR`, since `not`, `if`, `in`, `is`, `as`, `for` and `from` can never be an identifier:
`{% trans "hi" as x as y %}` errors on the second `as`. A tag missing a required argument, such as
`{% with a = 1 %}`, is an `ERROR` too; `unexpected_argument` only takes words after a tag that is otherwise
complete.

Where several tags share a name the group holds only once — `{% if %}{% else %}{% else %}{% else %}` — reading
any of them as the stray parses, so the ties are broken by dynamic precedence: `unexpected_tag` is -2 when
`_group_held` confirms the innermost group already holds the name and -4 otherwise, so each tag flagged
before its time costs one more, and the first still opens the block however many follow.

**Builtin names must stay keyword-extractable, or the fallbacks swallow them.** `word: $.identifier` turns
each builtin's name into its own token, which the lexer prefers wherever it is valid, so a builtin commits
to its own alternative and its argument rules still apply. This is why a filter's name and its `:` are
separate tokens: while the name was spelled `"add:"` it was not word-shaped, so a bare `add` lexed as an
identifier and reached the fallback, losing the arity check for all 20 filters that require an argument.
`FILTERS_WITHOUT_ARGUMENT` / `FILTERS_WITH_ARGUMENT` / `FILTERS_WITH_OPTIONAL_ARGUMENT` carry that arity,
which Django itself enforces while it parses the template ("add requires 2 arguments, 1 provided").

### First-party libraries

Django ships `i18n`, `l10n`, `static`, `cache` and `tz` as libraries rather than builtins. Their tags are
modelled as ordinary tags (so far: `static`, `l10n`, `tz`, `cache` and `i18n`), and the grammar **never requires the `{% load %}`**, because
an engine can preload a library through `OPTIONS: {"builtins": [...]}`, which makes `{% static "a" %}` valid
with no load at all.

Whether a load is present is a question about the order of nodes, not about how the template parses, so it
belongs in a tool. `queries/libraries.scm` supplies the halves — `@library.<name>` on each tag, and
`@load.library` / `@load.name` on the loads — and the tool does the join, because tree-sitter queries have
no ordering predicate to express it. What such a tool needs to know about Django's behaviour:

- Loading is **positional**: `{% trans "x" %}{% load i18n %}` is an error, because the parser adds to its
  tag dict as it walks tokens.
- Loading is **per file**: a `{% load %}` does not reach a template that `{% extends %}` or `{% include %}`
  this one.
- `{% load trans from i18n %}` makes only `trans` available, not the rest of the library.

A library's filters go in their own arity tables (`L10N_FILTERS_WITHOUT_ARGUMENT`,
`TZ_FILTERS_WITH_ARGUMENT`, …) rather than into the builtin ones, so that the builtin parity check still has an exact list to compare against.

Modelling a library name makes it keyword-extracted, so it no longer reaches `custom_tag`. A project that
registers its own tag under one of these names and a different signature therefore gets a parse error. That
is accepted: these names are common enough that shadowing them is the mistake.

**Adding a library tag or filter touches four files**: `grammar.js`, `queries/libraries.scm` (which library
it came from), `queries/highlights.scm` (the name, spelled out) and the table in `README.md`, which lists
every library name the grammar models. All four are hand-maintained, and **no test catches a name left out
of any of them**: `tree-sitter test` loads every `.scm` under `queries/`, so a pattern naming a node that
no longer exists aborts the whole run, but a name that was never added passes 304/304 in silence, and the
README nothing checks at all. The library-contents differential below, run against
`django.templatetags.<name>.register`, is what finds a gap.

### External scanner (`src/scanner.c`)

Twenty external tokens, for the constraints a context-free grammar cannot express:

- `push_block`/`push_partial`/`push_verbatim` and the matching `pop_*` — the names in
  `{% block a %}…{% endblock a %}`. A push token reads the name into `pending_name`, which the group takes
  when `_group_open_tag_push` pushes it; a pop token checks the written name, or its absence, against the
  innermost open group, the only one the grammar lets it close, and `_group_close` then pops it.
- `verbatim_content`, `comment_content` — raw text up to the matching close tag.
- `_tag_open`, `_bt_option`, `_kwarg_name` — what a tag has already been given, so that a repeated
  argument is refused: `_bt_option` for `{% blocktranslate %}`'s five options, `_kwarg_name` for `name=value`
  arguments — those of a tag registered with `simple_tag`, and the `with` bindings of `{% with %}`,
  `{% include %}` and `{% blocktranslate %}` — whose names are open-ended and so are kept as names rather
  than as a mask. `_kwarg_name` returns nothing unless the name is followed by `=`, which
  is also what tells an argument apart from the `as name` clause or the end of the tag. `_tag_open` clears
  both, and every rule that uses either has one. Both are zero width and hidden, so no tree changes. The state is one byte of
  flags, not a stack, because the tag cannot nest: its body admits no tags. It lives outside the
  `stack`/`error` union, is cleared in both branches of `reset_scanner`, and is serialized after the
  status character, so a GLR stack split copies it like everything else.
- `_group_open_tag_read`, `_group_open_tag_push`, `_group_follow`, `_group_repeat`, `_group_close`, `_group_held` —
  the stack
  of open groups, which the scanner keeps **without knowing any group by name**, so a grammar extending this
  one gets it for its own groups. The read token sits right after `{%` in every tag, always returns, and
  reads the tag's name into `read_word`. The other four sit right before `%}` and check that it follows:
  push opens a group expecting `"end"` + `read_word` (every Django end tag is spelled that way), follow
  records a middle tag the innermost group may hold only once, repeat records nothing, and close pops the
  innermost group once `read_word` is what it expects. `_group_held` belongs to `unexpected_tag` (see the load section) and returns only when
  the innermost group already holds a tag named `read_word`. Reading `count` (`_bt_option`) marks the group about
  to be pushed as waiting for `plural`, which its follow clears. An entry is
  `{expected, name, middles, awaits_plural}`, serialized as `expected:name:middles:flag` and a newline per
  entry; entries hold whole names, so serializing checks `TREE_SITTER_SERIALIZATION_BUFFER_SIZE` and goes to
  the error state rather than overrun it. Push and follow before `%}` cost 247 states over the table-driven
  scanner (3,057 to 3,304), because argument-final states stop being shared with tags where the token is not
  valid; after `%}`, where each tag's states are its own, saved only 15.
- `_missing_tag` — the marker where a group's required tag block is missing, aliased at each use to
  `missing_endX_tag` or `missing_plural_tag` (`MISSING_TAGS`). It is placed only at the end of input;
  before the end tag of a group enclosing the innermost one; before an `endblock`/`endpartialdef` naming a
  block further out; or before a second `plural`, since a blocktranslate body holds no tag that could take it.
  Any other tag is taken to be the innermost group's, as Django's parser takes it — a middle tag it has not
  held yet, and a second of one it holds only once, which is an `unexpected_tag` inside it. Marking the
  group closed before a second `else` was tried and dropped: inside `{% block %}`, where most template code
  sits, there is always an enclosing group to hand the `else` to, so the inner group's own end tag became a
  stray instead. A counted blocktranslate is missing its
  plural before anything but `plural`. The marker closes the group as its end tag would (or records the
  plural), the group reduces with no `ERROR`, and `queries/errors.scm` reports it. Tags in a raw body are
  not read: the marker tells the body is raw from `verbatim_content`/`comment_content` being valid, and only
  the end of input counts there. **It is one token aliased per group, never a token per group**: tree-sitter
  does not merge parse states whose valid external tokens differ, so a token per group cloned every body
  state for each kind of group and took the table from 1,818 states to 17,664. A supertype would be the
  natural way to match every marker, but the generator silently drops one whose members are aliases, so
  `errors.scm` lists them.

  What a stack that knows no group cannot tell: a middle tag of an enclosing group that the innermost group
  has never held — `{% if a %}{% for x in y %}{% else %}` — is an `ERROR`, not a marker. Learning each
  group's middle tags as they appear was tried first and put false markers on valid nesting such as
  `{% if %}…{% else %}{% if %}…{% else %}`, because the outer group, having held an `else`, claimed the
  inner one's.

- `_text_brace` — a single `{` that is text although the content regex cannot tell: before `{%` or `{#`,
  and at the end of input. Django splits a template with `({%.*?%}|{{.*?}}|{#.*?#})`, so a `{` is text
  unless one of those starts at it; `{{{` is not a text brace, since `{{` starts a variable at its first
  brace. `content` is a right-associative run of `_text` and `_text_brace`, so it stays one node. Inside a
  group's body the missing-tag check reads past `{` first, so it scans the brace itself
  (`scan_text_brace_rest`) rather than give up the position.
- `_missing_variable_close` — the marker where a variable's `}}` is missing, aliased to
  `missing_variable_close`. It is placed only where the variable cannot go on — at the end of input, or
  before a `}` that is not `}}`, a `%` or a `{` — so `{{ hello }{% if x %}` closes the variable, reads the
  stray `}` as text and parses the `if` normally, where error recovery used to take the rest of the file.
  It is valid only after a variable's value, never where `content` is. It added 18 states.
- `matcher_error` — used by no rule; returned only if the scanner reaches an error state.

`check_space()` defines whitespace for the scanner and **must stay in sync with `_sep`**, because the
scanner skips whitespace itself while matching names.

Prefer the grammar over the scanner. A useful test: does the constraint change how the input _parses_?
Block-name matching does — it decides which close tag closes what.

"This keyword may not repeat" normally does not, and belongs in a linter. `blocktranslate` is the exception,
and the reason is worth keeping straight: its five options are a fixed set, so uniqueness _is_ expressible —
`arrangements` expresses exactly that — and the scanner is standing in for a grammar that would be correct
but takes minutes to build. Where the set is open, as with `{% querystring %}`'s `**kwargs`, no grammar
can express it at all, and the scanner remembers the names the tag has been given instead of a mask of
known ones. Only `parse_bits` actually refuses a repeat: the `with` bindings of
`{% with %}`, `{% include %}` and `{% blocktranslate %}` all go through `token_kwargs`, which builds a dict
and lets the later value win. They are guarded here anyway, as a deliberate divergence — writing a name
twice has no use and is a mistake worth reporting. `{% blocktranslate %}`'s `count` is not part of that,
because Django keeps it in a dict of its own, so `{% blocktranslate with a=1 count a=2 %}` is accepted here
as it is there.

Zero-width guards carry one hazard: `recover_with_missing` can supply one without the scanner running. The
`if (valid_symbols[MatcherError]) return false;` near the top of `scan` is what stops a guard being scanned
during tree-sitter's mark-everything-valid recovery pass, so new guards go **below** it — and above the
whitespace loop, which skips the separator the grammar still has to match. A guard must also never be valid
where `content` is: a zero-width token at a content boundary preempts the internal lexer and `content` stops
matching. `_missing_tag` is the one exception, and is safe only because it returns a token solely at the end
of input or at `{%`, where `content` cannot match. It is scanned above the raw-text scanners, which find
nothing at the end of an empty body.

The group tokens were checked against that hazard with malformed groups — a missing predicate, junk before
`%}`, in an end tag, in `{% else %}` and in `{% plural %}`, and a mismatched block name. Recovery never
inserted one of them as `MISSING`, and the stack never fell out of step with the parser: later groups still
got the right markers. Junk inside an end tag does leave the whole file an `ERROR`.

### Queries

`queries/highlights.scm` lists tag and filter names explicitly, so adding either to the grammar means
adding it there too; filter names appear there without their colon. A name the grammar does not know is
captured through its field instead (`(filter filter_name: (identifier))`, `(custom_tag tag_name: (identifier))`).

`queries/libraries.scm` is not a query editors run themselves; it is data for a linter, and every tag or
filter added from a library belongs in it. `queries/locals.scm` relies on the `variable:` field to tell a
binding from a reference — another reason to keep `asVariable` uniform. `{% blocktranslate %}`'s `asvar`
target carries an `asvar:` field instead, because it is written inside the tag but assigned outside it:
the scope on `blocktrans_block` / `blocktranslate_block` confines the `with` and `count` bindings, and would swallow `asvar`
too if it were spelled the same way. `queries/exports.scm` names it for a tool to
bind in the scope around the block, which a locals query cannot do: it places a definition in the innermost
scope containing it and offers no way out again.

`queries/conditionals.scm` is data for a tool too: `@conditional.group` / `@conditional.branch` /
`@conditional.default` say which bodies render on only some passes, so a checker can tell that
`{% if a %}{% now "Y" as n %}{% endif %}{{ n }}` may reach `{{ n }}` with nothing bound. A locals query
cannot: it matches a reference to the definition whose scope contains it and has no notion of a path not
taken. A group is exhaustive iff it has a `@conditional.default`, so each pattern captures the group
alongside one of its parts and a tool needs no list of which block belongs to which group. Adding a tag
group whose body may be skipped means adding it there — `{% cache %}` is the non-obvious one, since a
cache hit skips the body outright and `CacheNode` pushes no context.

A branch that is also a `@local.scope` confines its bindings whichever way the branch goes, which is why
`{% for %}` is inert for that check: `ForNode` renders `nodelist_empty` **inside** the same
`context.push()` as the loop body, so `empty_block` is a scope alongside `for_block`.

`queries/errors.scm` collects what an editor should report: `(ERROR)` as `@error.syntax`, `(MISSING)` as
`@error.missing`, every missing-tag marker as `@error.missing_tag`, `unexpected_tag` and
`unexpected_argument` as `@error.unexpected_tag` and `@error.unexpected_argument`, and
`missing_variable_close` as `@error.missing_variable_close`. A tree holding only markers has
no `has_error`, so a tool that checks that flag alone misses them. Adding a tag group means adding its
marker there, since no supertype matches them all.

## Testing

Corpus tests live in `test/corpus/*.txt`, one file per tag, and `missing.txt` for the missing-tag markers.

- A test whose header carries `:error` asserts only that the parse fails. Prefer it for syntax-error cases
  over pinning an error-recovery tree, which is brittle.
- An omitted field annotation in an expected tree is treated as "don't care"; `--show-fields` forces fields
  into diffs.
- `test/highlights/` is **not** run by `tree-sitter test` (which looks for `test/highlight/`), and its
  `example.html` uses a custom `{% greet %}` tag the grammar does not model, so it parses with errors by
  design.

### Checking against Django itself

`Pipfile.lock` pins Django (`Pipfile` itself says `django = "*"`) so the grammar can be diffed against the
real implementation instead of against documentation or memory. Both checks are worth repeating whenever tags, filters or literals change:

- **Builtin parity** — enumerate `Engine.default_builtins`, then compare `register.tags` and
  `register.filters` against `grammar.js`. For filters also compare argument arity via
  `inspect.signature`, ignoring the framework-injected `autoescape` parameter, which is never written in a
  template. Note `querystring` and `csp_nonce_attr` register as `simple_tag`, not `@register.tag`.
- **Library contents** — `django.templatetags.<name>.register` gives a library's `tags` and `filters`.
  All of them are registered with `@register.tag`, so `inspect.signature` only reports `(parser, token)`
  and a signature has to come from the differential below or from reading the compile function.
- **Syntax differential** — run candidate snippets through `django.template.Template()` and through
  `tree-sitter parse`, and compare accept/reject. This is how the `simple_tag` argument rules, the `_()`
  literal, and the filter-pipe spacing were pinned down.

`pipenv run` has resolved to a _different_ project's virtualenv from this directory; confirm with
`python -c "import django; print(django.get_version())"` and fall back to the explicit venv path.

### Deliberate divergences from Django

`README.md` carries a shorter list of its own, and the two are not the same set on purpose: the README
lists only the divergences where Django accepts an input that is almost certainly a mistake, which is the
rule it states. Anything added here that fits that rule belongs there too; anything that does not — an
unavoidable limitation, or a case where this grammar is the more permissive one — stays here.

- Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal,
  so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here. `1.` is the
  one that needs care: `Variable` calls `float()` and then rejects what it parsed if the last character is
  a dot (`# "2." is invalid`), so a trailing dot is a number only with an exponent after it — `1.e5` is a
  float and `1.` is not. The `number` regex is spelled to say that, and `test/corpus/template_variable.txt`
  pins all three.
- Django has no boolean or `None` token either. `True`, `False` and `None` are lookups, which resolve
  because `BaseContext._reset_dicts` seeds every context with `{"True": True, "False": False, "None": None}`.
  They are `boolean` and `none` literals here, so that a tool reading `queries/locals.scm` does not count
  them as references to something the template never defines. Shadowing still parses — `{% with True=1 %}`
  binds an `identifier`, because the keyword is not valid where a binding name is — but `{{ True.x }}` is
  an error here and an ordinary two-part lookup in Django, which renders `string_if_invalid`.
- A `with` binding written twice (`{% with a=1 a=2 %}`, and the same in `{% include %}` and
  `{% blocktranslate %}`) is reported here, as an `unexpected_argument`, and not in Django, where
  `token_kwargs` builds a dict and the later value wins. It has no use, so it is treated as the mistake it almost certainly is.
- `{% csp_nonce_attr "a" media="b" %}` is accepted, matching Django, which only rejects it at render time.
- A misspelled tag or filter cannot be caught: `{{ x|lenght }}` is indistinguishable from a filter some
  library registered. Cross-referencing names against `{% load %}` needs the project's Python, so it
  belongs in a linter; the `load_tag` rule already parses the library and `from` names for one to use.
- `{% mytag %}…{% endmytag %}` parses as two sibling `custom_tag`s. Nesting them would mean guessing that
  an unknown tag is a block tag.
- A custom tag's contents are only assumed to be the `simple_tag` shape. `@register.tag` receives the raw
  token and may parse anything, so `{% mytag <<>> %}` is rejected here and not by Django.
- A library may override a builtin (`Parser.add_library` is `self.filters.update(...)`), so a project that
  redefines `length` with a different arity gets a false error here.
- Django's `static` tags ignore what follows their argument: `StaticNode.handle_token` looks for `as` two
  bits from the end, so `{% static "a" junk %}` and `{% static "a" as u v %}` are accepted there and
  reported here as an `unexpected_argument`, and `PrefixNode.handle_token` raises `IndexError` rather than `TemplateSyntaxError` on
  `{% get_static_prefix as %}`.
- Django's `{% cache %}` takes its fragment name as the raw word in that position and finds the cache to
  use by testing whether the last word starts with `using=`, and only when something precedes it. So
  `{% cache 500 sidebar.name %}` names a fragment there, `{% cache 500 using="c" %}` names a fragment
  called `using="c"` rather than choosing a cache, and `{% cache 500 sidebar using= %}` takes an empty
  name. All three are errors here.
- `partial`/`partialdef` are Django builtins as of Django 6; `elif`/`else`/`empty` are modelled as parts of
  their enclosing tag rather than as separate tags.
- An unterminated group parses here and raises `TemplateSyntaxError` in Django: `{% if a %}x`, or a
  `{% for %}` closed while an `{% if %}` inside it is still open, holds a `missing_endif_tag` marker
  where the end tag belongs. The marker is the error, reported through `queries/errors.scm`; a check
  for `ERROR` nodes alone accepts the tree.
- A `{{` that does not close is text in Django, whose lexer only takes a variable written `{{ … }}` on one
  line, so `{{ hello }` renders as written. Here it is a `template_variable` closed by a
  `missing_variable_close` marker, reported through `queries/errors.scm`, since a half-written variable is
  what it almost always is. The same lexer makes `{{ hello\n}}` text too, which this grammar accepts as a
  variable, because its separators allow newlines.