tree-sitter-django/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

A tree-sitter grammar for the Django Template Language: {% %} tags and {{ }} variables.

Commands

npm run parser-generate   # regenerate src/parser.c, grammar.json, node-types.json from grammar.js
npm run parser-test       # run the corpus tests
npm run parser-build      # build tree-sitter-django.wasm
npm run playground        # explore the grammar interactively
npm install               # also builds the Node binding, through node-gyp-build
npm run binding-test      # the Node binding's own test
cargo test                # the Rust binding
make                      # the C library, and make clean to remove it

npx tree-sitter test -i "Cycle With As"      # one test, by name regex
npx tree-sitter test --file-name cycle.txt   # one corpus file
npx tree-sitter test -r                      # force a rebuild of the parser library
npx tree-sitter parse file.html              # parse one file
npx tree-sitter parse --debug normal f.html  # lex/shift/reduce trace
npx prettier --write grammar.js queries test CLAUDE.md   # not `.` — see below

src/parser.c, src/grammar.json and src/node-types.json are generated and committed, so run parser-generate after every change to grammar.js or src/scanner.c.

Format with prettier over specific paths rather than npm run format (prettier --write .): that also rewrites the generated src/grammar.json and src/node-types.json, which parser-generate then writes back, producing churn in both directions.

npm run lint does not work: eslint.config.mjs was copied from a Next.js project and enables @typescript-eslint and react-hooks rules whose plugins aren't installed. Formatting is prettier's job.

Bindings

bindings/ and the manifests beside it — binding.gyp, Cargo.toml, pyproject.toml, setup.py, go.mod, Package.swift, Makefile, CMakeLists.txt — are generated by tree-sitter init from the bindings and metadata sections of tree-sitter.json, and .gitattributes marks them linguist-generated. Do not edit them by hand: change tree-sitter.json and regenerate.

tree-sitter init --update only rewrites what it considers outdated, and a changed version or description does not count, so the manifests keep the old values until they are deleted and init is run again. It leaves README.md, package.json and .gitignore alone, which is why the Node binding's package.json entries — main/types, the . export, files, node-addon-api, node-gyp-build, the install script and the optional tree-sitter peer — are maintained here rather than generated.

Every binding was built and run on macOS: npm install plus npm run binding-test, cargo test, go test ./bindings/go/, pip install . with a parse, swift build, and make. The Go binding needs a go.sum, which go mod tidy writes and which is committed with the rest.

src/scanner.c compiles without warnings, and should stay that way: cc and clang warn about && within || (-Wlogical-op-parentheses), so a comparison chain written with both needs its && groups parenthesized.

CI and releasing

.github/workflows/ci.yml runs on pull requests, and the release workflow calls it: the corpus tests, a check that npm run parser-generate leaves src/ unchanged, the Node binding on Linux, macOS and Windows, and the Rust, Python, Go and Swift bindings.

.github/workflows/release.yml runs on every push to main but only releases when package.json's version differs from the previous commit's and no v<version> tag exists yet. Then it runs CI, prebuildify on five runners (Linux x64 and arm64, macOS x64 and arm64, Windows x64), and a last job that tags v<version>, creates the GitHub release with tree-sitter-django.wasm attached, and runs npm stage publish --provenance. A release is a version-bump commit on main; no other push starts one.

The workflow stages the publish; it does not publish. npm stage publish hands npm the tarball and holds it there, unavailable to install, until a maintainer runs npm stage approve <stage-id> and answers a 2FA prompt — proof of presence deferred rather than dropped, which is npm's answer to deprecating the tokens that let a workflow publish unattended. The job summary prints the view/download/approve/reject commands for the stage it made. Approving needs npm 12 (npm install -g npm@^12), which is also why the job installs it: no runner image ships an npm that has the command.

Running the workflow by hand (workflow_dispatch) is the retry: it stages whatever version package.json holds as long as npm view does not find it on the registry already, and it skips the tag and the release when a previous attempt left them behind, replacing only the Wasm asset. So a run that tagged and then failed to stage is finished by running it again, and a run against an already published version stops at the version job. A version that is staged and not yet approved is invisible to npm view while still occupying npm's version index, so a second stage of it is an error rather than a no-op; the stage job asks npm stage list first and skips straight to the approval summary when it finds one. That listing is best-effort — a short-lived OIDC token may not run npm stage subcommands at all — and a failure there just means the stage is attempted.

Releasing needs three things to line up:

• credentials to stage with: either a trusted publisher configured for this workflow (npm trust github --allow-stage-publish), which needs no secret at all because npm exchanges the workflow's OIDC token for one of its own, or the NPM_TOKEN secret as the fallback. Staging works with any token type, so that token no longer needs — and should no longer have — the 2FA bypass an automation token carries. Both the exchange and provenance need id-token: write, which the publish job already declares.
• the version in package.json and in tree-sitter.json's metadata. The release trigger reads the first; the second is compiled into src/parser.c as the language's version, so bumping one alone leaves the parser disagreeing with the manifest.
• npm run parser-generate after that bump, committed with it, or CI's up-to-date check fails on a two-line diff in src/parser.c.

The arm64 Linux runner (ubuntu-24.04-arm) is free for public repositories only. tree-sitter build --wasm runs Emscripten through Docker, which the ubuntu runners provide.

The npm package must carry its prebuilds, or npm install runs node-gyp-build, finds nothing and compiles, which fails on an image without python3, make and a compiler — node:20-slim, for instance. 0.1.0 was published by hand before the workflow existed and shipped none, which is what that image hit. The prebuild jobs are what produce them, and files carries prebuilds/**.

npm run prebuildify passes --napi --strip --tag-libc. --napi makes one binary per platform that every Node ABI can load, so the file needs no ABI tag and is named after the package. --tag-libc adds glibc or musl: node-gyp-build treats a file with no libc tag as matching anything, so an Alpine install would otherwise load a glibc binary and fail at run time; with the tag it compiles instead. Only glibc is built, so musl installs need a toolchain.

package.json exports the Wasm build as tree-sitter-django/tree-sitter-django.wasm. An exports map hides everything it does not name, so each published path — the queries, grammar.js, src/scanner.c — needs its own entry.

Stale parser library

tree-sitter caches the compiled parser in ~/.cache/tree-sitter/lib/django.*, and it has served a stale build after edits — producing parses that contradict the current grammar (debug traces naming tokens that no longer exist). If a parse result disagrees with grammar.js, suspect this first: rerun with npx tree-sitter test -r, or delete ~/.cache/tree-sitter/lib/django*.

Architecture

Whitespace is syntax, not extras

extras: []. Django's Token.split_contents() splits a tag's contents on whitespace before parsing any argument, so {% cycle 1as x %} is the single token 1as — an error, not 1 followed by as. extras can never be made mandatory, so the separator is an explicit token and every rule is built from helpers that place it:

• SEP — sym("_sep"), the hidden rule _sep: /[ \t\r\n]+/, which a grammar extending this one writes as $._sep. Newlines included: Django matches tags with re.DOTALL, so tags may span lines.
• part(...) — one tag part, taking the whitespace that separates it from the part before.
• joined(first, ...rest) — parts separated from each other, with no leading separator (not in).
• tag(name, ...args) — {%, optional separator, tag name, args, optional separator, %}; tags may hug their delimiters ({%cycle 1%}).
• asVariable($) — the as name clause. Every tag that takes one names the target field("variable", …); do not invent a second name for it.
• simpleTag($, name, args, kwargs) — a tag registered with Django's simple_tag, which always accepts positional filter expressions, then name=value keywords, then as name. args: true (*args), a count (at most N), or false. kwargs: true (**kwargs), false, or {name: rule} for a known signature. Put the Python signature in a comment above the call site.
• arrangements(items) — every ordered subset of items, so that a tag's options may be written in any order but never twice. Used by translate_tag and include_tag; keyword arguments are guarded by the scanner instead. Grows as Σ C(n,k)·k! (65 for four names), which is why it is only used for signatures that name their arguments.
• wordAsValueRules(word) / wordAsValue($, word, continued) — a tag's own keyword written where a value goes. Django tells its flag from a variable by comparing the whole bit, so reversed is the flag of {% for %} while reversed.x and reversed|last are lookups; continued selects that second group. Needed because keyword extraction hands the parser the keyword wherever both it and identifier are valid, so the reading in which the word names a variable has to be built back out of it. The rules are hidden and aliased at the use site, one per level (variable_attribute, value, filtered_value), because an alias renames the node its own rule makes and cannot nest one inside another — writing it as nested inline alias(alias(...)) calls silently yields a flat, partly zero-width tree that still generates and still parses. Check the tree, not just the accept.

Consequences worth knowing before editing a rule:

• No token.immediate anywhere — without extras, adjacency is already the default. Use it only to forbid whitespace somewhere a separator is otherwise allowed.
• Every optional trailing part needs somewhere for its separator to go. Getting this wrong does not fail at generate time; it shows up as MISSING nodes in a passing-looking parse.
• conflicts entries are the normal fix, not precedence. With explicit separators the lookahead at a part boundary is SEP instead of the meaningful token, so LR(1) loses its discriminator. That is why predicate, binaryOperator and [$.library, $.load] are listed there. Where two readings both stay valid to the end — is not as one operator or as is and a prefix not — the conflict needs a prec.dynamic alongside it to settle which wins; a static prec resolves the shift/reduce while the table is built and discards the other parse outright, which is how a bare {% if x is y %} came to be an error.

A tag group is a sequence of blocks

Every group is split so that each tag that opens a body owns a node of its own:

if_tag: ($) => startTag("if", part($.predicate)),
if_block: ($) => seq($.if_tag, optional($.template)),
endif_tag: ($) => endTag("endif"),
if_group: ($) =>
  seq(
    $.if_block,
    repeat($.elif_block),
    optional($.else_block),
    choice($.endif_tag, alias($._missing_tag, $.missing_endif_tag)),
  ),

A block is complete without the closing tag, and that is the point. While a template is being edited the closer usually is not there yet, and a group rule cannot be reduced without it — so {% if a %}{% x %} used to produce a top-level ERROR with the body's template parented to nothing that says which tag opened it. A block reduces anyway, so it survives error recovery and tooling can walk up from the cursor to find the enclosing tag. That is what makes completion of {% elif %}, {% else %}, {% empty %} and the end tags possible.

Every {% %} is a node of its own, named <name>_tag after the tag's name as Django spells it: if_tag, elif_tag, endif_tag, block_tag, templatetag_tag, custom_tag. A block is its opening tag and a body; a group is its blocks and its end tag. A tag ends at %} and nothing can extend it, so it reduces on whatever token follows — the end of input included — and a half-written template still holds the last {% for … %} or {% endif %} typed as a whole node, not as loose tokens. Stripping _tag gives the tag name; a new tag, or a tag group's new part, follows the same scheme.

A group's last child is its end block or, where the end tag is missing, a zero-width marker named for it (missing_endif_tag; see the scanner section). So an unterminated group still reduces to a group node, and a group's tag written where no open group can hold it is an unexpected_tag (see the load section below), so only a body the grammar cannot read produces an ERROR.

Every tag is built from ten inline utility rules — simpleTagOpen/simpleTagClose, startTagOpen/startTagClose, followTagOpen/followTagClose, repeatTagOpen/repeatTagClose, endTagOpen/endTagClose — through the helpers tag (a tag outside any group, from simpleTagOpen/simpleTagClose), startTag, followTag (a middle tag a group holds at most once: else, empty, plural), repeatTag (one it may repeat: elif) and endTag. They are inline rules rather than JS helpers so that a grammar extending this one can use them. Every tag must begin with one of the opens, one an extending grammar adds included: the token they read the tag's name with is valid wherever a tag's name is and the scanner always returns it, so a tag written from a bare "{%" never matches: its name is read as a custom_tag instead, with no error to show for it. Choose followTag or repeatTag by what the grammar allows, because the scanner trusts it: a tag marked once-only that can in fact repeat is an unexpected_tag from its second use on, and one marked repeatable that cannot is never flagged when repeated.

The names themselves need no hand-maintained list either: in the generated src/node-types.json, a group's children are its block types and its end tag, and a block's children include its opening tag, so the candidates for a group are its end tag plus the opening tag of each block it can hold. For if_group that is endif_tag from the group and if_tag, elif_tag, else_tag from its three blocks. Going the other way — from a block the cursor sits in to the groups that can hold it — is the same table read backwards, which is what a block needs when error recovery has left it without its group.

Do not reach for lookahead to solve that instead: ts_language_next_state follows only the shift on {%, which lands in the "a tag starts here" state no matter how the tree is shaped.

else_block is shared by if_group and ifchanged_group. comment_block and verbatim_block hold raw scanner text that admits no tags. blocktranslate_group has one block per spelling, blocktrans_block and blocktranslate_block, because each must be closed by its own end tag; the counted form nests a plural_block inside it, since {% plural %} is only valid after count. The block has to know which form its opening tag was, so blocktransRules builds each as a hidden rule (_blocktrans_tag, _blocktrans_count_tag) and aliases both to the one visible blocktrans_tag where the block uses it. Their bodies are _translate_body, not template, which is why these four blocks need no conflicts entry — the rule below is about a body that can itself start with {%.

Every block needs a conflicts entry. The generator offers a left associativity instead; taking it is what broke {% for %} in 06aa870, where prec.left resolved the shift/reduce on {% statically and silently discarded the parse in which the body continues, so every tag inside a loop became an ERROR.

Sharing the blocks also made the table markedly smaller — 7,200 states and a 515KB library became 5,150 and 383KB — for the same reason hoisting blocktranslate's pieces into hidden rules did: the automaton stops duplicating the body states in every group's inline context. Giving every {% %} its own _tag rule shrank it again, from 1,999 states to 1,818, by the same mechanism. The group stack's tokens (see the scanner section) cost 247, to 3,304.

Filter pipes differ by context

Django parses everything between {{ and }} — and the entire remainder of {% filter %} — as one FilterExpression, whose regex allows \s* on both sides of |. Every other tag splits its arguments on whitespace first, so a pipe there must be tight. Hence _filtered_value_spaced / _filter_expression_spaced, aliased back to the plain node names so the tree shape stays identical.

Names a {% load %} brought in

A library's tags and filters are registered in Python, so the grammar cannot know their names, their arity, or whether a tag is a block tag. Two fallback rules accept them:

• custom_tag — simpleTag($, $.identifier), i.e. what simple_tag/inclusion_tag accept, which is how a tag is registered unless it needs the parser itself.
• _custom_filter, the last alternative of filter, whose name is $.identifier rather than one of the builtin literals. Hidden, so the tree is what an inline alternative would give, but named so that a grammar extending this one can filter it out of filter's members the way it can custom_tag — see the extension section in README.md.

A name that only exists inside a tag group has to be reserved instead. endif, else, empty and the rest are tokens only in the state their group opens, so where a tag is named the lexer reads them as an identifier and custom_tag takes them: {% endif %} on its own parsed as a tag from some library. They are listed in NAMES_INSIDE_A_TAG_GROUP and applied through the tag_name reserved context, which wraps custom_tag's name and nothing else — a reserved context replaces the global one inside whatever it wraps, so wrapping the whole rule would un-reserve as and reject {% mytag endif %}. Adding a tag group with a new part or end tag means adding its name there too.

The same names are what unexpected_tag accepts. Once reserved, a group's tag written where no open group can hold it — {% endif %} with nothing open, {% else %} in a {% for %} — used to be an ERROR, and error recovery placed it badly: every tag starts with {% and the read token, which the parser has already accepted when the name turns out to be wrong, so recovery rewinds to just after them and folds the stray tag into whichever tag comes next, which then holds the ERROR. unexpected_tag takes any of NAMES_INSIDE_A_TAG_GROUP with loose arguments, so there is nothing to recover; prec.dynamic(-2), or -4, keeps it from displacing a tag the group does take, and queries/errors.scm reports it as @error.unexpected_tag. It added 31 states and no conflicts entry.

Words a tag does not take are an unexpected_argument inside it. Every close utility accepts one before its scanner token, so {% else x %} stays the else of its group, {% endif junk %} still closes its if (it used to leave the whole file an ERROR), and {% static "a.css" "b.css" %} is a static_tag. The node is a run of hidden _unexpected_words, and three details make it work:

• Lexical precedence -1. A loose word is token(prec(-1, /(?:[^\s%}]|\}[^\s%}])+/)), which never holds }}: lexer states are shared between parse states, so the token is tried inside {{ }} too, and after {{ x| it once swallowed the closing braces and left recovery nothing to close the variable with. Tree-sitter weighs lexical precedence before match length, so a token an argument really takes wins wherever one is valid (b|upper stays a filtered value) and a loose word only matches where nothing else can.
• Identifiers are words too. Wherever a keyword is valid — after {% for a in b, where reversed may follow, or after as, only, noop, silent — tree-sitter lexes every word as an identifier to check for the keyword, and the loose-word token never gets a turn. So a word may also be an identifier, aliased to an anonymous unexpected_word so queries/locals.scm does not count it as a reference, with whatever touches it (a=2, x.y|upper, "z"|lower:"q") as an immediate tail.
• -1 of dynamic precedence per word, not per node, so a reading that flags more words than it must always loses: {% cycle a b as x silent zqx %} flags zqx, not everything after a. unexpected_tag costs -2 or -4 so that a misused tag its group does take beats reading it as a stray.

A tag with a known keyword signature (simpleTag given names, so far only csp_nonce_attr) needs one more piece: the scanner's _kwarg_name accepts any name followed by = and commits the parser to a keyword argument before the name is checked, so {% csp_nonce_attr other="b" %} would be an ERROR. simpleTag offers _unexpected_keyword — an identifier and its tail, aliased to unexpected_argument — beside the names it knows.

A separator after any argument can now also start stray words, so this needs conflicts entries for unexpected_argument, predicate, _translate_option, and _unexpected_word with variable_attribute, resetcycle_tag and library. It took the table from 3,363 states to 4,913 and the Wasm build from 251KB to 323KB. Two consequences: a repeat the scanner refuses — {% with a=1 a=2 %}, {% blocktrans trimmed trimmed %} — is reported as an unexpected argument rather than an ERROR; and a stray reserved word is still an ERROR, since not, if, in, is, as, for and from can never be an identifier: {% trans "hi" as x as y %} errors on the second as. A tag missing a required argument, such as {% with a = 1 %}, is an ERROR too; unexpected_argument only takes words after a tag that is otherwise complete.

Where several tags share a name the group holds only once — {% if %}{% else %}{% else %}{% else %} — reading any of them as the stray parses, so the ties are broken by dynamic precedence: unexpected_tag is -2 when _group_held confirms the innermost group already holds the name and -4 otherwise, so each tag flagged before its time costs one more, and the first still opens the block however many follow.

Builtin names must stay keyword-extractable, or the fallbacks swallow them. word: $.identifier turns each builtin's name into its own token, which the lexer prefers wherever it is valid, so a builtin commits to its own alternative and its argument rules still apply. This is why a filter's name and its : are separate tokens: while the name was spelled "add:" it was not word-shaped, so a bare add lexed as an identifier and reached the fallback, losing the arity check for all 20 filters that require an argument. FILTERS_WITHOUT_ARGUMENT / FILTERS_WITH_ARGUMENT / FILTERS_WITH_OPTIONAL_ARGUMENT carry that arity, which Django itself enforces while it parses the template ("add requires 2 arguments, 1 provided").

First-party libraries

Django ships i18n, l10n, static, cache and tz as libraries rather than builtins. Their tags are modelled as ordinary tags (so far: static, l10n, tz, cache and i18n), and the grammar never requires the {% load %}, because an engine can preload a library through OPTIONS: {"builtins": [...]}, which makes {% static "a" %} valid with no load at all.

Whether a load is present is a question about the order of nodes, not about how the template parses, so it belongs in a tool. queries/libraries.scm supplies the halves — @library.<name> on each tag, and @load.library / @load.name on the loads — and the tool does the join, because tree-sitter queries have no ordering predicate to express it. What such a tool needs to know about Django's behaviour:

• Loading is positional: {% trans "x" %}{% load i18n %} is an error, because the parser adds to its tag dict as it walks tokens.
• Loading is per file: a {% load %} does not reach a template that {% extends %} or {% include %} this one.
• {% load trans from i18n %} makes only trans available, not the rest of the library.

A library's filters go in their own arity tables (L10N_FILTERS_WITHOUT_ARGUMENT, TZ_FILTERS_WITH_ARGUMENT, …) rather than into the builtin ones, so that the builtin parity check still has an exact list to compare against.

Modelling a library name makes it keyword-extracted, so it no longer reaches custom_tag. A project that registers its own tag under one of these names and a different signature therefore gets a parse error. That is accepted: these names are common enough that shadowing them is the mistake.

Adding a library tag or filter touches four files: grammar.js, queries/libraries.scm (which library it came from), queries/highlights.scm (the name, spelled out) and the table in README.md, which lists every library name the grammar models. All four are hand-maintained, and no test catches a name left out of any of them: tree-sitter test loads every .scm under queries/, so a pattern naming a node that no longer exists aborts the whole run, but a name that was never added passes 304/304 in silence, and the README nothing checks at all. The library-contents differential below, run against django.templatetags.<name>.register, is what finds a gap.

External scanner (src/scanner.c)

Twenty external tokens, for the constraints a context-free grammar cannot express:

• push_block/push_partial/push_verbatim and the matching pop_* — the names in {% block a %}…{% endblock a %}. A push token reads the name into pending_name, which the group takes when _group_open_tag_push pushes it; a pop token checks the written name, or its absence, against the innermost open group, the only one the grammar lets it close, and _group_close then pops it.

• verbatim_content, comment_content — raw text up to the matching close tag.

• _tag_open, _bt_option, _kwarg_name — what a tag has already been given, so that a repeated argument is refused: _bt_option for {% blocktranslate %}'s five options, _kwarg_name for name=value arguments — those of a tag registered with simple_tag, and the with bindings of {% with %}, {% include %} and {% blocktranslate %} — whose names are open-ended and so are kept as names rather than as a mask. _kwarg_name returns nothing unless the name is followed by =, which is also what tells an argument apart from the as name clause or the end of the tag. _tag_open clears both, and every rule that uses either has one. Both are zero width and hidden, so no tree changes. The state is one byte of flags, not a stack, because the tag cannot nest: its body admits no tags. It lives outside the stack/error union, is cleared in both branches of reset_scanner, and is serialized after the status character, so a GLR stack split copies it like everything else.

• _group_open_tag_read, _group_open_tag_push, _group_follow, _group_repeat, _group_close, _group_held — the stack of open groups, which the scanner keeps without knowing any group by name, so a grammar extending this one gets it for its own groups. The read token sits right after {% in every tag, always returns, and reads the tag's name into read_word. The other four sit right before %} and check that it follows: push opens a group expecting "end" + read_word (every Django end tag is spelled that way), follow records a middle tag the innermost group may hold only once, repeat records nothing, and close pops the innermost group once read_word is what it expects. _group_held belongs to unexpected_tag (see the load section) and returns only when the innermost group already holds a tag named read_word. Reading count (_bt_option) marks the group about to be pushed as waiting for plural, which its follow clears. An entry is {expected, name, middles, awaits_plural}, serialized as expected:name:middles:flag and a newline per entry; entries hold whole names, so serializing checks TREE_SITTER_SERIALIZATION_BUFFER_SIZE and goes to the error state rather than overrun it. Push and follow before %} cost 247 states over the table-driven scanner (3,057 to 3,304), because argument-final states stop being shared with tags where the token is not valid; after %}, where each tag's states are its own, saved only 15.

• _missing_tag — the marker where a group's required tag block is missing, aliased at each use to missing_endX_tag or missing_plural_tag (MISSING_TAGS). It is placed only at the end of input; before the end tag of a group enclosing the innermost one; before an endblock/endpartialdef naming a block further out; or before a second plural, since a blocktranslate body holds no tag that could take it. Any other tag is taken to be the innermost group's, as Django's parser takes it — a middle tag it has not held yet, and a second of one it holds only once, which is an unexpected_tag inside it. Marking the group closed before a second else was tried and dropped: inside {% block %}, where most template code sits, there is always an enclosing group to hand the else to, so the inner group's own end tag became a stray instead. A counted blocktranslate is missing its plural before anything but plural. The marker closes the group as its end tag would (or records the plural), the group reduces with no ERROR, and queries/errors.scm reports it. Tags in a raw body are not read: the marker tells the body is raw from verbatim_content/comment_content being valid, and only the end of input counts there. It is one token aliased per group, never a token per group: tree-sitter does not merge parse states whose valid external tokens differ, so a token per group cloned every body state for each kind of group and took the table from 1,818 states to 17,664. A supertype would be the natural way to match every marker, but the generator silently drops one whose members are aliases, so errors.scm lists them.

  What a stack that knows no group cannot tell: a middle tag of an enclosing group that the innermost group has never held — {% if a %}{% for x in y %}{% else %} — is an ERROR, not a marker. Learning each group's middle tags as they appear was tried first and put false markers on valid nesting such as {% if %}…{% else %}{% if %}…{% else %}, because the outer group, having held an else, claimed the inner one's.

• _text_brace — a single { that is text although the content regex cannot tell: before {% or {#, and at the end of input. Django splits a template with ({%.*?%}|{{.*?}}|{#.*?#}), so a { is text unless one of those starts at it; {{{ is not a text brace, since {{ starts a variable at its first brace. content is a right-associative run of _text and _text_brace, so it stays one node. Inside a group's body the missing-tag check reads past { first, so it scans the brace itself (scan_text_brace_rest) rather than give up the position.

• _missing_variable_close — the marker where a variable's }} is missing, aliased to missing_variable_close. It is placed only where the variable cannot go on — at the end of input, or before a } that is not }}, a % or a { — so {{ hello }{% if x %} closes the variable, reads the stray } as text and parses the if normally, where error recovery used to take the rest of the file. It is valid only after a variable's value, never where content is. It added 18 states.

• matcher_error — used by no rule; returned only if the scanner reaches an error state.

check_space() defines whitespace for the scanner and must stay in sync with _sep, because the scanner skips whitespace itself while matching names.

Prefer the grammar over the scanner. A useful test: does the constraint change how the input parses? Block-name matching does — it decides which close tag closes what.

"This keyword may not repeat" normally does not, and belongs in a linter. blocktranslate is the exception, and the reason is worth keeping straight: its five options are a fixed set, so uniqueness is expressible — arrangements expresses exactly that — and the scanner is standing in for a grammar that would be correct but takes minutes to build. Where the set is open, as with {% querystring %}'s **kwargs, no grammar can express it at all, and the scanner remembers the names the tag has been given instead of a mask of known ones. Only parse_bits actually refuses a repeat: the with bindings of {% with %}, {% include %} and {% blocktranslate %} all go through token_kwargs, which builds a dict and lets the later value win. They are guarded here anyway, as a deliberate divergence — writing a name twice has no use and is a mistake worth reporting. {% blocktranslate %}'s count is not part of that, because Django keeps it in a dict of its own, so {% blocktranslate with a=1 count a=2 %} is accepted here as it is there.

Zero-width guards carry one hazard: recover_with_missing can supply one without the scanner running. The if (valid_symbols[MatcherError]) return false; near the top of scan is what stops a guard being scanned during tree-sitter's mark-everything-valid recovery pass, so new guards go below it — and above the whitespace loop, which skips the separator the grammar still has to match. A guard must also never be valid where content is: a zero-width token at a content boundary preempts the internal lexer and content stops matching. _missing_tag is the one exception, and is safe only because it returns a token solely at the end of input or at {%, where content cannot match. It is scanned above the raw-text scanners, which find nothing at the end of an empty body.

The group tokens were checked against that hazard with malformed groups — a missing predicate, junk before %}, in an end tag, in {% else %} and in {% plural %}, and a mismatched block name. Recovery never inserted one of them as MISSING, and the stack never fell out of step with the parser: later groups still got the right markers. Junk inside an end tag does leave the whole file an ERROR.

Queries

queries/highlights.scm lists tag and filter names explicitly, so adding either to the grammar means adding it there too; filter names appear there without their colon. A name the grammar does not know is captured through its field instead ((filter filter_name: (identifier)), (custom_tag tag_name: (identifier))).

queries/libraries.scm is not a query editors run themselves; it is data for a linter, and every tag or filter added from a library belongs in it. queries/locals.scm relies on the variable: field to tell a binding from a reference — another reason to keep asVariable uniform. {% blocktranslate %}'s asvar target carries an asvar: field instead, because it is written inside the tag but assigned outside it: the scope on blocktrans_block / blocktranslate_block confines the with and count bindings, and would swallow asvar too if it were spelled the same way. queries/exports.scm names it for a tool to bind in the scope around the block, which a locals query cannot do: it places a definition in the innermost scope containing it and offers no way out again.

queries/conditionals.scm is data for a tool too: @conditional.group / @conditional.branch / @conditional.default say which bodies render on only some passes, so a checker can tell that {% if a %}{% now "Y" as n %}{% endif %}{{ n }} may reach {{ n }} with nothing bound. A locals query cannot: it matches a reference to the definition whose scope contains it and has no notion of a path not taken. A group is exhaustive iff it has a @conditional.default, so each pattern captures the group alongside one of its parts and a tool needs no list of which block belongs to which group. Adding a tag group whose body may be skipped means adding it there — {% cache %} is the non-obvious one, since a cache hit skips the body outright and CacheNode pushes no context.

A branch that is also a @local.scope confines its bindings whichever way the branch goes, which is why {% for %} is inert for that check: ForNode renders nodelist_empty inside the same context.push() as the loop body, so empty_block is a scope alongside for_block.

queries/errors.scm collects what an editor should report: (ERROR) as @error.syntax, (MISSING) as @error.missing, every missing-tag marker as @error.missing_tag, unexpected_tag and unexpected_argument as @error.unexpected_tag and @error.unexpected_argument, and missing_variable_close as @error.missing_variable_close. A tree holding only markers has no has_error, so a tool that checks that flag alone misses them. Adding a tag group means adding its marker there, since no supertype matches them all.

Testing

Corpus tests live in test/corpus/*.txt, one file per tag, and missing.txt for the missing-tag markers.

• A test whose header carries :error asserts only that the parse fails. Prefer it for syntax-error cases over pinning an error-recovery tree, which is brittle.
• An omitted field annotation in an expected tree is treated as "don't care"; --show-fields forces fields into diffs.
• test/highlights/ is not run by tree-sitter test (which looks for test/highlight/), and its example.html uses a custom {% greet %} tag the grammar does not model, so it parses with errors by design.

Checking against Django itself

Pipfile.lock pins Django (Pipfile itself says django = "*") so the grammar can be diffed against the real implementation instead of against documentation or memory. Both checks are worth repeating whenever tags, filters or literals change:

• Builtin parity — enumerate Engine.default_builtins, then compare register.tags and register.filters against grammar.js. For filters also compare argument arity via inspect.signature, ignoring the framework-injected autoescape parameter, which is never written in a template. Note querystring and csp_nonce_attr register as simple_tag, not @register.tag.
• Library contents — django.templatetags.<name>.register gives a library's tags and filters. All of them are registered with @register.tag, so inspect.signature only reports (parser, token) and a signature has to come from the differential below or from reading the compile function.
• Syntax differential — run candidate snippets through django.template.Template() and through tree-sitter parse, and compare accept/reject. This is how the simple_tag argument rules, the _() literal, and the filter-pipe spacing were pinned down.

pipenv run has resolved to a different project's virtualenv from this directory; confirm with python -c "import django; print(django.get_version())" and fall back to the explicit venv path.

Deliberate divergences from Django

README.md carries a shorter list of its own, and the two are not the same set on purpose: the README lists only the divergences where Django accepts an input that is almost certainly a mistake, which is the rule it states. Anything added here that fits that rule belongs there too; anything that does not — an unavoidable limitation, or a case where this grammar is the more permissive one — stays here.

• Django has no number token: [\w.]+ is one lexeme and int()/float() decides whether it is a literal, so 1a, 1e, 0x1f, 1.2.3, 1__0 and 1. are variable lookups there and errors here. 1. is the one that needs care: Variable calls float() and then rejects what it parsed if the last character is a dot (# "2." is invalid), so a trailing dot is a number only with an exponent after it — 1.e5 is a float and 1. is not. The number regex is spelled to say that, and test/corpus/template_variable.txt pins all three.
• Django has no boolean or None token either. True, False and None are lookups, which resolve because BaseContext._reset_dicts seeds every context with {"True": True, "False": False, "None": None}. They are boolean and none literals here, so that a tool reading queries/locals.scm does not count them as references to something the template never defines. Shadowing still parses — {% with True=1 %} binds an identifier, because the keyword is not valid where a binding name is — but {{ True.x }} is an error here and an ordinary two-part lookup in Django, which renders string_if_invalid.
• A with binding written twice ({% with a=1 a=2 %}, and the same in {% include %} and {% blocktranslate %}) is reported here, as an unexpected_argument, and not in Django, where token_kwargs builds a dict and the later value wins. It has no use, so it is treated as the mistake it almost certainly is.
• {% csp_nonce_attr "a" media="b" %} is accepted, matching Django, which only rejects it at render time.
• A misspelled tag or filter cannot be caught: {{ x|lenght }} is indistinguishable from a filter some library registered. Cross-referencing names against {% load %} needs the project's Python, so it belongs in a linter; the load_tag rule already parses the library and from names for one to use.
• {% mytag %}…{% endmytag %} parses as two sibling custom_tags. Nesting them would mean guessing that an unknown tag is a block tag.
• A custom tag's contents are only assumed to be the simple_tag shape. @register.tag receives the raw token and may parse anything, so {% mytag <<>> %} is rejected here and not by Django.
• A library may override a builtin (Parser.add_library is self.filters.update(...)), so a project that redefines length with a different arity gets a false error here.
• Django's static tags ignore what follows their argument: StaticNode.handle_token looks for as two bits from the end, so {% static "a" junk %} and {% static "a" as u v %} are accepted there and reported here as an unexpected_argument, and PrefixNode.handle_token raises IndexError rather than TemplateSyntaxError on {% get_static_prefix as %}.
• Django's {% cache %} takes its fragment name as the raw word in that position and finds the cache to use by testing whether the last word starts with using=, and only when something precedes it. So {% cache 500 sidebar.name %} names a fragment there, {% cache 500 using="c" %} names a fragment called using="c" rather than choosing a cache, and {% cache 500 sidebar using= %} takes an empty name. All three are errors here.
• partial/partialdef are Django builtins as of Django 6; elif/else/empty are modelled as parts of their enclosing tag rather than as separate tags.
• An unterminated group parses here and raises TemplateSyntaxError in Django: {% if a %}x, or a {% for %} closed while an {% if %} inside it is still open, holds a missing_endif_tag marker where the end tag belongs. The marker is the error, reported through queries/errors.scm; a check for ERROR nodes alone accepts the tree.
• A {{ that does not close is text in Django, whose lexer only takes a variable written {{ … }} on one line, so {{ hello } renders as written. Here it is a template_variable closed by a missing_variable_close marker, reported through queries/errors.scm, since a half-written variable is what it almost always is. The same lexer makes {{ hello\n}} text too, which this grammar accepts as a variable, because its separators allow newlines.