Compare commits

..
1 Commits
Author SHA1 Message Date
danhanson 01d0289091 add exports and library info to README.md 2026-09-11 19:09:49 -05:00
7 changed files with 39237 additions and 39552 deletions

No files matched your search

+4 -11
View File
@@ -138,10 +138,7 @@ arity, or whether a tag is a block tag. Two fallback rules accept them:
- `custom_tag` — `simpleTag($, $.identifier)`, i.e. what `simple_tag`/`inclusion_tag` accept, which is how - `custom_tag` — `simpleTag($, $.identifier)`, i.e. what `simple_tag`/`inclusion_tag` accept, which is how
a tag is registered unless it needs the parser itself. a tag is registered unless it needs the parser itself.
- `_custom_filter`, the last alternative of `filter`, whose name is `$.identifier` rather than one of the - the last alternative of `filter`, whose name is `$.identifier` rather than one of the builtin literals.
builtin literals. Hidden, so the tree is what an inline alternative would give, but named so that a
grammar extending this one can filter it out of `filter`'s `members` the way it can `custom_tag` — see
the extension section in `README.md`.
**A name that only exists inside a tag group has to be reserved instead.** `endif`, `else`, `empty` and **A name that only exists inside a tag group has to be reserved instead.** `endif`, `else`, `empty` and
the rest are tokens only in the state their group opens, so where a tag is named the lexer reads them as an the rest are tokens only in the state their group opens, so where a tag is named the lexer reads them as an
@@ -280,8 +277,8 @@ Corpus tests live in `test/corpus/*.txt`, one file per tag.
### Checking against Django itself ### Checking against Django itself
`Pipfile.lock` pins Django (`Pipfile` itself says `django = "*"`) so the grammar can be diffed against the The `Pipfile` pins Django so the grammar can be diffed against the real implementation instead of against
real implementation instead of against documentation or memory. Both checks are worth repeating whenever tags, filters or literals change: documentation or memory. Both checks are worth repeating whenever tags, filters or literals change:
- **Builtin parity** — enumerate `Engine.default_builtins`, then compare `register.tags` and - **Builtin parity** — enumerate `Engine.default_builtins`, then compare `register.tags` and
`register.filters` against `grammar.js`. For filters also compare argument arity via `register.filters` against `grammar.js`. For filters also compare argument arity via
@@ -305,11 +302,7 @@ rule it states. Anything added here that fits that rule belongs there too; anyth
unavoidable limitation, or a case where this grammar is the more permissive one — stays here. unavoidable limitation, or a case where this grammar is the more permissive one — stays here.
- Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal, - Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal,
so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here. `1.` is the so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here.
one that needs care: `Variable` calls `float()` and then rejects what it parsed if the last character is
a dot (`# "2." is invalid`), so a trailing dot is a number only with an exponent after it — `1.e5` is a
float and `1.` is not. The `number` regex is spelled to say that, and `test/corpus/template_variable.txt`
pins all three.
- Django has no boolean or `None` token either. `True`, `False` and `None` are lookups, which resolve - Django has no boolean or `None` token either. `True`, `False` and `None` are lookups, which resolve
because `BaseContext._reset_dicts` seeds every context with `{"True": True, "False": False, "None": None}`. because `BaseContext._reset_dicts` seeds every context with `{"True": True, "False": False, "None": None}`.
They are `boolean` and `none` literals here, so that a tool reading `queries/locals.scm` does not count They are `boolean` and `none` literals here, so that a tool reading `queries/locals.scm` does not count
-21
View File
@@ -1,21 +0,0 @@
MIT License
Copyright (c) 2026 Daniel Hanson
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+9 -102
View File
@@ -1,35 +1,6 @@
# tree-sitter-django # tree-sitter-django
A [tree-sitter](https://tree-sitter.github.io/tree-sitter/) grammar for the Django Template Language (DTL) — the `{% %}` tags, `{{ }}` variables and `{# #}` comments used in Django templates. A [tree-sitter](https://tree-sitter.github.io/tree-sitter/) grammar for the Django Template Language (DTL) — the `{% %}` tags and `{{ }}` variables used in Django templates.
## Coverage
Every tag and filter Django ships as a builtin is modelled — 28 tags and 57 filters as of Django 6.1 —
and so is every tag and filter of the five libraries below. Arguments are part of the grammar rather than
something a linter checks later, so a builtin written with the wrong ones is a parse error: `{{ x|length:2 }}`
and `{% now %}` do not parse, because `length` takes no argument and `now` requires one.
The target is Django 6.1, including the builtins Django 6 added (`{% partial %}`, `{% partialdef %}`,
`{% querystring %}`, `{% csp_nonce_attr %}`). Rather than being written from the documentation, the grammar
is checked against a real Django — `Pipfile.lock` pins the version — by enumerating `Engine.default_builtins`
for the tag and filter lists and their arities, and by running candidate snippets through both
`django.template.Template()` and this parser to compare what each accepts. Where the two deliberately
disagree is listed under [Divergences](#divergences-from-django).
## Using it
There is no published package yet, and no language bindings — those are coming. What is here today:
- **`src/parser.c` and `src/scanner.c` are generated and committed**, so any tree-sitter host can compile
them directly without running the CLI. The scanner is required: block-name matching
(`{% block a %}…{% endblock a %}`) and the repeated-argument guards live in it.
- **`npm run parser-build`** produces `tree-sitter-django.wasm` for a `web-tree-sitter` host or an editor
that loads Wasm grammars. It is a build artifact rather than a checked-in one, so it has to be built
(or shipped by a release) before anything can load it.
- **`queries/`** holds the highlight, locals, tags and injection queries an editor loads, plus three files
meant for a linter. See [Queries](#queries).
- **`tree-sitter.json`** names the grammar `django`, scopes it `source.django`, and points at the query
files, which is what a tree-sitter host reads to wire all of the above together.
## Development ## Development
@@ -41,45 +12,6 @@ npm run parser-build # build the WASM binary
npm run playground # interactively explore the grammar npm run playground # interactively explore the grammar
``` ```
`src/parser.c`, `src/grammar.json` and `src/node-types.json` are generated **and committed**, so a change
to `grammar.js` or `src/scanner.c` is only half made until `parser-generate` has run and the result is
committed alongside it. `tree-sitter test` compiles the parser itself, so a stale `src/` shows up as tests
that pass against the wrong grammar.
## Queries
Four of the files in `queries/` are the standard query files an editor loads:
- **`highlights.scm`** — syntax highlighting. Tag and filter names are listed one by one, so a name the
grammar knows is captured as `@function` and a name it is only tolerating as `@function.call`.
- **`locals.scm`** — scopes, definitions and references, so an editor can resolve a variable to the tag
that bound it. A definition is a `variable:` field; the scopes are the clauses whose bindings do not
outlive them (`{% with %}`, `{% for %}`, `{% empty %}`, `{% block %}`, `{% partialdef %}`,
`{% blocktranslate %}`).
- **`tags.scm`** — the symbol index behind `tree-sitter tag`. A `{% partialdef %}` and the names a
`{% load %}` brings in are definitions; every tag and filter used is a reference.
- **`injections.scm`** — marks template text and `{% verbatim %}` bodies as another language's content, so
the HTML around the tags can be parsed by its own grammar. It names no language, leaving the editor to
supply one.
The other three are data for a tool rather than queries an editor runs. Each captures the two halves of a
relation a tree-sitter query cannot express, and the tool does the join:
- **`libraries.scm`** — which library each tag or filter is registered in, and what each `{% load %}`
brings in. The grammar never requires the load, so reporting a missing one is a linter's job; a query
has no ordering predicate to compare the positions itself.
- **`exports.scm`** — a name a tag binds in the scope _around_ it rather than the scope it opens. Only
`{% blocktranslate %}`'s `asvar` does that, and a locals query cannot say so: it places a definition in
the innermost scope containing it and offers no way out again.
- **`conditionals.scm`** — which bodies render on only some passes, as `@conditional.group` /
`@conditional.branch` / `@conditional.default`, so a checker can tell that
`{% if a %}{% now "Y" as n %}{% endif %}{{ n }}` may reach `{{ n }}` with nothing bound. A group is
exhaustive exactly when one of its branches is a default. A locals query has no notion of a path not
taken.
`tree-sitter test` loads every file in the directory, so a pattern naming a node that no longer exists
fails the suite rather than quietly matching nothing.
## Django libraries ## Django libraries
Five of Django's template libraries are not builtins: their names have to be `{% load %}`ed, or preloaded Five of Django's template libraries are not builtins: their names have to be `{% load %}`ed, or preloaded
@@ -175,7 +107,7 @@ export default grammar(django, {
}); });
``` ```
Five things to know: Four things to know:
- **Whitespace is explicit.** `extras` is empty, because Django splits a tag's contents before parsing any - **Whitespace is explicit.** `extras` is empty, because Django splits a tag's contents before parsing any
argument, so every separator has to be written out. The `part`/`block`/`simpleTag` helpers the base argument, so every separator has to be written out. The `part`/`block`/`simpleTag` helpers the base
@@ -184,13 +116,13 @@ Five things to know:
alternative and your arity applies — `{{ x|shout }}` is now an error, while names you have not modelled alternative and your arity applies — `{{ x|shout }}` is now an error, while names you have not modelled
still fall through to `(filter name: (identifier))`. still fall through to `(filter name: (identifier))`.
- **A tag with a body needs a `conflicts` entry**, as every clause in the base grammar does: - **A tag with a body needs a `conflicts` entry**, as every clause in the base grammar does:
`conflicts: ($, previous) => [...previous, [$.map_clause]]` for a `map_clause` holding the body. The `conflicts: ($, previous) => [...previous, [$.map_clause]]`. The generator offers an associativity
generator offers an associativity instead; taking it silently discards the parse in which the body continues. Its end tag is not reserved
instead; taking it silently discards the parse in which the body continues. either, so a stray `{% endmap %}` parses as a `custom_tag` rather than an error — `reserved` takes no
- **End tags are not reserved**, a stray `{% endmap %}` parses as a `custom_tag` rather than an error — `reserved` takes no `previous`, so the `tag_name` list cannot be added to a name at a time.
`previous`, so the `tag_name` list cannot be added to a name at a time. Instead you may drop the `custom_tag` rule to avoid parsing end tags as custom tags.
- **The external scanner has to be re-exported under your grammar's name**, or block-name matching and the - **The external scanner has to be re-exported under your grammar's name**, or block-name matching and the
repeated-argument guards will not link. Give your grammar a `src/scanner.c` that renames the base's five entry points to the repeated-argument guards will not link. `exports` cannot deliver it — that is Node's resolver, and the
scanner is C — so give your grammar a `src/scanner.c` that renames the base's five entry points to the
ones your generated `parser.c` calls, and includes it: ones your generated `parser.c` calls, and includes it:
```c ```c
@@ -209,26 +141,6 @@ path, hence the relative `#include`; the package also exports `tree-sitter-djang
build step can hand the compiler an absolute path from `import.meta.resolve` where a package manager's build step can hand the compiler an absolute path from `import.meta.resolve` where a package manager's
layout makes the relative one unreliable. layout makes the relative one unreliable.
**Turning the fallbacks off.** A grammar that models every tag and filter its project uses probably does
not want `custom_tag` and `_custom_filter` accepting unrecognized names as they allow typos to parse. `previous` is the base rule whose `members` are filterable. We can filter out these rules to make the grammar more strict:
```js
template_block_groups: ($, previous) =>
choice(
...previous.members.filter((m) => m.name !== "custom_tag"),
$.map_tag,
),
filter: ($, previous) =>
choice(
...previous.members.filter((m) => m.name !== "_custom_filter"),
seq(field("name", "shout"), ":", field("argument", $.value)),
),
```
`{% mytag %}` and `{{ x|unknown }}` are then errors, while the builtins, the libraries and your own rules
carry on as before.
## Divergences from Django ## Divergences from Django
The grammar aims to accept what Django accepts, with one deliberate exception. Where Django takes an The grammar aims to accept what Django accepts, with one deliberate exception. Where Django takes an
@@ -238,8 +150,7 @@ Every case below is input Django itself accepts:
- **Number-shaped names.** Django has no number token: a word is a literal if `int()` or `float()` parses - **Number-shaped names.** Django has no number token: a word is a literal if `int()` or `float()` parses
it and a variable lookup otherwise, so `0x1f`, `1a`, `1.`, `1.2.3` and `1__0` look up a variable of that it and a variable lookup otherwise, so `0x1f`, `1a`, `1.`, `1.2.3` and `1__0` look up a variable of that
name and render nothing. (`1.` really is a lookup there and not a float — `Variable` rejects a trailing name and render nothing.
dot after parsing it — while `1.e5` is a float, and both are treated that way here.)
- **An attribute of `True`, `False` or `None`.** Every Django context is seeded with those three names, so - **An attribute of `True`, `False` or `None`.** Every Django context is seeded with those three names, so
`{{ True.x }}` is an ordinary two-part lookup that can never resolve. `{{ True.x }}` is an ordinary two-part lookup that can never resolve.
- **A keyword argument written twice.** `{% with a=1 a=2 %}`, and the same in `{% include %}` and - **A keyword argument written twice.** `{% with a=1 a=2 %}`, and the same in `{% include %}` and
@@ -253,7 +164,3 @@ Every case below is input Django itself accepts:
In the other direction the grammar is more permissive only where it cannot know better, which is what the In the other direction the grammar is more permissive only where it cannot know better, which is what the
fallbacks above are for. fallbacks above are for.
## License
MIT — see [LICENSE](LICENSE).
+9 -23
View File
@@ -395,17 +395,10 @@ const django = grammar({
value: ($) => choice($.literal, $.variable_attribute), value: ($) => choice($.literal, $.variable_attribute),
literal: ($) => literal: ($) =>
choice($.number, $.string, $.translated_string, $.boolean, $.none), choice($.number, $.string, $.translated_string, $.boolean, $.none),
/** // digits, with single underscores allowed between them, as Python's
* Digits, with single underscores allowed between them, as Python's int() // int() and float() accept ("1_000") and Django's Variable relies on
* and float() accept ("1_000") and Django's Variable relies on.
*
* A trailing dot needs an exponent after it. Variable calls float() and
* then rejects what it just parsed if the last character is a dot — the
* comment in Django reads `# "2." is invalid` — so "1." is a two-part
* lookup there rather than a number, while "1.e5" is a float.
*/
number: ($) => number: ($) =>
/[-+]?(?:(?:[0-9](?:_?[0-9])*(?:\.[0-9](?:_?[0-9])*)?|\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?|[0-9](?:_?[0-9])*\.[eE][0-9](?:_?[0-9])*)/, /[-+]?(?:[0-9](?:_?[0-9])*(?:\.(?:[0-9](?:_?[0-9])*)?)?|\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?/,
string: ($) => STRING, string: ($) => STRING,
/** /**
* A string to translate at render time. Django matches "_(" and ")" as * A string to translate at render time. Django matches "_(" and ")" as
@@ -556,19 +549,12 @@ const django = grammar({
field("name", choice(...FILTERS_WITH_OPTIONAL_ARGUMENT)), field("name", choice(...FILTERS_WITH_OPTIONAL_ARGUMENT)),
optional(seq(":", field("argument", $.value))), optional(seq(":", field("argument", $.value))),
), ),
$._custom_filter, // a filter a "load" brought in: its name is not known here, and so
), // neither is whether it takes an argument
/** seq(
* A filter a "load" brought in: its name is not known here, and so neither field("name", $.identifier),
* is whether it takes an argument. Hidden, so the tree is the same as if optional(seq(":", field("argument", $.value))),
* it were spelled inline, but named so that a grammar extending this one ),
* can drop it from filter's alternatives the way it can drop custom_tag
* from template_block_groups.
*/
_custom_filter: ($) =>
seq(
field("name", $.identifier),
optional(seq(":", field("argument", $.value))),
), ),
autoescape_clause: ($) => autoescape_clause: ($) =>
seq(block("autoescape", part(choice("on", "off"))), optional($.template)), seq(block("autoescape", part(choice("on", "off"))), optional($.template)),
+31 -35
View File
@@ -135,7 +135,7 @@
}, },
"number": { "number": {
"type": "PATTERN", "type": "PATTERN",
"value": "[-+]?(?:(?:[0-9](?:_?[0-9])*(?:\\.[0-9](?:_?[0-9])*)?|\\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?|[0-9](?:_?[0-9])*\\.[eE][0-9](?:_?[0-9])*)" "value": "[-+]?(?:[0-9](?:_?[0-9])*(?:\\.(?:[0-9](?:_?[0-9])*)?)?|\\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?"
}, },
"string": { "string": {
"type": "PATTERN", "type": "PATTERN",
@@ -1313,44 +1313,40 @@
] ]
}, },
{ {
"type": "SYMBOL", "type": "SEQ",
"name": "_custom_filter"
}
]
},
"_custom_filter": {
"type": "SEQ",
"members": [
{
"type": "FIELD",
"name": "name",
"content": {
"type": "SYMBOL",
"name": "identifier"
}
},
{
"type": "CHOICE",
"members": [ "members": [
{ {
"type": "SEQ", "type": "FIELD",
"members": [ "name": "name",
{ "content": {
"type": "STRING", "type": "SYMBOL",
"value": ":" "name": "identifier"
}, }
{
"type": "FIELD",
"name": "argument",
"content": {
"type": "SYMBOL",
"name": "value"
}
}
]
}, },
{ {
"type": "BLANK" "type": "CHOICE",
"members": [
{
"type": "SEQ",
"members": [
{
"type": "STRING",
"value": ":"
},
{
"type": "FIELD",
"name": "argument",
"content": {
"type": "SYMBOL",
"name": "value"
}
}
]
},
{
"type": "BLANK"
}
]
} }
] ]
} }
+39184 -39323
View File
File diff suppressed because it is too large. Load diff
-37
View File
@@ -461,40 +461,3 @@ Template Variable Attribute Of A Literal
--- ---
==================
Template Variable Trailing Dot With An Exponent
==================
{{ 1.e5 }}
---
(template
(content)
(template_variable
(filtered_value
(value
(literal
(number)))))
(content))
==================
Template Variable Trailing Dot
:error
==================
{{ 1. }}
---
==================
Template Variable Signed Trailing Dot
:error
==================
{{ -1. }}
---