add license, review README.md, fix found bugs

This commit is contained in:
danhanson committed 2026-09-12 16:36:46 -05:00
1 parent 9a254328f6
commit 22362ed393
7 files changed
+2133 -2025

No files matched your search

+7 -3
View File
@@ -280,8 +280,8 @@ Corpus tests live in `test/corpus/*.txt`, one file per tag.
### Checking against Django itself ### Checking against Django itself
The `Pipfile` pins Django so the grammar can be diffed against the real implementation instead of against `Pipfile.lock` pins Django (`Pipfile` itself says `django = "*"`) so the grammar can be diffed against the
documentation or memory. Both checks are worth repeating whenever tags, filters or literals change: real implementation instead of against documentation or memory. Both checks are worth repeating whenever tags, filters or literals change:
- **Builtin parity** — enumerate `Engine.default_builtins`, then compare `register.tags` and - **Builtin parity** — enumerate `Engine.default_builtins`, then compare `register.tags` and
`register.filters` against `grammar.js`. For filters also compare argument arity via `register.filters` against `grammar.js`. For filters also compare argument arity via
@@ -305,7 +305,11 @@ rule it states. Anything added here that fits that rule belongs there too; anyth
unavoidable limitation, or a case where this grammar is the more permissive one — stays here. unavoidable limitation, or a case where this grammar is the more permissive one — stays here.
- Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal, - Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal,
so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here. so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here. `1.` is the
one that needs care: `Variable` calls `float()` and then rejects what it parsed if the last character is
a dot (`# "2." is invalid`), so a trailing dot is a number only with an exponent after it — `1.e5` is a
float and `1.` is not. The `number` regex is spelled to say that, and `test/corpus/template_variable.txt`
pins all three.
- Django has no boolean or `None` token either. `True`, `False` and `None` are lookups, which resolve - Django has no boolean or `None` token either. `True`, `False` and `None` are lookups, which resolve
because `BaseContext._reset_dicts` seeds every context with `{"True": True, "False": False, "None": None}`. because `BaseContext._reset_dicts` seeds every context with `{"True": True, "False": False, "None": None}`.
They are `boolean` and `none` literals here, so that a tool reading `queries/locals.scm` does not count They are `boolean` and `none` literals here, so that a tool reading `queries/locals.scm` does not count
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Daniel Hanson
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+44 -4
View File
@@ -1,6 +1,35 @@
# tree-sitter-django # tree-sitter-django
A [tree-sitter](https://tree-sitter.github.io/tree-sitter/) grammar for the Django Template Language (DTL) — the `{% %}` tags and `{{ }}` variables used in Django templates. A [tree-sitter](https://tree-sitter.github.io/tree-sitter/) grammar for the Django Template Language (DTL) — the `{% %}` tags, `{{ }}` variables and `{# #}` comments used in Django templates.
## Coverage
Every tag and filter Django ships as a builtin is modelled — 28 tags and 57 filters as of Django 6.1 —
and so is every tag and filter of the five libraries below. Arguments are part of the grammar rather than
something a linter checks later, so a builtin written with the wrong ones is a parse error: `{{ x|length:2 }}`
and `{% now %}` do not parse, because `length` takes no argument and `now` requires one.
The target is Django 6.1, including the builtins Django 6 added (`{% partial %}`, `{% partialdef %}`,
`{% querystring %}`, `{% csp_nonce_attr %}`). Rather than being written from the documentation, the grammar
is checked against a real Django — `Pipfile.lock` pins the version — by enumerating `Engine.default_builtins`
for the tag and filter lists and their arities, and by running candidate snippets through both
`django.template.Template()` and this parser to compare what each accepts. Where the two deliberately
disagree is listed under [Divergences](#divergences-from-django).
## Using it
There is no published package yet, and no language bindings — those are coming. What is here today:
- **`src/parser.c` and `src/scanner.c` are generated and committed**, so any tree-sitter host can compile
them directly without running the CLI. The scanner is required: block-name matching
(`{% block a %}…{% endblock a %}`) and the repeated-argument guards live in it.
- **`npm run parser-build`** produces `tree-sitter-django.wasm` for a `web-tree-sitter` host or an editor
that loads Wasm grammars. It is a build artifact rather than a checked-in one, so it has to be built
(or shipped by a release) before anything can load it.
- **`queries/`** holds the highlight, locals, tags and injection queries an editor loads, plus three files
meant for a linter. See [Queries](#queries).
- **`tree-sitter.json`** names the grammar `django`, scopes it `source.django`, and points at the query
files, which is what a tree-sitter host reads to wire all of the above together.
## Development ## Development
@@ -12,6 +41,11 @@ npm run parser-build # build the WASM binary
npm run playground # interactively explore the grammar npm run playground # interactively explore the grammar
``` ```
`src/parser.c`, `src/grammar.json` and `src/node-types.json` are generated **and committed**, so a change
to `grammar.js` or `src/scanner.c` is only half made until `parser-generate` has run and the result is
committed alongside it. `tree-sitter test` compiles the parser itself, so a stale `src/` shows up as tests
that pass against the wrong grammar.
## Queries ## Queries
Four of the files in `queries/` are the standard query files an editor loads: Four of the files in `queries/` are the standard query files an editor loads:
@@ -150,7 +184,8 @@ Five things to know:
alternative and your arity applies — `{{ x|shout }}` is now an error, while names you have not modelled alternative and your arity applies — `{{ x|shout }}` is now an error, while names you have not modelled
still fall through to `(filter name: (identifier))`. still fall through to `(filter name: (identifier))`.
- **A tag with a body needs a `conflicts` entry**, as every clause in the base grammar does: - **A tag with a body needs a `conflicts` entry**, as every clause in the base grammar does:
`conflicts: ($, previous) => [...previous, [$.map_clause]]`. The generator offers an associativity `conflicts: ($, previous) => [...previous, [$.map_clause]]` for a `map_clause` holding the body. The
generator offers an associativity
instead; taking it silently discards the parse in which the body continues. instead; taking it silently discards the parse in which the body continues.
- **End tags are not reserved**, a stray `{% endmap %}` parses as a `custom_tag` rather than an error — `reserved` takes no - **End tags are not reserved**, a stray `{% endmap %}` parses as a `custom_tag` rather than an error — `reserved` takes no
`previous`, so the `tag_name` list cannot be added to a name at a time. Instead you may drop the `custom_tag` rule to avoid parsing end tags as custom tags. `previous`, so the `tag_name` list cannot be added to a name at a time. Instead you may drop the `custom_tag` rule to avoid parsing end tags as custom tags.
@@ -181,7 +216,7 @@ not want `custom_tag` and `_custom_filter` accepting unrecognized names as they
template_block_groups: ($, previous) => template_block_groups: ($, previous) =>
choice( choice(
...previous.members.filter((m) => m.name !== "custom_tag"), ...previous.members.filter((m) => m.name !== "custom_tag"),
$.map_group, $.map_tag,
), ),
filter: ($, previous) => filter: ($, previous) =>
@@ -203,7 +238,8 @@ Every case below is input Django itself accepts:
- **Number-shaped names.** Django has no number token: a word is a literal if `int()` or `float()` parses - **Number-shaped names.** Django has no number token: a word is a literal if `int()` or `float()` parses
it and a variable lookup otherwise, so `0x1f`, `1a`, `1.`, `1.2.3` and `1__0` look up a variable of that it and a variable lookup otherwise, so `0x1f`, `1a`, `1.`, `1.2.3` and `1__0` look up a variable of that
name and render nothing. name and render nothing. (`1.` really is a lookup there and not a float — `Variable` rejects a trailing
dot after parsing it — while `1.e5` is a float, and both are treated that way here.)
- **An attribute of `True`, `False` or `None`.** Every Django context is seeded with those three names, so - **An attribute of `True`, `False` or `None`.** Every Django context is seeded with those three names, so
`{{ True.x }}` is an ordinary two-part lookup that can never resolve. `{{ True.x }}` is an ordinary two-part lookup that can never resolve.
- **A keyword argument written twice.** `{% with a=1 a=2 %}`, and the same in `{% include %}` and - **A keyword argument written twice.** `{% with a=1 a=2 %}`, and the same in `{% include %}` and
@@ -217,3 +253,7 @@ Every case below is input Django itself accepts:
In the other direction the grammar is more permissive only where it cannot know better, which is what the In the other direction the grammar is more permissive only where it cannot know better, which is what the
fallbacks above are for. fallbacks above are for.
## License
MIT — see [LICENSE](LICENSE).
+10 -3
View File
@@ -395,10 +395,17 @@ const django = grammar({
value: ($) => choice($.literal, $.variable_attribute), value: ($) => choice($.literal, $.variable_attribute),
literal: ($) => literal: ($) =>
choice($.number, $.string, $.translated_string, $.boolean, $.none), choice($.number, $.string, $.translated_string, $.boolean, $.none),
// digits, with single underscores allowed between them, as Python's /**
// int() and float() accept ("1_000") and Django's Variable relies on * Digits, with single underscores allowed between them, as Python's int()
* and float() accept ("1_000") and Django's Variable relies on.
*
* A trailing dot needs an exponent after it. Variable calls float() and
* then rejects what it just parsed if the last character is a dot — the
* comment in Django reads `# "2." is invalid` — so "1." is a two-part
* lookup there rather than a number, while "1.e5" is a float.
*/
number: ($) => number: ($) =>
/[-+]?(?:[0-9](?:_?[0-9])*(?:\.(?:[0-9](?:_?[0-9])*)?)?|\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?/, /[-+]?(?:(?:[0-9](?:_?[0-9])*(?:\.[0-9](?:_?[0-9])*)?|\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?|[0-9](?:_?[0-9])*\.[eE][0-9](?:_?[0-9])*)/,
string: ($) => STRING, string: ($) => STRING,
/** /**
* A string to translate at render time. Django matches "_(" and ")" as * A string to translate at render time. Django matches "_(" and ")" as
+1 -1
View File
@@ -135,7 +135,7 @@
}, },
"number": { "number": {
"type": "PATTERN", "type": "PATTERN",
"value": "[-+]?(?:[0-9](?:_?[0-9])*(?:\\.(?:[0-9](?:_?[0-9])*)?)?|\\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?" "value": "[-+]?(?:(?:[0-9](?:_?[0-9])*(?:\\.[0-9](?:_?[0-9])*)?|\\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?|[0-9](?:_?[0-9])*\\.[eE][0-9](?:_?[0-9])*)"
}, },
"string": { "string": {
"type": "PATTERN", "type": "PATTERN",
+2013 -2014
View File
File diff suppressed because it is too large. Load diff
+37
View File
@@ -461,3 +461,40 @@ Template Variable Attribute Of A Literal
--- ---
==================
Template Variable Trailing Dot With An Exponent
==================
{{ 1.e5 }}
---
(template
(content)
(template_variable
(filtered_value
(value
(literal
(number)))))
(content))
==================
Template Variable Trailing Dot
:error
==================
{{ 1. }}
---
==================
Template Variable Signed Trailing Dot
:error
==================
{{ -1. }}
---