add license, review README.md, fix found bugs
This commit is contained in:
danhanson committed
1 parent
9a254328f6
commit
22362ed393
7 files changed
+2133
-2025
No files matched your search
@@ -280,8 +280,8 @@ Corpus tests live in `test/corpus/*.txt`, one file per tag.
|
||||
|
||||
### Checking against Django itself
|
||||
|
||||
The `Pipfile` pins Django so the grammar can be diffed against the real implementation instead of against
|
||||
documentation or memory. Both checks are worth repeating whenever tags, filters or literals change:
|
||||
`Pipfile.lock` pins Django (`Pipfile` itself says `django = "*"`) so the grammar can be diffed against the
|
||||
real implementation instead of against documentation or memory. Both checks are worth repeating whenever tags, filters or literals change:
|
||||
|
||||
- **Builtin parity** — enumerate `Engine.default_builtins`, then compare `register.tags` and
|
||||
`register.filters` against `grammar.js`. For filters also compare argument arity via
|
||||
@@ -305,7 +305,11 @@ rule it states. Anything added here that fits that rule belongs there too; anyth
|
||||
unavoidable limitation, or a case where this grammar is the more permissive one — stays here.
|
||||
|
||||
- Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal,
|
||||
so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here.
|
||||
so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here. `1.` is the
|
||||
one that needs care: `Variable` calls `float()` and then rejects what it parsed if the last character is
|
||||
a dot (`# "2." is invalid`), so a trailing dot is a number only with an exponent after it — `1.e5` is a
|
||||
float and `1.` is not. The `number` regex is spelled to say that, and `test/corpus/template_variable.txt`
|
||||
pins all three.
|
||||
- Django has no boolean or `None` token either. `True`, `False` and `None` are lookups, which resolve
|
||||
because `BaseContext._reset_dicts` seeds every context with `{"True": True, "False": False, "None": None}`.
|
||||
They are `boolean` and `none` literals here, so that a tool reading `queries/locals.scm` does not count
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Daniel Hanson
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -1,6 +1,35 @@
|
||||
# tree-sitter-django
|
||||
|
||||
A [tree-sitter](https://tree-sitter.github.io/tree-sitter/) grammar for the Django Template Language (DTL) — the `{% %}` tags and `{{ }}` variables used in Django templates.
|
||||
A [tree-sitter](https://tree-sitter.github.io/tree-sitter/) grammar for the Django Template Language (DTL) — the `{% %}` tags, `{{ }}` variables and `{# #}` comments used in Django templates.
|
||||
|
||||
## Coverage
|
||||
|
||||
Every tag and filter Django ships as a builtin is modelled — 28 tags and 57 filters as of Django 6.1 —
|
||||
and so is every tag and filter of the five libraries below. Arguments are part of the grammar rather than
|
||||
something a linter checks later, so a builtin written with the wrong ones is a parse error: `{{ x|length:2 }}`
|
||||
and `{% now %}` do not parse, because `length` takes no argument and `now` requires one.
|
||||
|
||||
The target is Django 6.1, including the builtins Django 6 added (`{% partial %}`, `{% partialdef %}`,
|
||||
`{% querystring %}`, `{% csp_nonce_attr %}`). Rather than being written from the documentation, the grammar
|
||||
is checked against a real Django — `Pipfile.lock` pins the version — by enumerating `Engine.default_builtins`
|
||||
for the tag and filter lists and their arities, and by running candidate snippets through both
|
||||
`django.template.Template()` and this parser to compare what each accepts. Where the two deliberately
|
||||
disagree is listed under [Divergences](#divergences-from-django).
|
||||
|
||||
## Using it
|
||||
|
||||
There is no published package yet, and no language bindings — those are coming. What is here today:
|
||||
|
||||
- **`src/parser.c` and `src/scanner.c` are generated and committed**, so any tree-sitter host can compile
|
||||
them directly without running the CLI. The scanner is required: block-name matching
|
||||
(`{% block a %}…{% endblock a %}`) and the repeated-argument guards live in it.
|
||||
- **`npm run parser-build`** produces `tree-sitter-django.wasm` for a `web-tree-sitter` host or an editor
|
||||
that loads Wasm grammars. It is a build artifact rather than a checked-in one, so it has to be built
|
||||
(or shipped by a release) before anything can load it.
|
||||
- **`queries/`** holds the highlight, locals, tags and injection queries an editor loads, plus three files
|
||||
meant for a linter. See [Queries](#queries).
|
||||
- **`tree-sitter.json`** names the grammar `django`, scopes it `source.django`, and points at the query
|
||||
files, which is what a tree-sitter host reads to wire all of the above together.
|
||||
|
||||
## Development
|
||||
|
||||
@@ -12,6 +41,11 @@ npm run parser-build # build the WASM binary
|
||||
npm run playground # interactively explore the grammar
|
||||
```
|
||||
|
||||
`src/parser.c`, `src/grammar.json` and `src/node-types.json` are generated **and committed**, so a change
|
||||
to `grammar.js` or `src/scanner.c` is only half made until `parser-generate` has run and the result is
|
||||
committed alongside it. `tree-sitter test` compiles the parser itself, so a stale `src/` shows up as tests
|
||||
that pass against the wrong grammar.
|
||||
|
||||
## Queries
|
||||
|
||||
Four of the files in `queries/` are the standard query files an editor loads:
|
||||
@@ -150,7 +184,8 @@ Five things to know:
|
||||
alternative and your arity applies — `{{ x|shout }}` is now an error, while names you have not modelled
|
||||
still fall through to `(filter name: (identifier))`.
|
||||
- **A tag with a body needs a `conflicts` entry**, as every clause in the base grammar does:
|
||||
`conflicts: ($, previous) => [...previous, [$.map_clause]]`. The generator offers an associativity
|
||||
`conflicts: ($, previous) => [...previous, [$.map_clause]]` for a `map_clause` holding the body. The
|
||||
generator offers an associativity
|
||||
instead; taking it silently discards the parse in which the body continues.
|
||||
- **End tags are not reserved**, a stray `{% endmap %}` parses as a `custom_tag` rather than an error — `reserved` takes no
|
||||
`previous`, so the `tag_name` list cannot be added to a name at a time. Instead you may drop the `custom_tag` rule to avoid parsing end tags as custom tags.
|
||||
@@ -181,7 +216,7 @@ not want `custom_tag` and `_custom_filter` accepting unrecognized names as they
|
||||
template_block_groups: ($, previous) =>
|
||||
choice(
|
||||
...previous.members.filter((m) => m.name !== "custom_tag"),
|
||||
$.map_group,
|
||||
$.map_tag,
|
||||
),
|
||||
|
||||
filter: ($, previous) =>
|
||||
@@ -203,7 +238,8 @@ Every case below is input Django itself accepts:
|
||||
|
||||
- **Number-shaped names.** Django has no number token: a word is a literal if `int()` or `float()` parses
|
||||
it and a variable lookup otherwise, so `0x1f`, `1a`, `1.`, `1.2.3` and `1__0` look up a variable of that
|
||||
name and render nothing.
|
||||
name and render nothing. (`1.` really is a lookup there and not a float — `Variable` rejects a trailing
|
||||
dot after parsing it — while `1.e5` is a float, and both are treated that way here.)
|
||||
- **An attribute of `True`, `False` or `None`.** Every Django context is seeded with those three names, so
|
||||
`{{ True.x }}` is an ordinary two-part lookup that can never resolve.
|
||||
- **A keyword argument written twice.** `{% with a=1 a=2 %}`, and the same in `{% include %}` and
|
||||
@@ -217,3 +253,7 @@ Every case below is input Django itself accepts:
|
||||
|
||||
In the other direction the grammar is more permissive only where it cannot know better, which is what the
|
||||
fallbacks above are for.
|
||||
|
||||
## License
|
||||
|
||||
MIT — see [LICENSE](LICENSE).
|
||||
+10
-3
@@ -395,10 +395,17 @@ const django = grammar({
|
||||
value: ($) => choice($.literal, $.variable_attribute),
|
||||
literal: ($) =>
|
||||
choice($.number, $.string, $.translated_string, $.boolean, $.none),
|
||||
// digits, with single underscores allowed between them, as Python's
|
||||
// int() and float() accept ("1_000") and Django's Variable relies on
|
||||
/**
|
||||
* Digits, with single underscores allowed between them, as Python's int()
|
||||
* and float() accept ("1_000") and Django's Variable relies on.
|
||||
*
|
||||
* A trailing dot needs an exponent after it. Variable calls float() and
|
||||
* then rejects what it just parsed if the last character is a dot — the
|
||||
* comment in Django reads `# "2." is invalid` — so "1." is a two-part
|
||||
* lookup there rather than a number, while "1.e5" is a float.
|
||||
*/
|
||||
number: ($) =>
|
||||
/[-+]?(?:[0-9](?:_?[0-9])*(?:\.(?:[0-9](?:_?[0-9])*)?)?|\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?/,
|
||||
/[-+]?(?:(?:[0-9](?:_?[0-9])*(?:\.[0-9](?:_?[0-9])*)?|\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?|[0-9](?:_?[0-9])*\.[eE][0-9](?:_?[0-9])*)/,
|
||||
string: ($) => STRING,
|
||||
/**
|
||||
* A string to translate at render time. Django matches "_(" and ")" as
|
||||
|
||||
+1
-1
@@ -135,7 +135,7 @@
|
||||
},
|
||||
"number": {
|
||||
"type": "PATTERN",
|
||||
"value": "[-+]?(?:[0-9](?:_?[0-9])*(?:\\.(?:[0-9](?:_?[0-9])*)?)?|\\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?"
|
||||
"value": "[-+]?(?:(?:[0-9](?:_?[0-9])*(?:\\.[0-9](?:_?[0-9])*)?|\\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?|[0-9](?:_?[0-9])*\\.[eE][0-9](?:_?[0-9])*)"
|
||||
},
|
||||
"string": {
|
||||
"type": "PATTERN",
|
||||
|
||||
+2013
-2014
File diff suppressed because it is too large.
Load diff
@@ -461,3 +461,40 @@ Template Variable Attribute Of A Literal
|
||||
|
||||
---
|
||||
|
||||
|
||||
==================
|
||||
Template Variable Trailing Dot With An Exponent
|
||||
==================
|
||||
|
||||
{{ 1.e5 }}
|
||||
|
||||
---
|
||||
|
||||
(template
|
||||
(content)
|
||||
(template_variable
|
||||
(filtered_value
|
||||
(value
|
||||
(literal
|
||||
(number)))))
|
||||
(content))
|
||||
|
||||
==================
|
||||
Template Variable Trailing Dot
|
||||
:error
|
||||
==================
|
||||
|
||||
{{ 1. }}
|
||||
|
||||
---
|
||||
|
||||
|
||||
==================
|
||||
Template Variable Signed Trailing Dot
|
||||
:error
|
||||
==================
|
||||
|
||||
{{ -1. }}
|
||||
|
||||
---
|
||||
|
||||
Reference in new issue
Block a user