danhanson/tree-sitter-django
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add exports and library info to README.md

Browse Source
This commit is contained in:
danhanson committed 2026-09-12 15:29:56 -05:00
1 parent b239bf6509
commit 1f556f6687
6 files changed
+42114 -41813
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CLAUDE.md
+17 -1
View File
@@ -138,7 +138,10 @@ arity, or whether a tag is a block tag. Two fallback rules accept them:
- `custom_tag` — `simpleTag($, $.identifier)`, i.e. what `simple_tag`/`inclusion_tag` accept, which is how
a tag is registered unless it needs the parser itself.
- the last alternative of `filter`, whose name is `$.identifier` rather than one of the builtin literals.
- `_custom_filter`, the last alternative of `filter`, whose name is `$.identifier` rather than one of the
builtin literals. Hidden, so the tree is what an inline alternative would give, but named so that a
grammar extending this one can filter it out of `filter`'s `members` the way it can `custom_tag` — see
the extension section in `README.md`.
**A name that only exists inside a tag group has to be reserved instead.** `endif`, `else`, `empty` and
the rest are tokens only in the state their group opens, so where a tag is named the lexer reads them as an
@@ -181,6 +184,14 @@ Modelling a library name makes it keyword-extracted, so it no longer reaches `cu
registers its own tag under one of these names and a different signature therefore gets a parse error. That
is accepted: these names are common enough that shadowing them is the mistake.
**Adding a library tag or filter touches four files**: `grammar.js`, `queries/libraries.scm` (which library
it came from), `queries/highlights.scm` (the name, spelled out) and the table in `README.md`, which lists
every library name the grammar models. All four are hand-maintained, and **no test catches a name left out
of any of them**: `tree-sitter test` loads every `.scm` under `queries/`, so a pattern naming a node that
no longer exists aborts the whole run, but a name that was never added passes 304/304 in silence, and the
README nothing checks at all. The library-contents differential below, run against
`django.templatetags.<name>.register`, is what finds a gap.
### External scanner (`src/scanner.c`)
Twelve external tokens, for the constraints a context-free grammar cannot express:
@@ -288,6 +299,11 @@ documentation or memory. Both checks are worth repeating whenever tags, filters
### Deliberate divergences from Django
`README.md` carries a shorter list of its own, and the two are not the same set on purpose: the README
lists only the divergences where Django accepts an input that is almost certainly a mistake, which is the
rule it states. Anything added here that fits that rule belongs there too; anything that does not — an
unavoidable limitation, or a case where this grammar is the more permissive one — stays here.
- Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal,
so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here.
- Django has no boolean or `None` token either. `True`, `False` and `None` are lookups, which resolve
README.md
+131 -3
View File
@@ -12,6 +12,135 @@ npm run parser-build # build the WASM binary
npm run playground # interactively explore the grammar
```
## Django libraries
Five of Django's template libraries are not builtins: their names have to be `{% load %}`ed, or preloaded
through the engine's `OPTIONS: {"builtins": [...]}`. Every tag and filter all five register is modelled,
with the same argument and arity rules as a builtin:
| library | tags | filters |
| -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| `i18n` | `trans`, `translate`, `blocktrans`, `blocktranslate`, `language`, `get_available_languages`, `get_current_language`, `get_current_language_bidi`, `get_language_info`, `get_language_info_list` | `language_bidi`, `language_name`, `language_name_local`, `language_name_translated` |
| `l10n` | `localize` | `localize`, `unlocalize` |
| `static` | `static`, `get_static_prefix`, `get_media_prefix` | — |
| `cache` | `cache` | — |
| `tz` | `localtime`, `timezone`, `get_current_timezone` | `localtime`, `timezone`, `utc` |
**The `{% load %}` is never required**, because an engine may have preloaded the library. Whether one is
present is a question about the order of nodes rather than about how the template parses, so it is left to
a tool: `queries/libraries.scm` marks each name with the library it came from and each `{% load %}` with
what it brings in, and the tool joins the two. Such a tool needs to know that loading is positional
(`{% trans "x" %}{% load i18n %}` is an error), that it is per file rather than inherited through
`{% extends %}` or `{% include %}`, and that `{% load trans from i18n %}` brings in only the names it lists.
Because these names are modelled, they win over the fallbacks below: a project that registers its own tag
under one of them with a different signature gets a parse error. They are common enough that shadowing
them is treated as the mistake.
## Custom tags and filters
A library's tags and filters are registered in Python, so the grammar cannot know their names, their
arity, or whether a tag opens a body. Two fallback rules accept them:
- **Tags.** Any tag the grammar does not recognise parses as `custom_tag`, with the shape `simple_tag` and
`inclusion_tag` give: positional filter expressions, then `name=value` keyword arguments, then an
optional `as name`. So `{% mytag a b|upper "s" k=1 as out %}` parses, with the name on the `tag:` field.
A keyword argument written twice is refused, as `parse_bits` refuses it.
- **Filters.** An unknown filter parses as `(filter name: (identifier))` with at most one `:argument`,
which is all Django's expression regex allows. No arity check is possible.
Both are captured as `@function.call` rather than `@function`, so an editor can distinguish a name the
grammar knows from one it is only tolerating.
What the fallbacks cannot do:
- **A block tag is not nested.** `{% mytag %}…{% endmytag %}` is two sibling `custom_tag`s; pairing them
would mean guessing that an unknown tag opens a body.
- **A tag registered with `@register.tag` may parse anything**, since it receives the raw token, so one
whose arguments are not the `simple_tag` shape — `{% mytag <<>> %}` — is an error here and not in Django.
- **A misspelled name cannot be caught.** `{{ x|lenght }}` is indistinguishable from a filter some library
registered. Cross-referencing names against `{% load %}` needs the project's Python, so it belongs in a
linter; `queries/libraries.scm` and the `load` rule supply what one needs.
- **Names that only exist inside a tag group are reserved**, so `{% endif %}` or `{% empty %}` standing on
its own is an error rather than a call to a custom tag of that name.
### Adding rules for your own tags and filters
For a project whose custom tags deserve real rules rather than the fallbacks, use this grammar as a base
grammar. `grammar(base, {...})` merges your rules into it, and a rule that redefines one the base already
has receives it as `previous`. A tag is an alternative of `template_block_groups`; a filter is an
alternative of `filter`.
```js
import django from "tree-sitter-django/grammar";
const SEP = /[ \t\r\n]+/;
export default grammar(django, {
name: "djangox",
rules: {
// {% map items over rows %}
template_block_groups: ($, previous) => choice(previous, $.map_tag),
map_tag: ($) =>
seq(
"{%",
optional(SEP),
"map",
SEP,
$.filtered_value,
SEP,
"over",
SEP,
$.filtered_value,
optional(SEP),
"%}",
),
// {{ value|shout:"!" }}
filter: ($, previous) =>
choice(
previous,
seq(field("name", "shout"), ":", field("argument", $.value)),
),
},
});
```
Four things to know:
- **Whitespace is explicit.** `extras` is empty, because Django splits a tag's contents before parsing any
argument, so every separator has to be written out. The `part`/`block`/`simpleTag` helpers the base
grammar is built from are private to it, hence the `SEP` above.
- **A new name wins over the fallback.** `shout` becomes a token of its own, so it commits to your
alternative and your arity applies — `{{ x|shout }}` is now an error, while names you have not modelled
still fall through to `(filter name: (identifier))`.
- **A tag with a body needs a `conflicts` entry**, as every clause in the base grammar does:
`conflicts: ($, previous) => [...previous, [$.map_clause]]`. The generator offers an associativity
instead; taking it silently discards the parse in which the body continues. Its end tag is not reserved
either, so a stray `{% endmap %}` parses as a `custom_tag` rather than an error — `reserved` takes no
`previous`, so the `tag_name` list cannot be added to a name at a time.
- **The external scanner has to be re-exported under your grammar's name**, or block-name matching and the
repeated-argument guards will not link. `exports` cannot deliver it — that is Node's resolver, and the
scanner is C — so give your grammar a `src/scanner.c` that renames the base's five entry points to the
ones your generated `parser.c` calls, and includes it:
```c
#define tree_sitter_django_external_scanner_create tree_sitter_djangox_external_scanner_create
#define tree_sitter_django_external_scanner_destroy tree_sitter_djangox_external_scanner_destroy
#define tree_sitter_django_external_scanner_scan tree_sitter_djangox_external_scanner_scan
#define tree_sitter_django_external_scanner_serialize tree_sitter_djangox_external_scanner_serialize
#define tree_sitter_django_external_scanner_deserialize tree_sitter_djangox_external_scanner_deserialize
#include "../node_modules/tree-sitter-django/src/scanner.c"
```
The base's `tree_sitter/parser.h` then resolves to your own generated copy, and the externals keep the
order they have here, so the scanner's token enum stays valid. `tree-sitter build` takes no include
path, hence the relative `#include`; the package also exports `tree-sitter-django/src/scanner.c`, so a
build step can hand the compiler an absolute path from `import.meta.resolve` where a package manager's
layout makes the relative one unreliable.
## Divergences from Django
The grammar aims to accept what Django accepts, with one deliberate exception. Where Django takes an
@@ -33,6 +162,5 @@ Every case below is input Django itself accepts:
in that position, so `{% cache 500 using="c" %}` names a fragment called `using="c"` rather than choosing
a cache, and `{% cache 500 sidebar.name %}` names one with a dot in it.
In the other direction the grammar is more permissive only where it cannot know better: the names a
`{% load %}` brings in live in Python, so `{{ x|lenght }}` cannot be told from a filter some library
registered, and an unknown tag's arguments are assumed to have the shape `simple_tag` gives them.
In the other direction the grammar is more permissive only where it cannot know better, which is what the
fallbacks above are for.
grammar.js
+13 -6
View File
@@ -549,12 +549,19 @@ const django = grammar({
field("name", choice(...FILTERS_WITH_OPTIONAL_ARGUMENT)),
optional(seq(":", field("argument", $.value))),
),
// a filter a "load" brought in: its name is not known here, and so
// neither is whether it takes an argument
seq(
field("name", $.identifier),
optional(seq(":", field("argument", $.value))),
),
$._custom_filter,
),
/**
* A filter a "load" brought in: its name is not known here, and so neither
* is whether it takes an argument. Hidden, so the tree is the same as if
* it were spelled inline, but named so that a grammar extending this one
* can drop it from filter's alternatives the way it can drop custom_tag
* from template_block_groups.
*/
_custom_filter: ($) =>
seq(
field("name", $.identifier),
optional(seq(":", field("argument", $.value))),
),
autoescape_clause: ($) =>
seq(block("autoescape", part(choice("on", "off"))), optional($.template)),
package.json
+6
View File
@@ -4,6 +4,12 @@
"private": true,
"license": "MIT",
"type": "module",
"exports": {
"./grammar": "./grammar.js",
"./src/scanner.c": "./src/scanner.c",
"./queries/*": "./queries/*",
"./package.json": "./package.json"
},
"scripts": {
"lint": "eslint",
"lint:fix": "eslint --fix",
src/grammar.json
+30 -26
View File
@@ -1313,40 +1313,44 @@
]
},
{
"type": "SEQ",
"type": "SYMBOL",
"name": "_custom_filter"
}
]
},
"_custom_filter": {
"type": "SEQ",
"members": [
{
"type": "FIELD",
"name": "name",
"content": {
"type": "SYMBOL",
"name": "identifier"
}
},
{
"type": "CHOICE",
"members": [
{
"type": "FIELD",
"name": "name",
"content": {
"type": "SYMBOL",
"name": "identifier"
}
},
{
"type": "CHOICE",
"type": "SEQ",
"members": [
{
"type": "SEQ",
"members": [
{
"type": "STRING",
"value": ":"
},
{
"type": "FIELD",
"name": "argument",
"content": {
"type": "SYMBOL",
"name": "value"
}
}
]
"type": "STRING",
"value": ":"
},
{
"type": "BLANK"
"type": "FIELD",
"name": "argument",
"content": {
"type": "SYMBOL",
"name": "value"
}
}
]
},
{
"type": "BLANK"
}
]
}
src/parser.c
+41917 -41777
View File
File diff suppressed because it is too large. Load diff