danhanson/tree-sitter-django
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add divergences section to README.md and fixed reversed and noop arguments

Browse Source
This commit is contained in:
danhanson committed 2026-09-10 15:56:16 -05:00
1 parent 913ca69e6c
commit 9701fab04a
7 files changed
+41143 -38781
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CLAUDE.md
+9 -6
View File
@@ -61,6 +61,15 @@ that place it:
order but never twice. Used by `translate` and `include`; keyword arguments are guarded by the scanner order but never twice. Used by `translate` and `include`; keyword arguments are guarded by the scanner
instead. Grows as `Σ C(n,k)·k!` (65 for four names), which is instead. Grows as `Σ C(n,k)·k!` (65 for four names), which is
why it is only used for signatures that name their arguments. why it is only used for signatures that name their arguments.
- `wordAsValueRules(word)` / `wordAsValue($, word, continued)` — a tag's own keyword written where a value
goes. Django tells its flag from a variable by comparing the whole bit, so `reversed` is the flag of
`{% for %}` while `reversed.x` and `reversed|last` are lookups; `continued` selects that second group.
Needed because keyword extraction hands the parser the keyword wherever both it and `identifier` are
valid, so the reading in which the word names a variable has to be built back out of it. The rules are
hidden and aliased at the use site, one per level (`variable_attribute`, `value`, `filtered_value`),
because an alias renames the node its own rule makes and **cannot nest one inside another** — writing
it as nested inline `alias(alias(...))` calls silently yields a flat, partly zero-width tree that still
generates and still parses. Check the tree, not just the accept.
Consequences worth knowing before editing a rule: Consequences worth knowing before editing a rule:
@@ -305,11 +314,5 @@ documentation or memory. Both checks are worth repeating whenever tags, filters
`{% cache 500 sidebar.name %}` names a fragment there, `{% cache 500 using="c" %}` names a fragment `{% cache 500 sidebar.name %}` names a fragment there, `{% cache 500 using="c" %}` names a fragment
called `using="c"` rather than choosing a cache, and `{% cache 500 sidebar using= %}` takes an empty called `using="c"` rather than choosing a cache, and `{% cache 500 sidebar using= %}` takes an empty
name. All three are errors here. name. All three are errors here.
- `do_for` reads `reversed` off the end of the tag before it looks for `in`, so `{% for x in reversed %}`
is an error there and a loop over a variable named `reversed` here. Writing the flag as well
(`{% for x in reversed reversed %}`) is accepted by both, which is why the name is not simply reserved
in that position.
- `do_translate` refuses `as` and `noop` as the value of its `context` option, so
`{% trans "hi" context noop %}` is an error there and a context named by the variable `noop` here.
- `partial`/`partialdef` are Django builtins as of Django 6; `elif`/`else`/`empty` are modelled as parts of - `partial`/`partialdef` are Django builtins as of Django 6; `elif`/`else`/`empty` are modelled as parts of
their enclosing tag rather than as separate tags. their enclosing tag rather than as separate tags.
README.md
+25
View File
@@ -11,3 +11,28 @@ npm run parser-test # run the corpus tests in test/corpus
npm run parser-build # build the WASM binary npm run parser-build # build the WASM binary
npm run playground # interactively explore the grammar npm run playground # interactively explore the grammar
``` ```
## Divergences from Django
The grammar aims to accept what Django accepts, with one deliberate exception. Where Django takes an
input without complaint but the input is almost certainly a mistake — it parses, and then silently
renders nothing or quietly ignores half of what was written — the grammar reports an error instead.
Every case below is input Django itself accepts:
- **Number-shaped names.** Django has no number token: a word is a literal if `int()` or `float()` parses
it and a variable lookup otherwise, so `0x1f`, `1a`, `1.`, `1.2.3` and `1__0` look up a variable of that
name and render nothing.
- **An attribute of `True`, `False` or `None`.** Every Django context is seeded with those three names, so
`{{ True.x }}` is an ordinary two-part lookup that can never resolve.
- **A keyword argument written twice.** `{% with a=1 a=2 %}`, and the same in `{% include %}` and
`{% blocktranslate %}`, where Django builds a dict and lets the last value win.
- **Anything after a `{% static %}` path.** `StaticNode.handle_token` looks for `as` two words from the end
and ignores the rest, so `{% static "a" junk %}` and `{% static "a" as url junk %}` render as if the
extra words were not there.
- **A `{% cache %}` argument meant as something other than the fragment name.** Django takes the raw word
in that position, so `{% cache 500 using="c" %}` names a fragment called `using="c"` rather than choosing
a cache, and `{% cache 500 sidebar.name %}` names one with a dot in it.
In the other direction the grammar is more permissive only where it cannot know better: the names a
`{% load %}` brings in live in Python, so `{{ x|lenght }}` cannot be told from a filter some library
registered, and an unknown tag's arguments are assumed to have the shape `simple_tag` gives them.
grammar.js
+72 -4
View File
@@ -64,6 +64,56 @@ function asVariable($) {
return part("as", field("variable", $.identifier)); return part("as", field("variable", $.identifier));
} }
/**
* A tag's own keyword written where a value goes, with the tree an ordinary
* lookup would have so that a tool sees no difference. Where a keyword and an
* identifier are both valid, tree-sitter hands the parser the keyword, so the
* reading in which the word names a variable has to be built back out of it —
* and out of hidden rules, because an alias only renames the node its rule
* makes and cannot nest one inside another.
*
* Django tells such a keyword from a variable by comparing the whole bit, so
* "reversed" is its flag while "reversed.x" and "reversed|last" are lookups.
* That second group is what `continued` selects.
*
* @param {string} word
* @returns {Record<string, RuleBuilder<string>>}
*/
function wordAsValueRules(word) {
return {
[`_${word}_lookup`]: ($) => alias(word, $.identifier),
[`_${word}_dotted_lookup`]: ($) =>
seq(alias(word, $.identifier), ".", $.attribute),
[`_${word}_value`]: ($) =>
alias($[`_${word}_lookup`], $.variable_attribute),
[`_${word}_dotted_value`]: ($) =>
alias($[`_${word}_dotted_lookup`], $.variable_attribute),
[`_${word}_bare`]: ($) => alias($[`_${word}_value`], $.value),
[`_${word}_continued`]: ($) =>
choice(
seq(
alias($[`_${word}_dotted_value`], $.value),
optional(seq("|", $.filter_expression)),
),
seq(alias($[`_${word}_value`], $.value), "|", $.filter_expression),
),
};
}
/**
* The rules wordAsValueRules() built, where a value goes.
* @param {GrammarSymbols<string>} $
* @param {string} word
* @param {boolean} continued whether a "." or a "|" has to follow the word
* @returns {RuleOrLiteral}
*/
function wordAsValue($, word, continued) {
return alias(
$[`_${word}_${continued ? "continued" : "bare"}`],
$.filtered_value,
);
}
/** /**
* @param {RuleOrLiteral[]} chosen * @param {RuleOrLiteral[]} chosen
* @param {RuleOrLiteral[]} rest * @param {RuleOrLiteral[]} rest
@@ -355,6 +405,10 @@ const django = grammar({
* string: "_( 'a' )" is a syntax error there, not a translated literal. * string: "_( 'a' )" is a syntax error there, not a translated literal.
*/ */
translated_string: ($) => seq("_(", $.string, ")"), translated_string: ($) => seq("_(", $.string, ")"),
// the flags of "for" and "trans", spelled as the variable names they
// also are; see wordAsValueRules()
...wordAsValueRules("reversed"),
...wordAsValueRules("noop"),
boolean: ($) => choice("True", "False"), boolean: ($) => choice("True", "False"),
none: ($) => "None", none: ($) => "None",
attribute: ($) => /[a-zA-Z0-9][a-zA-Z0-9_]*(?:\.[a-zA-Z0-9][a-zA-Z0-9_]*)*/, attribute: ($) => /[a-zA-Z0-9][a-zA-Z0-9_]*(?:\.[a-zA-Z0-9][a-zA-Z0-9_]*)*/,
@@ -651,11 +705,20 @@ const django = grammar({
), ),
), ),
part("in"), part("in"),
part($.filtered_value), // do_for reads the flag off the end and takes the sequence from the
// do_for reads this off the end and takes the sequence from the bit // bit before it, so the flag comes last or not at all. A bare
// before it, so it comes last or not at all // "reversed" in the sequence position leaves nothing to iterate,
// naming a sequence only when a lookup or filter continues it or
// when the flag follows it as well
choice(
seq(part($.filtered_value), optional(part("reversed"))),
seq(
part(wordAsValue($, "reversed", true)),
optional(part("reversed")), optional(part("reversed")),
), ),
seq(part(wordAsValue($, "reversed", false)), part("reversed")),
),
),
optional($.template), optional($.template),
), ),
empty_clause: ($) => seq(block("empty"), optional($.template)), empty_clause: ($) => seq(block("empty"), optional($.template)),
@@ -814,7 +877,12 @@ const django = grammar({
optional( optional(
arrangements([ arrangements([
part("noop"), part("noop"),
part("context", $.filtered_value), // do_translate refuses "as" and "noop" as the context, comparing
// the whole bit, so a lookup that only begins with one is fine
part(
"context",
choice($.filtered_value, wordAsValue($, "noop", true)),
),
asVariable($), asVariable($),
]), ]),
), ),
src/grammar.json
+484
View File
@@ -158,6 +158,246 @@
} }
] ]
}, },
"_reversed_lookup": {
"type": "ALIAS",
"content": {
"type": "STRING",
"value": "reversed"
},
"named": true,
"value": "identifier"
},
"_reversed_dotted_lookup": {
"type": "SEQ",
"members": [
{
"type": "ALIAS",
"content": {
"type": "STRING",
"value": "reversed"
},
"named": true,
"value": "identifier"
},
{
"type": "STRING",
"value": "."
},
{
"type": "SYMBOL",
"name": "attribute"
}
]
},
"_reversed_value": {
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_reversed_lookup"
},
"named": true,
"value": "variable_attribute"
},
"_reversed_dotted_value": {
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_reversed_dotted_lookup"
},
"named": true,
"value": "variable_attribute"
},
"_reversed_bare": {
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_reversed_value"
},
"named": true,
"value": "value"
},
"_reversed_continued": {
"type": "CHOICE",
"members": [
{
"type": "SEQ",
"members": [
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_reversed_dotted_value"
},
"named": true,
"value": "value"
},
{
"type": "CHOICE",
"members": [
{
"type": "SEQ",
"members": [
{
"type": "STRING",
"value": "|"
},
{
"type": "SYMBOL",
"name": "filter_expression"
}
]
},
{
"type": "BLANK"
}
]
}
]
},
{
"type": "SEQ",
"members": [
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_reversed_value"
},
"named": true,
"value": "value"
},
{
"type": "STRING",
"value": "|"
},
{
"type": "SYMBOL",
"name": "filter_expression"
}
]
}
]
},
"_noop_lookup": {
"type": "ALIAS",
"content": {
"type": "STRING",
"value": "noop"
},
"named": true,
"value": "identifier"
},
"_noop_dotted_lookup": {
"type": "SEQ",
"members": [
{
"type": "ALIAS",
"content": {
"type": "STRING",
"value": "noop"
},
"named": true,
"value": "identifier"
},
{
"type": "STRING",
"value": "."
},
{
"type": "SYMBOL",
"name": "attribute"
}
]
},
"_noop_value": {
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_lookup"
},
"named": true,
"value": "variable_attribute"
},
"_noop_dotted_value": {
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_dotted_lookup"
},
"named": true,
"value": "variable_attribute"
},
"_noop_bare": {
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_value"
},
"named": true,
"value": "value"
},
"_noop_continued": {
"type": "CHOICE",
"members": [
{
"type": "SEQ",
"members": [
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_dotted_value"
},
"named": true,
"value": "value"
},
{
"type": "CHOICE",
"members": [
{
"type": "SEQ",
"members": [
{
"type": "STRING",
"value": "|"
},
{
"type": "SYMBOL",
"name": "filter_expression"
}
]
},
{
"type": "BLANK"
}
]
}
]
},
{
"type": "SEQ",
"members": [
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_value"
},
"named": true,
"value": "value"
},
{
"type": "STRING",
"value": "|"
},
{
"type": "SYMBOL",
"name": "filter_expression"
}
]
}
]
},
"boolean": { "boolean": {
"type": "CHOICE", "type": "CHOICE",
"members": [ "members": [
@@ -3426,6 +3666,12 @@
} }
] ]
}, },
{
"type": "CHOICE",
"members": [
{
"type": "SEQ",
"members": [
{ {
"type": "SEQ", "type": "SEQ",
"members": [ "members": [
@@ -3459,6 +3705,90 @@
"type": "BLANK" "type": "BLANK"
} }
] ]
}
]
},
{
"type": "SEQ",
"members": [
{
"type": "SEQ",
"members": [
{
"type": "PATTERN",
"value": "[ \\t\\r\\n]+"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_reversed_continued"
},
"named": true,
"value": "filtered_value"
}
]
},
{
"type": "CHOICE",
"members": [
{
"type": "SEQ",
"members": [
{
"type": "PATTERN",
"value": "[ \\t\\r\\n]+"
},
{
"type": "STRING",
"value": "reversed"
}
]
},
{
"type": "BLANK"
}
]
}
]
},
{
"type": "SEQ",
"members": [
{
"type": "SEQ",
"members": [
{
"type": "PATTERN",
"value": "[ \\t\\r\\n]+"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_reversed_bare"
},
"named": true,
"value": "filtered_value"
}
]
},
{
"type": "SEQ",
"members": [
{
"type": "PATTERN",
"value": "[ \\t\\r\\n]+"
},
{
"type": "STRING",
"value": "reversed"
}
]
}
]
}
]
}, },
{ {
"type": "CHOICE", "type": "CHOICE",
@@ -6733,9 +7063,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
@@ -6777,9 +7121,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
@@ -6929,9 +7287,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
@@ -6957,9 +7329,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
@@ -6986,9 +7372,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
@@ -7030,9 +7430,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
@@ -7104,9 +7518,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
@@ -7165,9 +7593,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
@@ -7360,9 +7802,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
@@ -7421,9 +7877,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
@@ -7482,9 +7952,23 @@
"type": "PATTERN", "type": "PATTERN",
"value": "[ \\t\\r\\n]+" "value": "[ \\t\\r\\n]+"
}, },
{
"type": "CHOICE",
"members": [
{ {
"type": "SYMBOL", "type": "SYMBOL",
"name": "filtered_value" "name": "filtered_value"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_noop_continued"
},
"named": true,
"value": "filtered_value"
}
]
} }
] ]
} }
src/parser.c
+40412 -38771
View File
File diff suppressed because it is too large. Load diff
test/corpus/for.txt
+86
View File
@@ -455,3 +455,89 @@ For With Reversed Before In
{% for x reversed in y %}a{% endfor %} {% for x reversed in y %}a{% endfor %}
--- ---
==================
For Over A Variable Named Reversed
==================
{% for x in reversed reversed %}a{% endfor %}
---
(template
(content)
(for_group
(for_clause
variable: (identifier)
(filtered_value
(value
(variable_attribute
(identifier))))
(template
(content))))
(content))
==================
For Over An Attribute Of A Variable Named Reversed
==================
{% for x in reversed.items %}a{% endfor %}
---
(template
(content)
(for_group
(for_clause
variable: (identifier)
(filtered_value
(value
(variable_attribute
(identifier)
(attribute))))
(template
(content))))
(content))
==================
For Over A Filtered Variable Named Reversed
==================
{% for x in reversed|last %}a{% endfor %}
---
(template
(content)
(for_group
(for_clause
variable: (identifier)
(filtered_value
(value
(variable_attribute
(identifier)))
(filter_expression
(filter)))
(template
(content))))
(content))
==================
For With Reversed As Its Whole Sequence
:error
==================
{% for x in reversed %}a{% endfor %}
---
==================
For With Reversed Three Times
:error
==================
{% for x in reversed reversed reversed %}a{% endfor %}
---
test/corpus/i18n.txt
+55
View File
@@ -488,3 +488,58 @@ Language Name Filter With An Argument
{{ "de"|language_name:1 }} {{ "de"|language_name:1 }}
--- ---
==================
Translate With A Context Named After Noop
:error
==================
{% trans "hi" context noop %}
---
==================
Translate With A Context From An Attribute Of Noop
==================
{% trans "hi" context noop.x %}
---
(template
(content)
(translate
(filtered_value
(value
(literal
(string))))
(filtered_value
(value
(variable_attribute
(identifier)
(attribute)))))
(content))
==================
Translate With A Filtered Context Named After Noop
==================
{% trans "hi" context noop|upper %}
---
(template
(content)
(translate
(filtered_value
(value
(literal
(string))))
(filtered_value
(value
(variable_attribute
(identifier)))
(filter_expression
(filter))))
(content))