danhanson/tree-sitter-django
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add support for custom tags and filters

Browse Source
This commit is contained in:
danhanson committed 2026-09-07 14:24:25 -05:00
1 parent 10e52ace5d
commit eb81b73104
7 files changed
+29218 -28775
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CLAUDE.md
+28 -1
View File
@@ -78,6 +78,23 @@ Django parses everything between `{{` and `}}` — and the entire remainder of `
whitespace first, so a pipe there must be tight. Hence `_filtered_value_spaced` /
`_filter_expression_spaced`, aliased back to the plain node names so the tree shape stays identical.
### Names a `{% load %}` brought in
A library's tags and filters are registered in Python, so the grammar cannot know their names, their
arity, or whether a tag is a block tag. Two fallback rules accept them:
- `custom_tag` — `simpleTag($, $.identifier)`, i.e. what `simple_tag`/`inclusion_tag` accept, which is how
a tag is registered unless it needs the parser itself.
- the last alternative of `filter`, whose name is `$.identifier` rather than one of the builtin literals.
**Builtin names must stay keyword-extractable, or the fallbacks swallow them.** `word: $.identifier` turns
each builtin's name into its own token, which the lexer prefers wherever it is valid, so a builtin commits
to its own alternative and its argument rules still apply. This is why a filter's name and its `:` are
separate tokens: while the name was spelled `"add:"` it was not word-shaped, so a bare `add` lexed as an
identifier and reached the fallback, losing the arity check for all 20 filters that require an argument.
`FILTERS_WITHOUT_ARGUMENT` / `FILTERS_WITH_ARGUMENT` / `FILTERS_WITH_OPTIONAL_ARGUMENT` carry that arity,
which Django itself enforces while it parses the template ("add requires 2 arguments, 1 provided").
### External scanner (`src/scanner.c`)
Nine external tokens, for the constraints a context-free grammar cannot express:
@@ -99,7 +116,8 @@ do not, and belong in a linter instead.
### Queries
`queries/highlights.scm` lists tag and filter names explicitly, so adding either to the grammar means
adding it there too. `queries/locals.scm` relies on the `variable:` field to tell a binding from a
adding it there too; filter names appear there without their colon. A name the grammar does not know is
captured through its field instead (`(filter name: (identifier))`, `(custom_tag tag: (identifier))`). `queries/locals.scm` relies on the `variable:` field to tell a binding from a
reference — another reason to keep `asVariable` uniform.
## Testing
@@ -137,5 +155,14 @@ documentation or memory. Both checks are worth repeating whenever tags, filters
- `{% querystring page=2 page=3 %}` (a repeated `**kwargs` key) is a Django error but is not expressible in
a context-free grammar. Repeats _are_ rejected wherever the signature names its arguments.
- `{% csp_nonce_attr "a" media="b" %}` is accepted, matching Django, which only rejects it at render time.
- A misspelled tag or filter cannot be caught: `{{ x|lenght }}` is indistinguishable from a filter some
library registered. Cross-referencing names against `{% load %}` needs the project's Python, so it
belongs in a linter; the `load` rule already parses the library and `from` names for one to use.
- `{% mytag %}…{% endmytag %}` parses as two sibling `custom_tag`s. Nesting them would mean guessing that
an unknown tag is a block tag.
- A custom tag's contents are only assumed to be the `simple_tag` shape. `@register.tag` receives the raw
token and may parse anything, so `{% mytag <<>> %}` is rejected here and not by Django.
- A library may override a builtin (`Parser.add_library` is `self.filters.update(...)`), so a project that
redefines `length` with a different arity gets a false error here.
- `partial`/`partialdef` are Django builtins as of Django 6; `elif`/`else`/`empty` are modelled as parts of
their enclosing tag rather than as separate tags.
grammar.js
+104 -59
View File
@@ -38,7 +38,7 @@ function part(first, ...rest) {
/**
* A tag, which may hug its delimiters ("{%cycle 1%}") or not.
* @param {string} tag
* @param {RuleOrLiteral} tag
* @param {...RuleOrLiteral} args
* @returns {SeqRule}
*/
@@ -121,7 +121,7 @@ function arrangements(items) {
* @param {boolean|Record<string, RuleOrLiteral>} kwargs true for any
* "name=value" pair (**kwargs), false for none, or the accepted names mapped
* to their value rules, each of which may be given at most once
* @param {string} tag
* @param {RuleOrLiteral} tag
* @returns {SeqRule}
*/
function simpleTag($, tag, args = true, kwargs = true) {
@@ -148,6 +148,79 @@ function simpleTag($, tag, args = true, kwargs = true) {
return block(tag, ...parts, optional(asVariable($)));
}
/**
* Django checks a filter's argument count against the registered function
* while it parses the template, so the arity of a builtin is syntax here:
* "length:2" and a bare "add" are both errors.
*/
const FILTERS_WITHOUT_ARGUMENT = [
"addslashes",
"capfirst",
"escape",
"escapejs",
"escapeseq",
"filesizeformat",
"first",
"force_escape",
"iriencode",
"last",
"length",
"linebreaks",
"linebreaksbr",
"linenumbers",
"lower",
"make_list",
"phone2numeric",
"pprint",
"random",
"safe",
"safeseq",
"slugify",
"striptags",
"title",
"unordered_list",
"upper",
"urlize",
"wordcount",
];
/** Builtin filters that require an argument. */
const FILTERS_WITH_ARGUMENT = [
"add",
"center",
"cut",
"default",
"default_if_none",
"dictsort",
"dictsortreversed",
"divisibleby",
"get_digit",
"join",
"ljust",
"rjust",
"slice",
"stringformat",
"truncatechars",
"truncatechars_html",
"truncatewords",
"truncatewords_html",
"urlizetrunc",
"wordwrap",
];
/** Builtin filters that take an argument or no argument. */
const FILTERS_WITH_OPTIONAL_ARGUMENT = [
"date",
"floatformat",
"json_script",
"pluralize",
"time",
"timesince",
"timeuntil",
"urlencode",
"yesno",
];
/** Django's string constant, which takes any escape but a line break. */
const STRING = /"(?:[^"\\]|\\[^\n])*"|'(?:[^'\\]|\\[^\n])*'/;
@@ -267,6 +340,7 @@ const django = grammar({
$.comment_group,
$.csp_nonce_attr,
$.csrf_token,
$.custom_tag,
$.cycle,
$.debug,
$.extends,
@@ -291,65 +365,30 @@ const django = grammar({
$.width_ratio,
$.with_group,
),
/**
* A filter is a name and, after a colon, at most one argument. The name is
* a token of its own rather than part of a "name:" token so that Django's
* builtins are keyword-extracted: a builtin written with the wrong arity
* then cannot be read as a filter from a loaded library instead.
*/
filter: ($) =>
choice(
seq("add:", $.value),
"addslashes",
"capfirst",
seq("center:", $.value),
seq("cut:", $.value),
choice(seq("date:", $.value), "date"),
seq("default:", $.value),
seq("default_if_none:", $.value),
seq("dictsort:", $.value),
seq("dictsortreversed:", $.value),
seq("divisibleby:", $.value),
"escape",
"escapejs",
"escapeseq",
"filesizeformat",
"first",
choice(seq("floatformat:", $.value), "floatformat"),
"force_escape",
seq("get_digit:", $.value),
"iriencode",
seq("join:", $.value),
choice(seq("json_script:", $.value), "json_script"),
"last",
"length",
"linebreaks",
"linebreaksbr",
"linenumbers",
seq("ljust:", $.value),
"lower",
"make_list",
"phone2numeric",
choice(seq("pluralize:", $.value), "pluralize"),
"pprint",
"random",
seq("rjust:", $.value),
"safe",
"safeseq",
seq("slice:", $.value),
"slugify",
seq("stringformat:", $.value),
"striptags",
choice(seq("time:", $.value), "time"),
choice(seq("timesince:", $.value), "timesince"),
choice(seq("timeuntil:", $.value), "timeuntil"),
"title",
seq("truncatechars:", $.value),
seq("truncatechars_html:", $.value),
seq("truncatewords:", $.value),
seq("truncatewords_html:", $.value),
"unordered_list",
"upper",
choice(seq("urlencode:", $.value), "urlencode"),
"urlize",
seq("urlizetrunc:", $.value),
"wordcount",
seq("wordwrap:", $.value),
choice(seq("yesno:", $.value), "yesno"),
field("name", choice(...FILTERS_WITHOUT_ARGUMENT)),
seq(
field("name", choice(...FILTERS_WITH_ARGUMENT)),
":",
field("argument", $.value),
),
seq(
field("name", choice(...FILTERS_WITH_OPTIONAL_ARGUMENT)),
optional(seq(":", field("argument", $.value))),
),
// a filter a "load" brought in: its name is not known here, and so
// neither is whether it takes an argument
seq(
field("name", $.identifier),
optional(seq(":", field("argument", $.value))),
),
),
autoescape_group: ($) =>
seq(
@@ -374,6 +413,12 @@ const django = grammar({
csp_nonce_attr: ($) =>
simpleTag($, "csp_nonce_attr", 1, { media: $.filtered_value }),
csrf_token: ($) => block("csrf_token"),
/**
* A tag a "load" brought in. Its arguments cannot be known here, so what
* is accepted is what Library.simple_tag and Library.inclusion_tag take,
* which is how a tag is registered unless it needs the parser itself.
*/
custom_tag: ($) => simpleTag($, $.identifier),
cycle: ($) =>
block(
"cycle",
queries/highlights.scm
+31 -32
View File
@@ -25,6 +25,7 @@ variable: (identifier) @variable.parameter
("in")
("=")
("|")
(":")
(".")] @operator
[
@@ -90,74 +91,72 @@ tag: [
("with")
("endwith")] @function
(filter .
[
("add:")
(filter
name: [
("add")
("addslashes")
("capfirst")
("center:")
("cut:")
("center")
("cut")
("date")
("date:")
("default:")
("default_if_none:")
("dictsort:")
("dictsortreversed:")
("divisibleby:")
("default")
("default_if_none")
("dictsort")
("dictsortreversed")
("divisibleby")
("escape")
("escapejs")
("escapeseq")
("filesizeformat")
("first")
("floatformat")
("floatformat:")
("force_escape")
("get_digit:")
("get_digit")
("iriencode")
("join:")
("join")
("json_script")
("json_script:")
("last")
("length")
("linebreaks")
("linebreaksbr")
("linenumbers")
("ljust:")
("ljust")
("lower")
("make_list")
("phone2numeric")
("pluralize")
("pluralize:")
("pprint")
("random")
("rjust:")
("rjust")
("safe")
("safeseq")
("slice:")
("slice")
("slugify")
("stringformat:")
("stringformat")
("striptags")
("time")
("time:")
("timesince")
("timesince:")
("timeuntil")
("timeuntil:")
("title")
("truncatechars:")
("truncatechars_html:")
("truncatewords:")
("truncatewords_html:")
("truncatechars")
("truncatechars_html")
("truncatewords")
("truncatewords_html")
("unordered_list")
("upper")
("urlencode")
("urlencode:")
("urlize")
("urlizetrunc:")
("urlizetrunc")
("wordcount")
("wordwrap:")
("yesno")
("yesno:")] @function)
("wordwrap")
("yesno")] @function)
; a filter or tag a "load" brought in, whose name the grammar does not know
(filter
name: (identifier) @function.call)
(custom_tag
tag: (identifier) @function.call)
(load
(identifier) @function)
src/grammar.json
+452 -536
View File
File diff suppressed because it is too large. Load diff
src/node-types.json
+318 -67
View File
@@ -23,6 +23,10 @@
"type": "csrf_token",
"named": true
},
{
"type": "custom_tag",
"named": true
},
{
"type": "cycle",
"named": true
@@ -300,6 +304,46 @@
}
}
},
{
"type": "custom_tag",
"named": true,
"fields": {
"tag": {
"multiple": false,
"required": true,
"types": [
{
"type": "identifier",
"named": true
}
]
},
"variable": {
"multiple": false,
"required": false,
"types": [
{
"type": "identifier",
"named": true
}
]
}
},
"children": {
"multiple": true,
"required": false,
"types": [
{
"type": "filtered_value",
"named": true
},
{
"type": "identifier",
"named": true
}
]
}
},
{
"type": "cycle",
"named": true,
@@ -381,16 +425,255 @@
{
"type": "filter",
"named": true,
"fields": {},
"children": {
"multiple": false,
"required": false,
"types": [
{
"type": "value",
"named": true
}
]
"fields": {
"argument": {
"multiple": false,
"required": false,
"types": [
{
"type": "value",
"named": true
}
]
},
"name": {
"multiple": false,
"required": true,
"types": [
{
"type": "add",
"named": false
},
{
"type": "addslashes",
"named": false
},
{
"type": "capfirst",
"named": false
},
{
"type": "center",
"named": false
},
{
"type": "cut",
"named": false
},
{
"type": "date",
"named": false
},
{
"type": "default",
"named": false
},
{
"type": "default_if_none",
"named": false
},
{
"type": "dictsort",
"named": false
},
{
"type": "dictsortreversed",
"named": false
},
{
"type": "divisibleby",
"named": false
},
{
"type": "escape",
"named": false
},
{
"type": "escapejs",
"named": false
},
{
"type": "escapeseq",
"named": false
},
{
"type": "filesizeformat",
"named": false
},
{
"type": "first",
"named": false
},
{
"type": "floatformat",
"named": false
},
{
"type": "force_escape",
"named": false
},
{
"type": "get_digit",
"named": false
},
{
"type": "identifier",
"named": true
},
{
"type": "iriencode",
"named": false
},
{
"type": "join",
"named": false
},
{
"type": "json_script",
"named": false
},
{
"type": "last",
"named": false
},
{
"type": "length",
"named": false
},
{
"type": "linebreaks",
"named": false
},
{
"type": "linebreaksbr",
"named": false
},
{
"type": "linenumbers",
"named": false
},
{
"type": "ljust",
"named": false
},
{
"type": "lower",
"named": false
},
{
"type": "make_list",
"named": false
},
{
"type": "phone2numeric",
"named": false
},
{
"type": "pluralize",
"named": false
},
{
"type": "pprint",
"named": false
},
{
"type": "random",
"named": false
},
{
"type": "rjust",
"named": false
},
{
"type": "safe",
"named": false
},
{
"type": "safeseq",
"named": false
},
{
"type": "slice",
"named": false
},
{
"type": "slugify",
"named": false
},
{
"type": "stringformat",
"named": false
},
{
"type": "striptags",
"named": false
},
{
"type": "time",
"named": false
},
{
"type": "timesince",
"named": false
},
{
"type": "timeuntil",
"named": false
},
{
"type": "title",
"named": false
},
{
"type": "truncatechars",
"named": false
},
{
"type": "truncatechars_html",
"named": false
},
{
"type": "truncatewords",
"named": false
},
{
"type": "truncatewords_html",
"named": false
},
{
"type": "unordered_list",
"named": false
},
{
"type": "upper",
"named": false
},
{
"type": "urlencode",
"named": false
},
{
"type": "urlize",
"named": false
},
{
"type": "urlizetrunc",
"named": false
},
{
"type": "wordcount",
"named": false
},
{
"type": "wordwrap",
"named": false
},
{
"type": "yesno",
"named": false
}
]
}
}
},
{
@@ -1317,6 +1600,10 @@
"type": ".",
"named": false
},
{
"type": ":",
"named": false
},
{
"type": "<",
"named": false
@@ -1346,7 +1633,7 @@
"named": false
},
{
"type": "add:",
"type": "add",
"named": false
},
{
@@ -1386,7 +1673,7 @@
"named": false
},
{
"type": "center:",
"type": "center",
"named": false
},
{
@@ -1426,7 +1713,7 @@
"named": false
},
{
"type": "cut:",
"type": "cut",
"named": false
},
{
@@ -1437,32 +1724,28 @@
"type": "date",
"named": false
},
{
"type": "date:",
"named": false
},
{
"type": "debug",
"named": false
},
{
"type": "default:",
"type": "default",
"named": false
},
{
"type": "default_if_none:",
"type": "default_if_none",
"named": false
},
{
"type": "dictsort:",
"type": "dictsort",
"named": false
},
{
"type": "dictsortreversed:",
"type": "dictsortreversed",
"named": false
},
{
"type": "divisibleby:",
"type": "divisibleby",
"named": false
},
{
@@ -1557,10 +1840,6 @@
"type": "floatformat",
"named": false
},
{
"type": "floatformat:",
"named": false
},
{
"type": "for",
"named": false
@@ -1574,7 +1853,7 @@
"named": false
},
{
"type": "get_digit:",
"type": "get_digit",
"named": false
},
{
@@ -1610,17 +1889,13 @@
"named": false
},
{
"type": "join:",
"type": "join",
"named": false
},
{
"type": "json_script",
"named": false
},
{
"type": "json_script:",
"named": false
},
{
"type": "last",
"named": false
@@ -1642,7 +1917,7 @@
"named": false
},
{
"type": "ljust:",
"type": "ljust",
"named": false
},
{
@@ -1725,10 +2000,6 @@
"type": "pluralize",
"named": false
},
{
"type": "pluralize:",
"named": false
},
{
"type": "pop_block",
"named": true
@@ -1774,7 +2045,7 @@
"named": false
},
{
"type": "rjust:",
"type": "rjust",
"named": false
},
{
@@ -1790,7 +2061,7 @@
"named": false
},
{
"type": "slice:",
"type": "slice",
"named": false
},
{
@@ -1806,7 +2077,7 @@
"named": true
},
{
"type": "stringformat:",
"type": "stringformat",
"named": false
},
{
@@ -1821,44 +2092,32 @@
"type": "time",
"named": false
},
{
"type": "time:",
"named": false
},
{
"type": "timesince",
"named": false
},
{
"type": "timesince:",
"named": false
},
{
"type": "timeuntil",
"named": false
},
{
"type": "timeuntil:",
"named": false
},
{
"type": "title",
"named": false
},
{
"type": "truncatechars:",
"type": "truncatechars",
"named": false
},
{
"type": "truncatechars_html:",
"type": "truncatechars_html",
"named": false
},
{
"type": "truncatewords:",
"type": "truncatewords",
"named": false
},
{
"type": "truncatewords_html:",
"type": "truncatewords_html",
"named": false
},
{
@@ -1877,16 +2136,12 @@
"type": "urlencode",
"named": false
},
{
"type": "urlencode:",
"named": false
},
{
"type": "urlize",
"named": false
},
{
"type": "urlizetrunc:",
"type": "urlizetrunc",
"named": false
},
{
@@ -1914,17 +2169,13 @@
"named": false
},
{
"type": "wordwrap:",
"type": "wordwrap",
"named": false
},
{
"type": "yesno",
"named": false
},
{
"type": "yesno:",
"named": false
},
{
"type": "{#",
"named": false
@@ -1945,4 +2196,4 @@
"type": "}}",
"named": false
}
]
]
src/parser.c
+28054 -28080
View File
File diff suppressed because it is too large. Load diff
test/corpus/custom.txt
+231
View File
@@ -0,0 +1,231 @@
==================
Custom Tag
==================
{% mytag %}
---
(template
(content)
(custom_tag
tag: (identifier))
(content))
==================
Custom Tag With Arguments
==================
{% mytag a "b" 3 %}
---
(template
(content)
(custom_tag
tag: (identifier)
(filtered_value
(value
(variable_attribute
(identifier))))
(filtered_value
(value
(literal
(string))))
(filtered_value
(value
(literal
(number)))))
(content))
==================
Custom Tag With Keyword Arguments
==================
{% mytag a b=1 %}
---
(template
(content)
(custom_tag
tag: (identifier)
(filtered_value
(value
(variable_attribute
(identifier))))
(identifier)
(filtered_value
(value
(literal
(number)))))
(content))
==================
Custom Tag With As
==================
{% mytag a as result %}
---
(template
(content)
(custom_tag
tag: (identifier)
(filtered_value
(value
(variable_attribute
(identifier))))
variable: (identifier))
(content))
==================
Custom Tag With Filtered Argument
==================
{% mytag a|upper %}
---
(template
(content)
(custom_tag
tag: (identifier)
(filtered_value
(value
(variable_attribute
(identifier)))
(filter_expression
(filter))))
(content))
==================
Custom Tag Does Not Nest
==================
{% mytag %}hi{% endmytag %}
---
(template
(content)
(custom_tag
tag: (identifier))
(content)
(custom_tag
tag: (identifier))
(content))
==================
Custom Tag Does Not Cover A Builtin Name
:error
==================
{% cycle %}
---
==================
Custom Filter
==================
{{ x|intcomma }}
---
(template
(content)
(template_variable
(filtered_value
(value
(variable_attribute
(identifier)))
(filter_expression
(filter
name: (identifier)))))
(content))
==================
Custom Filter With Argument
==================
{{ x|naturaltime:"y" }}
---
(template
(content)
(template_variable
(filtered_value
(value
(variable_attribute
(identifier)))
(filter_expression
(filter
name: (identifier)
argument: (value
(literal
(string)))))))
(content))
==================
Custom Filter In A Chain With A Builtin
==================
{{ x|intcomma|upper }}
---
(template
(content)
(template_variable
(filtered_value
(value
(variable_attribute
(identifier)))
(filter_expression
(filter
name: (identifier))
(filter))))
(content))
==================
Custom Filter In A Tag
==================
{% firstof a|intcomma %}
---
(template
(content)
(firstof
(filtered_value
(value
(variable_attribute
(identifier)))
(filter_expression
(filter
name: (identifier)))))
(content))
==================
Builtin Filter Without Its Required Argument
:error
==================
{{ x|add }}
---
==================
Builtin Filter With An Argument It Does Not Take
:error
==================
{{ x|length:2 }}
---