danhanson/tree-sitter-django
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add missing tag marker and error query

Browse Source
This commit is contained in:
danhanson committed 2026-09-12 19:21:12 -05:00
1 parent f2d3a727f6
commit 2d7d0db471
11 files changed
+34054 -19148
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CLAUDE.md
+39 -5
View File
@@ -109,13 +109,17 @@ extend it, so it reduces on whatever token follows — the end of input included
template still holds the last `{% for … %}` or `{% endif %}` typed as a whole node, not as loose tokens.
Stripping `_block` gives the name; a new tag, or a tag group's new part, follows the same scheme.
A group's last child is its end block or, where the end tag is missing, a zero-width marker named for it
(`missing_endif_block`; see the scanner section). So an unterminated group still reduces to a group node,
and only a stray tag or a body the grammar cannot read produces an `ERROR`.
The names themselves need no hand-maintained list either: in the generated `src/node-types.json`, a group's
`children` are its clause types and its end block, and a clause's children include its opening block, so
the candidates for a group are its end block plus the opening block of each clause it can hold. For
`if_group` that is `endif_block` from the group and `if_block`, `elif_block`, `else_block` from its three
clauses. Going the other way — from a clause the cursor sits in to the
groups that can hold it — is the same table read backwards, which is what an unterminated tag needs, since
the group node does not exist yet.
groups that can hold it — is the same table read backwards, which is what a clause needs when error
recovery has left it without its group.
Do not reach for lookahead to solve that instead: `ts_language_next_state` follows only the shift on `{%`,
which lands in the "a tag starts here" state no matter how the tree is shaped.
@@ -209,7 +213,7 @@ README nothing checks at all. The library-contents differential below, run again
### External scanner (`src/scanner.c`)
Twelve external tokens, for the constraints a context-free grammar cannot express:
Fifteen external tokens, for the constraints a context-free grammar cannot express:
- `push_block`/`push_partial`/`push_verbatim` and the matching `pop_*` — name matching for
`{% block a %}…{% endblock a %}`. They share **one** stack of `{kind, name}` entries, not one stack per
@@ -226,6 +230,25 @@ Twelve external tokens, for the constraints a context-free grammar cannot expres
flags, not a stack, because the tag cannot nest: its body admits no tags. It lives outside the
`stack`/`error` union, is cleared in both branches of `reset_scanner`, and is serialized after the
status character, so a GLR stack split copies it like everything else.
- `_group_open`, `_group_close` — zero width, before the name of a tag that opens or closes a tag group
(`groupBlock`). The scanner reads the name ahead of the grammar and pushes or pops a nameless entry, so
the stack holds every open group and not only the three whose names it matches; `block`, `partialdef`
and `verbatim` still push and pop through their own tokens, and are the only kinds with a pop token.
`{% plural %}` goes through `_group_close` too: reading the `count` option (`_bt_option`) turns a
blocktranslate entry into its counted kind, which waits for `plural` before its end tag. `tag_groups`
holds each kind's opening name and the names that belong to it, and `group_names` **must stay in sync
with `NAMES_INSIDE_A_TAG_GROUP`**.
- `_missing_tag` — the marker where a group's required tag is missing, aliased at each use to
`missing_endX_block` or `missing_plural_block` (`MISSING_BLOCKS`). It is placed only at the end of
input; before a tag whose name is reserved to some other group, which therefore cannot belong to this
body; or before an `endblock`/`endpartialdef` naming a block further out than the innermost. It closes
the innermost group on the stack as the end tag would, the group reduces with no `ERROR`, and
`queries/errors.scm` is what reports it. **It is one token aliased per group, never a token per
group**: tree-sitter does not merge parse states whose valid external tokens differ, so a token per
group cloned every body state for each kind of group and took the table from 1,818 states to 17,664
(it is 3,057 as built). That trade is also why the scanner keeps a stack of every group instead of
being told by the token which group it is in. A supertype would be the natural way to match every
marker, but the generator silently drops one whose members are aliases, so `errors.scm` lists them.
- `matcher_error` — used by no rule; returned only if the scanner reaches an error state.
`check_space()` defines whitespace for the scanner and **must stay in sync with `SEP`**, because the
@@ -251,7 +274,9 @@ Zero-width guards carry one hazard: `recover_with_missing` can supply one withou
during tree-sitter's mark-everything-valid recovery pass, so new guards go **below** it — and above the
whitespace loop, which skips the separator the grammar still has to match. A guard must also never be valid
where `content` is: a zero-width token at a content boundary preempts the internal lexer and `content` stops
matching.
matching. `_missing_tag` is the one exception, and is safe only because it returns a token solely at the end
of input or at `{%`, where `content` cannot match. It is scanned above the raw-text scanners, which find
nothing at the end of an empty body.
### Queries
@@ -281,9 +306,14 @@ A branch that is also a `@local.scope` confines its bindings whichever way the b
`{% for %}` is inert for that check: `ForNode` renders `nodelist_empty` **inside** the same
`context.push()` as the loop body, so `empty_clause` is a scope alongside `for_clause`.
`queries/errors.scm` collects what an editor should report: `(ERROR)` as `@error.syntax`, `(MISSING)` as
`@error.missing`, and every missing-tag marker as `@error.missing_tag`. A tree holding only markers has
no `has_error`, so a tool that checks that flag alone misses them. Adding a tag group means adding its
marker there, since no supertype matches them all.
## Testing
Corpus tests live in `test/corpus/*.txt`, one file per tag.
Corpus tests live in `test/corpus/*.txt`, one file per tag, and `missing.txt` for the missing-tag markers.
- A test whose header carries `:error` asserts only that the parse fails. Prefer it for syntax-error cases
over pinning an error-recovery tree, which is brittle.
@@ -355,3 +385,7 @@ unavoidable limitation, or a case where this grammar is the more permissive one
name. All three are errors here.
- `partial`/`partialdef` are Django builtins as of Django 6; `elif`/`else`/`empty` are modelled as parts of
their enclosing tag rather than as separate tags.
- An unterminated group parses here and raises `TemplateSyntaxError` in Django: `{% if a %}x`, or a
`{% for %}` closed while an `{% if %}` inside it is still open, holds a `missing_endif_block` marker
where the end tag belongs. The marker is the error, reported through `queries/errors.scm`; a check
for `ERROR` nodes alone accepts the tree.
README.md
+13 -4
View File
@@ -22,11 +22,11 @@ There is no published package yet, and no language bindings — those are coming
- **`src/parser.c` and `src/scanner.c` are generated and committed**, so any tree-sitter host can compile
them directly without running the CLI. The scanner is required: block-name matching
(`{% block a %}…{% endblock a %}`) and the repeated-argument guards live in it.
(`{% block a %}…{% endblock a %}`), the repeated-argument guards and the missing-tag markers live in it.
- **`npm run parser-build`** produces `tree-sitter-django.wasm` for a `web-tree-sitter` host or an editor
that loads Wasm grammars. It is a build artifact rather than a checked-in one, so it has to be built
(or shipped by a release) before anything can load it.
- **`queries/`** holds the highlight, locals, tags and injection queries an editor loads, plus three files
- **`queries/`** holds the highlight, locals, tags, injection and error queries an editor loads, plus three files
meant for a linter. See [Queries](#queries).
- **`tree-sitter.json`** names the grammar `django`, scopes it `source.django`, and points at the query
files, which is what a tree-sitter host reads to wire all of the above together.
@@ -48,7 +48,7 @@ that pass against the wrong grammar.
## Queries
Four of the files in `queries/` are the standard query files an editor loads:
Five of the files in `queries/` are query files an editor loads:
- **`highlights.scm`** — syntax highlighting. Tag and filter names are listed one by one, so a name the
grammar knows is captured as `@function` and a name it is only tolerating as `@function.call`.
@@ -61,6 +61,13 @@ Four of the files in `queries/` are the standard query files an editor loads:
- **`injections.scm`** — marks template text and `{% verbatim %}` bodies as another language's content, so
the HTML around the tags can be parsed by its own grammar. It names no language, leaving the editor to
supply one.
- **`errors.scm`** — what to report as a problem: `(ERROR)` as `@error.syntax`, tree-sitter's own
`(MISSING)` as `@error.missing`, and `@error.missing_tag` for the marker placed where a group's required
tag is missing. An unterminated `{% if a %}x` parses as an `if_group` ending in a zero-width
`missing_endif_block` rather than as an `ERROR`, so completion and the scope queries keep working while
a template is being written — and the tree reports no error of its own, so this query is how to find
it. A marker is placed only where the scanner can tell: at the end of input, before an end tag that
belongs to an enclosing group, or before an `{% endblock %}` naming a block further out.
The other three are data for a tool rather than queries an editor runs. Each captures the two halves of a
relation a tree-sitter query cannot express, and the tool does the join:
@@ -175,7 +182,7 @@ export default grammar(django, {
});
```
Five things to know:
Six things to know:
- **Whitespace is explicit.** `extras` is empty, because Django splits a tag's contents before parsing any
argument, so every separator has to be written out. The `part`/`block`/`simpleTag` helpers the base
@@ -190,6 +197,8 @@ Five things to know:
offers an associativity instead; taking it silently discards the parse in which the body continues.
- **End tags are not reserved**, a stray `{% endmap %}` parses as a `custom_tag_block` rather than an error — `reserved` takes no
`previous`, so the `tag_name` list cannot be added to a name at a time. Instead you may drop the `custom_tag_block` rule to avoid parsing end tags as custom tags.
- **Your own tag groups get no missing-tag markers.** Which groups the scanner tracks is fixed in its
tables, so an unterminated `{% map %}` is an `ERROR`.
- **The external scanner has to be re-exported under your grammar's name**, or block-name matching and the
repeated-argument guards will not link. Give your grammar a `src/scanner.c` that renames the base's five entry points to the
ones your generated `parser.c` calls, and includes it:
grammar.js
+244 -49
View File
@@ -53,6 +53,31 @@ function block(tag, ...args) {
);
}
/**
* A tag that opens or closes a tag group. The scanner keeps a stack of the
* groups that are open, which is what lets it tell a missing end tag from one
* that belongs to the group, so `marker` is a zero-width token placed before the
* tag's name: the scanner reads the name ahead of the grammar to push or pop the
* group. block, partialdef and verbatim push and pop through the tokens that
* match their names instead.
*
* @param {Rule} marker
* @param {RuleOrLiteral} tag
* @param {RuleOrLiteral[]} args
* @returns {SeqRule}
*/
function groupBlock(marker, tag, ...args) {
return seq(
"{%",
optional(SEP),
marker,
field("tag", tag),
...args,
optional(SEP),
"%}",
);
}
/**
* The "as name" clause that binds a tag's result in the context. Django spells
* it the same way in every tag that takes one, so the grammar names the target
@@ -217,9 +242,10 @@ function simpleTag($, tag, args = true, kwargs = true) {
function blocktransRules(tag) {
return {
[`_${tag}_block`]: ($) =>
block(tag, $._tag_open, repeat($._translate_option)),
groupBlock($._group_open, tag, $._tag_open, repeat($._translate_option)),
[`_${tag}_count_block`]: ($) =>
block(
groupBlock(
$._group_open,
tag,
$._tag_open,
repeat($._translate_option),
@@ -235,10 +261,13 @@ function blocktransRules(tag) {
seq(
alias($[`_${tag}_count_block`], $[`${tag}_block`]),
optional($._translate_body),
$.plural_clause,
choice(
$.plural_clause,
alias($._missing_tag, $.missing_plural_block),
),
),
),
[`end${tag}_block`]: ($) => block(`end${tag}`),
[`end${tag}_block`]: ($) => groupBlock($._group_close, `end${tag}`),
};
}
@@ -368,6 +397,41 @@ const NAMES_INSIDE_A_TAG_GROUP = [
"plural",
];
/**
* The names the scanner's marker takes where a tag group's required tag is
* missing, each named for the tag it stands in for. The scanner has one marker
* token, aliased to these at each use, so that every body accepts the same
* external tokens: tree-sitter will not merge states that differ in them, and a
* token per group made the parse table ten times the size.
*
* A marker is placed only at the end of input, before a tag whose name is
* reserved to some other group, or before an endblock or endpartialdef whose
* name closes a tag further out; the group then closes with no ERROR, and
* queries/errors.scm lists these names to report it. They are not a supertype:
* the generator drops a supertype whose members are aliases.
*/
const MISSING_BLOCKS = [
"missing_endautoescape_block",
"missing_endblock_block",
"missing_endblocktrans_block",
"missing_endblocktranslate_block",
"missing_endcache_block",
"missing_endcomment_block",
"missing_endfilter_block",
"missing_endfor_block",
"missing_endif_block",
"missing_endifchanged_block",
"missing_endlanguage_block",
"missing_endlocalize_block",
"missing_endlocaltime_block",
"missing_endpartialdef_block",
"missing_endspaceless_block",
"missing_endtimezone_block",
"missing_endverbatim_block",
"missing_endwith_block",
"missing_plural_block",
];
/** Django's string constant, which takes any escape but a line break. */
const STRING = /"(?:[^"\\]|\\[^\n])*"|'(?:[^'\\]|\\[^\n])*'/;
@@ -417,6 +481,9 @@ const django = grammar({
$._tag_open,
$._bt_option,
$._kwarg_name,
$._missing_tag,
$._group_open,
$._group_close,
],
reserved: {
global: ($) => ["not", "if", "in", "is", "as", "for", "from"],
@@ -608,23 +675,50 @@ const django = grammar({
field("name", $.identifier),
optional(seq(":", field("argument", $.value))),
),
autoescape_block: ($) => block("autoescape", part(choice("on", "off"))),
autoescape_block: ($) =>
groupBlock($._group_open, "autoescape", part(choice("on", "off"))),
autoescape_clause: ($) => seq($.autoescape_block, optional($.template)),
endautoescape_block: ($) => block("endautoescape"),
autoescape_group: ($) => seq($.autoescape_clause, $.endautoescape_block),
endautoescape_block: ($) => groupBlock($._group_close, "endautoescape"),
autoescape_group: ($) =>
seq(
$.autoescape_clause,
choice(
$.endautoescape_block,
alias($._missing_tag, $.missing_endautoescape_block),
),
),
// the scanner takes the whitespace before a name it has to match itself
block_block: ($) => block("block", field("name", $.push_block)),
block_clause: ($) => seq($.block_block, optional($.template)),
endblock_block: ($) => block("endblock", $.pop_block),
block_group: ($) => seq($.block_clause, $.endblock_block),
plural_block: ($) => block("plural"),
block_group: ($) =>
seq(
$.block_clause,
choice(
$.endblock_block,
alias($._missing_tag, $.missing_endblock_block),
),
),
plural_block: ($) => groupBlock($._group_close, "plural"),
plural_clause: ($) => seq($.plural_block, optional($._translate_body)),
...blocktransRules("blocktrans"),
...blocktransRules("blocktranslate"),
blocktranslate_group: ($) =>
choice(
seq($.blocktrans_clause, $.endblocktrans_block),
seq($.blocktranslate_clause, $.endblocktranslate_block),
seq(
$.blocktrans_clause,
choice(
$.endblocktrans_block,
alias($._missing_tag, $.missing_endblocktrans_block),
),
),
seq(
$.blocktranslate_clause,
choice(
$.endblocktranslate_block,
alias($._missing_tag, $.missing_endblocktranslate_block),
),
),
),
/**
* The scanner reads the name of the option before the grammar does, so it
@@ -671,7 +765,8 @@ const django = grammar({
),
),
cache_block: ($) =>
block(
groupBlock(
$._group_open,
"cache",
part($.filtered_value),
part(choice($.identifier, $.string)),
@@ -679,12 +774,27 @@ const django = grammar({
optional(part(seq("using=", $.filtered_value))),
),
cache_clause: ($) => seq($.cache_block, optional($.template)),
endcache_block: ($) => block("endcache"),
cache_group: ($) => seq($.cache_clause, $.endcache_block),
comment_block: ($) => block("comment", optional(part($.string))),
endcache_block: ($) => groupBlock($._group_close, "endcache"),
cache_group: ($) =>
seq(
$.cache_clause,
choice(
$.endcache_block,
alias($._missing_tag, $.missing_endcache_block),
),
),
comment_block: ($) =>
groupBlock($._group_open, "comment", optional(part($.string))),
comment_clause: ($) => seq($.comment_block, optional($.comment_content)),
endcomment_block: ($) => block("endcomment"),
comment_group: ($) => seq($.comment_clause, $.endcomment_block),
endcomment_block: ($) => groupBlock($._group_close, "endcomment"),
comment_group: ($) =>
seq(
$.comment_clause,
choice(
$.endcomment_block,
alias($._missing_tag, $.missing_endcomment_block),
),
),
// csp_nonce_attr(context, media=None)
csp_nonce_attr_block: ($) =>
simpleTag($, "csp_nonce_attr", 1, { media: $.filtered_value }),
@@ -713,13 +823,21 @@ const django = grammar({
// than as split arguments, so pipes here take whitespace just as they do
// between "{{" and "}}"
filter_block: ($) =>
block(
groupBlock(
$._group_open,
"filter",
part(alias($._filter_expression_spaced, $.filter_expression)),
),
filter_clause: ($) => seq($.filter_block, optional($.template)),
endfilter_block: ($) => block("endfilter"),
filter_group: ($) => seq($.filter_clause, $.endfilter_block),
endfilter_block: ($) => groupBlock($._group_close, "endfilter"),
filter_group: ($) =>
seq(
$.filter_clause,
choice(
$.endfilter_block,
alias($._missing_tag, $.missing_endfilter_block),
),
),
firstof_block: ($) =>
block(
"firstof",
@@ -727,7 +845,8 @@ const django = grammar({
optional(asVariable($)),
),
for_block: ($) =>
block(
groupBlock(
$._group_open,
"for",
part(field("variable", $.identifier)),
repeat(
@@ -756,9 +875,13 @@ const django = grammar({
for_clause: ($) => seq($.for_block, optional($.template)),
empty_block: ($) => block("empty"),
empty_clause: ($) => seq($.empty_block, optional($.template)),
endfor_block: ($) => block("endfor"),
endfor_block: ($) => groupBlock($._group_close, "endfor"),
for_group: ($) =>
seq($.for_clause, optional($.empty_clause), $.endfor_block),
seq(
$.for_clause,
optional($.empty_clause),
choice($.endfor_block, alias($._missing_tag, $.missing_endfor_block)),
),
/**
* django.templatetags.i18n. Each of these requires exactly "as <name>",
* or "for <expression> as <name>", and rejects anything further.
@@ -788,26 +911,34 @@ const django = grammar({
block("get_media_prefix", optional(asVariable($))),
get_static_prefix_block: ($) =>
block("get_static_prefix", optional(asVariable($))),
if_block: ($) => block("if", part($.predicate)),
if_block: ($) => groupBlock($._group_open, "if", part($.predicate)),
if_clause: ($) => seq($.if_block, optional($.template)),
elif_block: ($) => block("elif", part($.predicate)),
elif_clause: ($) => seq($.elif_block, optional($.template)),
/** Shared by if_group and ifchanged_group, which spell it the same way. */
else_block: ($) => block("else"),
else_clause: ($) => seq($.else_block, optional($.template)),
endif_block: ($) => block("endif"),
endif_block: ($) => groupBlock($._group_close, "endif"),
if_group: ($) =>
seq(
$.if_clause,
repeat($.elif_clause),
optional($.else_clause),
$.endif_block,
choice($.endif_block, alias($._missing_tag, $.missing_endif_block)),
),
ifchanged_block: ($) => block("ifchanged", repeat(part($.filtered_value))),
ifchanged_block: ($) =>
groupBlock($._group_open, "ifchanged", repeat(part($.filtered_value))),
ifchanged_clause: ($) => seq($.ifchanged_block, optional($.template)),
endifchanged_block: ($) => block("endifchanged"),
endifchanged_block: ($) => groupBlock($._group_close, "endifchanged"),
ifchanged_group: ($) =>
seq($.ifchanged_clause, optional($.else_clause), $.endifchanged_block),
seq(
$.ifchanged_clause,
optional($.else_clause),
choice(
$.endifchanged_block,
alias($._missing_tag, $.missing_endifchanged_block),
),
),
/**
* do_include takes "with" and "only" in either order and refuses one it has
* already been given, and "with" needs at least one argument. It calls
@@ -824,10 +955,18 @@ const django = grammar({
),
),
/** django.templatetags.i18n. The language tag takes the one argument. */
language_block: ($) => block("language", part($.filtered_value)),
language_block: ($) =>
groupBlock($._group_open, "language", part($.filtered_value)),
language_clause: ($) => seq($.language_block, optional($.template)),
endlanguage_block: ($) => block("endlanguage"),
language_group: ($) => seq($.language_clause, $.endlanguage_block),
endlanguage_block: ($) => groupBlock($._group_close, "endlanguage"),
language_group: ($) =>
seq(
$.language_clause,
choice(
$.endlanguage_block,
alias($._missing_tag, $.missing_endlanguage_block),
),
),
library: ($) => seq($.identifier, optional(seq(".", $.identifier))),
load_block: ($) =>
block(
@@ -842,16 +981,38 @@ const django = grammar({
* rejects anything but "on" or "off", including a second word.
*/
localize_block: ($) =>
block("localize", optional(part(choice("on", "off")))),
groupBlock(
$._group_open,
"localize",
optional(part(choice("on", "off"))),
),
localize_clause: ($) => seq($.localize_block, optional($.template)),
endlocalize_block: ($) => block("endlocalize"),
localize_group: ($) => seq($.localize_clause, $.endlocalize_block),
endlocalize_block: ($) => groupBlock($._group_close, "endlocalize"),
localize_group: ($) =>
seq(
$.localize_clause,
choice(
$.endlocalize_block,
alias($._missing_tag, $.missing_endlocalize_block),
),
),
/** django.templatetags.tz, taking "on" or "off" as localize does. */
localtime_block: ($) =>
block("localtime", optional(part(choice("on", "off")))),
groupBlock(
$._group_open,
"localtime",
optional(part(choice("on", "off"))),
),
localtime_clause: ($) => seq($.localtime_block, optional($.template)),
endlocaltime_block: ($) => block("endlocaltime"),
localtime_group: ($) => seq($.localtime_clause, $.endlocaltime_block),
endlocaltime_block: ($) => groupBlock($._group_close, "endlocaltime"),
localtime_group: ($) =>
seq(
$.localtime_clause,
choice(
$.endlocaltime_block,
alias($._missing_tag, $.missing_endlocaltime_block),
),
),
lorem_block: ($) =>
block(
"lorem",
@@ -865,7 +1026,14 @@ const django = grammar({
block("partialdef", $.push_partial, optional(part("inline"))),
partialdef_clause: ($) => seq($.partialdef_block, optional($.template)),
endpartialdef_block: ($) => block("endpartialdef", $.pop_partial),
partialdef_group: ($) => seq($.partialdef_clause, $.endpartialdef_block),
partialdef_group: ($) =>
seq(
$.partialdef_clause,
choice(
$.endpartialdef_block,
alias($._missing_tag, $.missing_endpartialdef_block),
),
),
// querystring(context, *args, **kwargs)
querystring_block: ($) => simpleTag($, "querystring"),
regroup_block: ($) =>
@@ -877,10 +1045,17 @@ const django = grammar({
optional(asVariable($)),
),
resetcycle_block: ($) => block("resetcycle", optional(part($.identifier))),
spaceless_block: ($) => block("spaceless"),
spaceless_block: ($) => groupBlock($._group_open, "spaceless"),
spaceless_clause: ($) => seq($.spaceless_block, optional($.template)),
endspaceless_block: ($) => block("endspaceless"),
spaceless_group: ($) => seq($.spaceless_clause, $.endspaceless_block),
endspaceless_block: ($) => groupBlock($._group_close, "endspaceless"),
spaceless_group: ($) =>
seq(
$.spaceless_clause,
choice(
$.endspaceless_block,
alias($._missing_tag, $.missing_endspaceless_block),
),
),
static_block: ($) =>
block("static", part($.filtered_value), optional(asVariable($))),
templatetag_block: ($) =>
@@ -900,10 +1075,18 @@ const django = grammar({
),
),
/** django.templatetags.tz. timezone_tag takes the one argument. */
timezone_block: ($) => block("timezone", part($.filtered_value)),
timezone_block: ($) =>
groupBlock($._group_open, "timezone", part($.filtered_value)),
timezone_clause: ($) => seq($.timezone_block, optional($.template)),
endtimezone_block: ($) => block("endtimezone"),
timezone_group: ($) => seq($.timezone_clause, $.endtimezone_block),
endtimezone_block: ($) => groupBlock($._group_close, "endtimezone"),
timezone_group: ($) =>
seq(
$.timezone_clause,
choice(
$.endtimezone_block,
alias($._missing_tag, $.missing_endtimezone_block),
),
),
/**
* django.templatetags.i18n, registered under both spellings. do_translate
* pops its options off one at a time and refuses one it has already seen,
@@ -941,7 +1124,14 @@ const django = grammar({
verbatim_block: ($) => block("verbatim", $.push_verbatim),
verbatim_clause: ($) => seq($.verbatim_block, optional($.verbatim_content)),
endverbatim_block: ($) => block("endverbatim", $.pop_verbatim),
verbatim_group: ($) => seq($.verbatim_clause, $.endverbatim_block),
verbatim_group: ($) =>
seq(
$.verbatim_clause,
choice(
$.endverbatim_block,
alias($._missing_tag, $.missing_endverbatim_block),
),
),
widthratio_block: ($) =>
block(
"widthratio",
@@ -950,10 +1140,15 @@ const django = grammar({
part($.filtered_value),
optional(asVariable($)),
),
with_block: ($) => block("with", $._tag_open, $._tag_kwargs),
with_block: ($) =>
groupBlock($._group_open, "with", $._tag_open, $._tag_kwargs),
with_clause: ($) => seq($.with_block, optional($.template)),
endwith_block: ($) => block("endwith"),
with_group: ($) => seq($.with_clause, $.endwith_block),
endwith_block: ($) => groupBlock($._group_close, "endwith"),
with_group: ($) =>
seq(
$.with_clause,
choice($.endwith_block, alias($._missing_tag, $.missing_endwith_block)),
),
},
});
queries/errors.scm
+42
View File
@@ -0,0 +1,42 @@
; Everything in a template that an editor should report as a problem.
;
; A group whose required tag is missing parses without an ERROR: the scanner
; supplies a zero-width marker where the tag belongs, so the group, its clauses
; and the scopes they open stay intact for completion and the other queries.
; Such a tree has no error of its own (ts_node_has_error is false), so a tool
; has to run this query to find them.
;
; @error.syntax input the grammar could not parse
; @error.missing a token that tree-sitter's error recovery inserted
; @error.missing_tag a tag its group requires, absent where the marker is;
; the marker's type names the tag
;
; The markers are listed one by one, as MISSING_BLOCKS in grammar.js: the
; generator drops a supertype whose members are aliases, so there is no single
; node type to match them all. A new tag group, or a new kind of marker, is
; added here.
(ERROR) @error.syntax
(MISSING) @error.missing
[
(missing_endautoescape_block)
(missing_endblock_block)
(missing_endblocktrans_block)
(missing_endblocktranslate_block)
(missing_endcache_block)
(missing_endcomment_block)
(missing_endfilter_block)
(missing_endfor_block)
(missing_endif_block)
(missing_endifchanged_block)
(missing_endlanguage_block)
(missing_endlocalize_block)
(missing_endlocaltime_block)
(missing_endpartialdef_block)
(missing_endspaceless_block)
(missing_endtimezone_block)
(missing_endverbatim_block)
(missing_endwith_block)
(missing_plural_block)] @error.missing_tag
src/grammar.json
+464 -40
View File
@@ -1375,6 +1375,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -1463,6 +1467,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -1497,8 +1505,22 @@
"name": "autoescape_clause"
},
{
"type": "SYMBOL",
"name": "endautoescape_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endautoescape_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endautoescape_block"
}
]
}
]
},
@@ -1633,8 +1655,22 @@
"name": "block_clause"
},
{
"type": "SYMBOL",
"name": "endblock_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endblock_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endblock_block"
}
]
}
]
},
@@ -1657,6 +1693,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -1723,6 +1763,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -1779,6 +1823,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -1881,8 +1929,22 @@
]
},
{
"type": "SYMBOL",
"name": "plural_clause"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "plural_clause"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_plural_block"
}
]
}
]
}
@@ -1907,6 +1969,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -1952,6 +2018,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -2008,6 +2078,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -2110,8 +2184,22 @@
]
},
{
"type": "SYMBOL",
"name": "plural_clause"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "plural_clause"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_plural_block"
}
]
}
]
}
@@ -2136,6 +2224,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -2173,8 +2265,22 @@
"name": "blocktrans_clause"
},
{
"type": "SYMBOL",
"name": "endblocktrans_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endblocktrans_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endblocktrans_block"
}
]
}
]
},
@@ -2186,8 +2292,22 @@
"name": "blocktranslate_clause"
},
{
"type": "SYMBOL",
"name": "endblocktranslate_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endblocktranslate_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endblocktranslate_block"
}
]
}
]
}
@@ -2422,6 +2542,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -2569,6 +2693,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -2603,8 +2731,22 @@
"name": "cache_clause"
},
{
"type": "SYMBOL",
"name": "endcache_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endcache_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endcache_block"
}
]
}
]
},
@@ -2627,6 +2769,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -2714,6 +2860,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -2748,8 +2898,22 @@
"name": "comment_clause"
},
{
"type": "SYMBOL",
"name": "endcomment_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endcomment_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endcomment_block"
}
]
}
]
},
@@ -3331,6 +3495,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -3415,6 +3583,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -3449,8 +3621,22 @@
"name": "filter_clause"
},
{
"type": "SYMBOL",
"name": "endfilter_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endfilter_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endfilter_block"
}
]
}
]
},
@@ -3572,6 +3758,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -3902,6 +4092,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -3948,8 +4142,22 @@
]
},
{
"type": "SYMBOL",
"name": "endfor_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endfor_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endfor_block"
}
]
}
]
},
@@ -4640,6 +4848,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -4864,6 +5076,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -4917,8 +5133,22 @@
]
},
{
"type": "SYMBOL",
"name": "endif_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endif_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endif_block"
}
]
}
]
},
@@ -4941,6 +5171,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -5023,6 +5257,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -5069,8 +5307,22 @@
]
},
{
"type": "SYMBOL",
"name": "endifchanged_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endifchanged_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endifchanged_block"
}
]
}
]
},
@@ -5283,6 +5535,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -5362,6 +5618,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -5396,8 +5656,22 @@
"name": "language_clause"
},
{
"type": "SYMBOL",
"name": "endlanguage_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endlanguage_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endlanguage_block"
}
]
}
]
},
@@ -5563,6 +5837,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -5659,6 +5937,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -5693,8 +5975,22 @@
"name": "localize_clause"
},
{
"type": "SYMBOL",
"name": "endlocalize_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endlocalize_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endlocalize_block"
}
]
}
]
},
@@ -5717,6 +6013,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -5813,6 +6113,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -5847,8 +6151,22 @@
"name": "localtime_clause"
},
{
"type": "SYMBOL",
"name": "endlocaltime_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endlocaltime_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endlocaltime_block"
}
]
}
]
},
@@ -6259,8 +6577,22 @@
"name": "partialdef_clause"
},
{
"type": "SYMBOL",
"name": "endpartialdef_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endpartialdef_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endpartialdef_block"
}
]
}
]
},
@@ -6607,6 +6939,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -6673,6 +7009,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -6707,8 +7047,22 @@
"name": "spaceless_clause"
},
{
"type": "SYMBOL",
"name": "endspaceless_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endspaceless_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endspaceless_block"
}
]
}
]
},
@@ -6918,6 +7272,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -6997,6 +7355,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -7031,8 +7393,22 @@
"name": "timezone_clause"
},
{
"type": "SYMBOL",
"name": "endtimezone_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endtimezone_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endtimezone_block"
}
]
}
]
},
@@ -8379,8 +8755,22 @@
"name": "verbatim_clause"
},
{
"type": "SYMBOL",
"name": "endverbatim_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endverbatim_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endverbatim_block"
}
]
}
]
},
@@ -8525,6 +8915,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "FIELD",
"name": "tag",
@@ -8599,6 +8993,10 @@
}
]
},
{
"type": "SYMBOL",
"name": "_group_close"
},
{
"type": "FIELD",
"name": "tag",
@@ -8633,8 +9031,22 @@
"name": "with_clause"
},
{
"type": "SYMBOL",
"name": "endwith_block"
"type": "CHOICE",
"members": [
{
"type": "SYMBOL",
"name": "endwith_block"
},
{
"type": "ALIAS",
"content": {
"type": "SYMBOL",
"name": "_missing_tag"
},
"named": true,
"value": "missing_endwith_block"
}
]
}
]
}
@@ -8764,6 +9176,18 @@
{
"type": "SYMBOL",
"name": "_kwarg_name"
},
{
"type": "SYMBOL",
"name": "_missing_tag"
},
{
"type": "SYMBOL",
"name": "_group_open"
},
{
"type": "SYMBOL",
"name": "_group_close"
}
],
"inline": [],
src/node-types.json
+156
View File
@@ -253,6 +253,10 @@
{
"type": "endautoescape_block",
"named": true
},
{
"type": "missing_endautoescape_block",
"named": true
}
]
}
@@ -322,6 +326,10 @@
{
"type": "endblock_block",
"named": true
},
{
"type": "missing_endblock_block",
"named": true
}
]
}
@@ -388,6 +396,10 @@
"type": "content",
"named": true
},
{
"type": "missing_plural_block",
"named": true
},
{
"type": "plural_clause",
"named": true
@@ -461,6 +473,10 @@
"type": "content",
"named": true
},
{
"type": "missing_plural_block",
"named": true
},
{
"type": "plural_clause",
"named": true
@@ -495,6 +511,14 @@
{
"type": "endblocktranslate_block",
"named": true
},
{
"type": "missing_endblocktrans_block",
"named": true
},
{
"type": "missing_endblocktranslate_block",
"named": true
}
]
}
@@ -572,6 +596,10 @@
{
"type": "endcache_block",
"named": true
},
{
"type": "missing_endcache_block",
"named": true
}
]
}
@@ -636,6 +664,10 @@
{
"type": "endcomment_block",
"named": true
},
{
"type": "missing_endcomment_block",
"named": true
}
]
}
@@ -1608,6 +1640,10 @@
{
"type": "filter_clause",
"named": true
},
{
"type": "missing_endfilter_block",
"named": true
}
]
}
@@ -1741,6 +1777,10 @@
{
"type": "for_clause",
"named": true
},
{
"type": "missing_endfor_block",
"named": true
}
]
}
@@ -2041,6 +2081,10 @@
{
"type": "if_clause",
"named": true
},
{
"type": "missing_endif_block",
"named": true
}
]
}
@@ -2109,6 +2153,10 @@
{
"type": "ifchanged_clause",
"named": true
},
{
"type": "missing_endifchanged_block",
"named": true
}
]
}
@@ -2209,6 +2257,10 @@
{
"type": "language_clause",
"named": true
},
{
"type": "missing_endlanguage_block",
"named": true
}
]
}
@@ -2339,6 +2391,10 @@
{
"type": "localize_clause",
"named": true
},
{
"type": "missing_endlocalize_block",
"named": true
}
]
}
@@ -2393,6 +2449,10 @@
{
"type": "localtime_clause",
"named": true
},
{
"type": "missing_endlocaltime_block",
"named": true
}
]
}
@@ -2542,6 +2602,10 @@
"type": "endpartialdef_block",
"named": true
},
{
"type": "missing_endpartialdef_block",
"named": true
},
{
"type": "partialdef_clause",
"named": true
@@ -2760,6 +2824,10 @@
"type": "endspaceless_block",
"named": true
},
{
"type": "missing_endspaceless_block",
"named": true
},
{
"type": "spaceless_clause",
"named": true
@@ -2916,6 +2984,10 @@
"type": "endtimezone_block",
"named": true
},
{
"type": "missing_endtimezone_block",
"named": true
},
{
"type": "timezone_clause",
"named": true
@@ -3117,6 +3189,10 @@
"type": "endverbatim_block",
"named": true
},
{
"type": "missing_endverbatim_block",
"named": true
},
{
"type": "verbatim_clause",
"named": true
@@ -3227,6 +3303,10 @@
"type": "endwith_block",
"named": true
},
{
"type": "missing_endwith_block",
"named": true
},
{
"type": "with_clause",
"named": true
@@ -3718,6 +3798,82 @@
"type": "media",
"named": false
},
{
"type": "missing_endautoescape_block",
"named": true
},
{
"type": "missing_endblock_block",
"named": true
},
{
"type": "missing_endblocktrans_block",
"named": true
},
{
"type": "missing_endblocktranslate_block",
"named": true
},
{
"type": "missing_endcache_block",
"named": true
},
{
"type": "missing_endcomment_block",
"named": true
},
{
"type": "missing_endfilter_block",
"named": true
},
{
"type": "missing_endfor_block",
"named": true
},
{
"type": "missing_endif_block",
"named": true
},
{
"type": "missing_endifchanged_block",
"named": true
},
{
"type": "missing_endlanguage_block",
"named": true
},
{
"type": "missing_endlocalize_block",
"named": true
},
{
"type": "missing_endlocaltime_block",
"named": true
},
{
"type": "missing_endpartialdef_block",
"named": true
},
{
"type": "missing_endspaceless_block",
"named": true
},
{
"type": "missing_endtimezone_block",
"named": true
},
{
"type": "missing_endverbatim_block",
"named": true
},
{
"type": "missing_endwith_block",
"named": true
},
{
"type": "missing_plural_block",
"named": true
},
{
"type": "none",
"named": true
src/parser.c
+32258 -19040
View File
File diff suppressed because it is too large. Load diff
src/scanner.c
+311 -8
View File
@@ -1,11 +1,20 @@
#include "tree_sitter/parser.h"
#include "tree_sitter/alloc.h"
#include "tree_sitter/array.h"
#include <string.h>
#define ERROR_SIZE 64
#define NAME_SEP ' '
/* A scanner is also built for Wasm, where only a few C library functions are
* available and string.h is not among them, so strings are compared here. */
static bool strings_equal(const char *left, const char *right) {
while (*left != '\0' && *left == *right) {
++left;
++right;
}
return *left == *right;
}
enum TokenType {
/* indicates that an error occurred */
MatcherError,
@@ -29,6 +38,14 @@ enum TokenType {
/* zero width, before the name of a "name=value" argument: record the name,
* and fail if the tag has already been given one by that name */
KwargName,
/* zero width, where a tag group's required tag is missing; see
* scan_missing_tag */
MissingTag,
/* zero width, before the name of a tag that opens a group: push the group */
GroupOpen,
/* zero width, before the name of a tag that closes a group, or of the plural
* of a counted blocktranslate: pop the group, or record its plural */
GroupClose,
};
/* The options blocktranslate takes, in the order of the bits recording them.
@@ -47,10 +64,67 @@ typedef enum {
BlockTag,
PartialTag,
VerbatimTag,
/* The groups below have no name to match, and are kept only so that a
* missing end tag can be told from one that belongs to the group. A counted
* blocktranslate is a kind of its own until its plural is read, because
* until then "plural" is the tag it is waiting for. */
AutoescapeTag,
BlocktransTag,
BlocktransCountTag,
BlocktranslateTag,
BlocktranslateCountTag,
CacheTag,
CommentTag,
FilterTag,
ForTag,
IfTag,
IfchangedTag,
LanguageTag,
LocalizeTag,
LocaltimeTag,
SpacelessTag,
TimezoneTag,
WithTag,
TagKindCount,
} TagKind;
/* one letter per kind, in TagKind order, used to serialize an open tag */
static const char tag_kind_chars[] = "bpv";
static const char tag_kind_chars[] = "bpvatTrRcmfoihlzesnw";
typedef struct {
/* the name of the tag GroupOpen pushes this kind for, or NULL */
const char *open_name;
/* The names that continue or close the group, space separated: a tag with
* one of them belongs to the group, so is no sign its end tag is missing.
* The last is the tag that closes it. */
const char *own_names;
/* The body is raw text in which no tag is read, so only the end of input
* can show that the end tag is missing. */
bool raw;
} TagGroup;
static const TagGroup tag_groups[TagKindCount] = {
[BlockTag] = { NULL, "endblock", false },
[PartialTag] = { NULL, "endpartialdef", false },
[VerbatimTag] = { NULL, "endverbatim", true },
[AutoescapeTag] = { "autoescape", "endautoescape", false },
[BlocktransTag] = { "blocktrans", "endblocktrans", false },
[BlocktransCountTag] = { NULL, "plural", false },
[BlocktranslateTag] = { "blocktranslate", "endblocktranslate", false },
[BlocktranslateCountTag] = { NULL, "plural", false },
[CacheTag] = { "cache", "endcache", false },
[CommentTag] = { "comment", "endcomment", true },
[FilterTag] = { "filter", "endfilter", false },
[ForTag] = { "for", "empty endfor", false },
[IfTag] = { "if", "elif else endif", false },
[IfchangedTag] = { "ifchanged", "else endifchanged", false },
[LanguageTag] = { "language", "endlanguage", false },
[LocalizeTag] = { "localize", "endlocalize", false },
[LocaltimeTag] = { "localtime", "endlocaltime", false },
[SpacelessTag] = { "spaceless", "endspaceless", false },
[TimezoneTag] = { "timezone", "endtimezone", false },
[WithTag] = { "with", "endwith", false },
};
typedef struct {
TagKind kind;
@@ -97,7 +171,7 @@ static void reset_scanner(struct Scanner *const scanner) {
* readable here and none of it holds a pointer to free: scanner_error
* resets before it sets has_error, which is what frees them. Zeroing the
* whole struct clears the error and the state it overlays at once. */
memset(scanner, 0, sizeof(*scanner));
*scanner = (struct Scanner) {0};
} else {
scanner->translate_seen = 0;
array_delete(&scanner->seen_kwargs);
@@ -349,12 +423,18 @@ void tree_sitter_django_external_scanner_deserialize(
}
if (name == NULL) {
// a tag starts with the letter for its kind
const char *kind = code > 0 && code < 128 ? strchr(tag_kind_chars, code) : NULL;
if (kind == NULL) {
int kind = -1;
for (int i = 0; i < TagKindCount; ++i) {
if (tag_kind_chars[i] == code) {
kind = i;
break;
}
}
if (kind < 0) {
scanner_error(scanner, "Invalid tag kind");
return;
}
OpenTag tag = { (TagKind) (kind - tag_kind_chars), array_new() };
OpenTag tag = { (TagKind) kind, array_new() };
array_push(&scanner->stack, tag);
name = &array_back(&scanner->stack)->name;
} else if (code == NAME_SEP) {
@@ -460,7 +540,7 @@ static int read_translate_option(TSLexer *const lexer) {
}
word[size] = '\0';
for (unsigned i = 0; i < sizeof(translate_options) / sizeof(*translate_options); ++i) {
if (strcmp(word, translate_options[i]) == 0) {
if (strings_equal(word, translate_options[i])) {
return (int) i;
}
}
@@ -515,6 +595,205 @@ static bool scan_kwarg_name(struct Scanner *const scanner, TSLexer *const lexer)
return true;
}
/* NAMES_INSIDE_A_TAG_GROUP in grammar.js, which this must be kept in step with:
* the names reserved to a tag group, which can therefore never continue the
* body of a group they do not belong to. */
static const char group_names[] =
"elif else empty endautoescape endblock endblocktrans endblocktranslate "
"endcache endcomment endfilter endfor endif endifchanged endlanguage "
"endlocalize endlocaltime endpartialdef endspaceless endtimezone endverbatim "
"endwith plural";
/* The longest of group_names and of the opening names, "endblocktranslate",
* is 17. */
#define GROUP_NAME_MAX 17
/* Whether `word` is one of the space-separated names in `list`. */
static bool name_in_list(const char *const word, const char *list) {
for (;;) {
const char *letter = word;
while (*letter != '\0' && *letter == *list) {
++letter;
++list;
}
if (*letter == '\0' && (*list == ' ' || *list == '\0')) {
return true;
}
while (*list != ' ' && *list != '\0') {
++list;
}
if (*list == '\0') {
return false;
}
++list;
}
}
/* The last of the space-separated names in `list`. */
static const char *last_name(const char *list) {
const char *last = list;
for (; *list != '\0'; ++list) {
if (*list == ' ') {
last = list + 1;
}
}
return last;
}
static bool names_equal(const Name *const left, const Name *const right) {
if (left->size != right->size) {
return false;
}
for (unsigned i = 0; i < left->size; ++i) {
if (left->contents[i] != right->contents[i]) {
return false;
}
}
return true;
}
/* Reads a tag's name into `word`, and reports whether it could be the name of
* a group's tag at all: one longer than any of them is not. */
static bool read_group_word(TSLexer *const lexer, char word[GROUP_NAME_MAX + 1]) {
unsigned size = 0;
while (check_name_char(lexer->lookahead)) {
if (size < GROUP_NAME_MAX) {
word[size] = (char) lexer->lookahead;
}
++size;
lexer->advance(lexer, false);
}
if (size == 0 || size > GROUP_NAME_MAX) {
return false;
}
word[size] = '';
return true;
}
/* Closes the innermost group, as its end tag does; for a counted
* blocktranslate still waiting for its plural, records the plural instead,
* which leaves the group open. */
static void close_group(struct Scanner *const scanner) {
OpenTag *const tag = array_back(&scanner->stack);
if (tag->kind == BlocktransCountTag || tag->kind == BlocktranslateCountTag) {
tag->kind = (TagKind) (tag->kind - 1);
return;
}
array_delete(&tag->name);
array_pop(&scanner->stack);
}
/* Scans GroupOpen or GroupClose, both zero width before a tag's name. Either
* may be valid where a tag starts, and nothing else of this scanner is. */
static bool scan_group_tag(
struct Scanner *const scanner,
TSLexer *const lexer,
const bool *const valid_symbols
) {
lexer->mark_end(lexer);
char word[GROUP_NAME_MAX + 1];
if (!read_group_word(lexer, word)) {
return false;
}
if (valid_symbols[GroupOpen]) {
for (unsigned kind = 0; kind < TagKindCount; ++kind) {
const char *const name = tag_groups[kind].open_name;
if (name != NULL && strings_equal(name, word)) {
OpenTag tag = { (TagKind) kind, array_new() };
array_push(&scanner->stack, tag);
lexer->result_symbol = GroupOpen;
return true;
}
}
}
if (valid_symbols[GroupClose] && scanner->stack.size > 0) {
const OpenTag *const innermost = array_back(&scanner->stack);
const char *const own_names = tag_groups[innermost->kind].own_names;
// block, partialdef and verbatim close through their pop tokens
if (innermost->kind > VerbatimTag && strings_equal(last_name(own_names), word)) {
close_group(scanner);
lexer->result_symbol = GroupClose;
return true;
}
}
return false;
}
typedef enum {
/* no marker here, and nothing has been read */
NotMissing,
/* the marker was scanned */
Missing,
/* no marker, but input was read to decide that, so nothing else may be
* scanned from here */
NotMissingAfterReading,
} MissingResult;
/* Scans the zero-width marker for the innermost group's missing tag. The
* marker is valid only where that group could close, which is after a body,
* where nothing else of this scanner is valid but the raw-text tokens, and
* those are scanned only when this returns NotMissing.
*
* The tag is taken to be missing at the end of input; before a tag whose name
* is reserved to some other group, since it cannot be part of this group's
* body; and before an endblock or endpartialdef whose name is not the innermost
* open one but one further out, which it closes instead. The marker then
* closes the group as the missing tag would have. */
static MissingResult scan_missing_tag(struct Scanner *const scanner, TSLexer *const lexer) {
if (scanner->stack.size == 0) {
// the stack has lost track of the group, so leave it to the grammar
return NotMissing;
}
const OpenTag *const innermost = array_back(&scanner->stack);
const TagGroup *const group = &tag_groups[innermost->kind];
lexer->mark_end(lexer);
if (!lexer->eof(lexer)) {
if (group->raw || lexer->lookahead != '{') {
return NotMissing;
}
lexer->advance(lexer, false);
if (lexer->lookahead != '%') {
return NotMissingAfterReading;
}
lexer->advance(lexer, false);
skip_whitespace(lexer);
char word[GROUP_NAME_MAX + 1];
if (!read_group_word(lexer, word)) {
return NotMissingAfterReading;
}
if (name_in_list(word, group->own_names)) {
// the group's own tag, unless it names a block further out
if (innermost->kind != BlockTag && innermost->kind != PartialTag) {
return NotMissingAfterReading;
}
if (!check_space(lexer->lookahead) || scanner->stack.size < 2) {
return NotMissingAfterReading;
}
skip_whitespace(lexer);
Name name = array_new();
bool closes_outer = false;
if (read_name(lexer, &name) && !names_equal(&innermost->name, &name)) {
for (unsigned i = scanner->stack.size - 1; i-- > 0;) {
const OpenTag *const tag = array_get(&scanner->stack, i);
if (tag->kind == innermost->kind && names_equal(&tag->name, &name)) {
closes_outer = true;
break;
}
}
}
array_delete(&name);
if (!closes_outer) {
return NotMissingAfterReading;
}
} else if (!name_in_list(word, group_names)) {
return NotMissingAfterReading;
}
}
close_group(scanner);
lexer->result_symbol = MissingTag;
return Missing;
}
const char endverbatim_chars[] = "endverbatim";
/* Called right after consuming "{%"; checks whether what follows closes the
@@ -669,6 +948,22 @@ bool tree_sitter_django_external_scanner_scan(
if (valid_symbols[MatcherError]) {
return false;
}
/* Above the raw-text scanners, which find nothing at the end of an empty
* body, and below the MatcherError check, which keeps these from being
* scanned during error recovery. */
if (valid_symbols[MissingTag]) {
switch (scan_missing_tag(scanner, lexer)) {
case Missing:
return true;
case NotMissingAfterReading:
return false;
case NotMissing:
break;
}
}
if (valid_symbols[GroupOpen] || valid_symbols[GroupClose]) {
return scan_group_tag(scanner, lexer, valid_symbols);
}
if (valid_symbols[VerbatimContent]) {
return scan_verbatim_content(scanner, lexer);
}
@@ -710,6 +1005,13 @@ bool tree_sitter_django_external_scanner_scan(
return false;
}
scanner->translate_seen |= bit;
if (option == 1 && scanner->stack.size > 0) {
// "count" makes the group one that waits for a plural
OpenTag *const tag = array_back(&scanner->stack);
if (tag->kind == BlocktransTag || tag->kind == BlocktranslateTag) {
tag->kind = (TagKind) (tag->kind + 1);
}
}
lexer->result_symbol = TranslateOption;
return true;
}
@@ -729,7 +1031,8 @@ bool tree_sitter_django_external_scanner_scan(
/* Only the innermost open tag can be closed here, so it is the one and only
* candidate: there is never a second name to try, and so never any input to
* reread. */
if (scanner->stack.size > 0) {
// only the kinds that match a name have a pop token
if (scanner->stack.size > 0 && array_back(&scanner->stack)->kind <= VerbatimTag) {
OpenTag *tag = array_back(&scanner->stack);
enum TokenType token = pop_token_for_kind(tag->kind);
if (valid_symbols[token]) {
test/corpus/blocktranslate.txt
+15 -1
View File
@@ -349,13 +349,27 @@ Blocktranslate Closed By The Other Spelling
==================
Blocktrans With Count And No Plural
:error
==================
{% blocktrans count n=x %}one{% endblocktrans %}
---
(template
(content)
(blocktranslate_group
(blocktrans_clause
(blocktrans_block
(identifier)
(filtered_value
(value
(variable_attribute
(identifier)))))
(content)
(missing_plural_block))
(endblocktrans_block))
(content))
==================
test/corpus/l10n.txt
+9 -1
View File
@@ -115,13 +115,21 @@ Localize With Two Arguments
==================
Localize Unclosed
:error
==================
{% localize %}
---
(template
(content)
(localize_group
(localize_clause
(localize_block)
(template
(content)))
(missing_endlocalize_block)))
==================
Localize Filter
test/corpus/missing.txt
+503
View File
@@ -0,0 +1,503 @@
==================
Missing Endif At End Of Input
==================
{% if a %}x
---
(template
(content)
(if_group
(if_clause
(if_block
(predicate
(filtered_value
(value
(variable_attribute
(identifier))))))
(template
(content)))
(missing_endif_block)))
==================
Missing Endfor After Empty
==================
{% for x in y %}a{% empty %}b
---
(template
(content)
(for_group
(for_clause
(for_block
(identifier)
(filtered_value
(value
(variable_attribute
(identifier)))))
(template
(content)))
(empty_clause
(empty_block)
(template
(content)))
(missing_endfor_block)))
==================
Missing Endblock At End Of Input
==================
{% block content %}x
---
(template
(content)
(block_group
(block_clause
(block_block
(push_block))
(template
(content)))
(missing_endblock_block)))
==================
Missing Endpartialdef At End Of Input
==================
{% partialdef p %}x
---
(template
(content)
(partialdef_group
(partialdef_clause
(partialdef_block
(push_partial))
(template
(content)))
(missing_endpartialdef_block)))
==================
Missing Endverbatim With An Empty Body
==================
{% verbatim %}
---
(template
(content)
(verbatim_group
(verbatim_clause
(verbatim_block
(push_verbatim))
(verbatim_content))
(missing_endverbatim_block)))
==================
Missing Endverbatim After A Tag In Its Text
==================
{% verbatim %}{% endif %}x
---
(template
(content)
(verbatim_group
(verbatim_clause
(verbatim_block
(push_verbatim))
(verbatim_content))
(missing_endverbatim_block)))
==================
Missing Endcomment At End Of Input
==================
{% comment %}x
---
(template
(content)
(comment_group
(comment_clause
(comment_block)
(comment_content))
(missing_endcomment_block)))
==================
Missing Endif Before An Outer End Tag
==================
{% for x in y %}{% if a %}{% endfor %}
---
(template
(content)
(for_group
(for_clause
(for_block
(identifier)
(filtered_value
(value
(variable_attribute
(identifier)))))
(template
(if_group
(if_clause
(if_block
(predicate
(filtered_value
(value
(variable_attribute
(identifier)))))))
(missing_endif_block))))
(endfor_block))
(content))
==================
Missing Endfor Before An Outer Else
==================
{% if a %}{% for x in y %}{% else %}b{% endif %}
---
(template
(content)
(if_group
(if_clause
(if_block
(predicate
(filtered_value
(value
(variable_attribute
(identifier))))))
(template
(for_group
(for_clause
(for_block
(identifier)
(filtered_value
(value
(variable_attribute
(identifier))))))
(missing_endfor_block))))
(else_clause
(else_block)
(template
(content)))
(endif_block))
(content))
==================
Missing Endwith Before An Outer Endblock
==================
{% block a %}{% with b=1 %}{% endblock a %}
---
(template
(content)
(block_group
(block_clause
(block_block
(push_block))
(template
(with_group
(with_clause
(with_block
(identifier)
(filtered_value
(value
(literal
(number))))))
(missing_endwith_block))))
(endblock_block
(pop_block)))
(content))
==================
Missing Two End Tags At End Of Input
==================
{% for x in y %}{% if a %}
---
(template
(content)
(for_group
(for_clause
(for_block
(identifier)
(filtered_value
(value
(variable_attribute
(identifier)))))
(template
(if_group
(if_clause
(if_block
(predicate
(filtered_value
(value
(variable_attribute
(identifier))))))
(template
(content)))
(missing_endif_block))))
(missing_endfor_block)))
==================
Missing Endblock Before An Outer Name
==================
{% block a %}{% block b %}{% endblock a %}
---
(template
(content)
(block_group
(block_clause
(block_block
(push_block))
(template
(block_group
(block_clause
(block_block
(push_block)))
(missing_endblock_block))))
(endblock_block
(pop_block)))
(content))
==================
Missing Endpartialdef Before An Outer Name
==================
{% partialdef a %}{% partialdef b %}{% endpartialdef a %}
---
(template
(content)
(partialdef_group
(partialdef_clause
(partialdef_block
(push_partial))
(template
(partialdef_group
(partialdef_clause
(partialdef_block
(push_partial)))
(missing_endpartialdef_block))))
(endpartialdef_block
(pop_partial)))
(content))
==================
Missing Plural At End Of Input
==================
{% blocktrans count n=x %}one
---
(template
(content)
(blocktranslate_group
(blocktrans_clause
(blocktrans_block
(identifier)
(filtered_value
(value
(variable_attribute
(identifier)))))
(content)
(missing_plural_block))
(missing_endblocktrans_block)))
==================
Missing Plural Before Endblocktranslate
==================
{% blocktranslate count n=x %}one{% endblocktranslate %}
---
(template
(content)
(blocktranslate_group
(blocktranslate_clause
(blocktranslate_block
(identifier)
(filtered_value
(value
(variable_attribute
(identifier)))))
(content)
(missing_plural_block))
(endblocktranslate_block))
(content))
==================
Missing Endblocktrans After Plural
==================
{% blocktrans count n=x %}one{% plural %}many
---
(template
(content)
(blocktranslate_group
(blocktrans_clause
(blocktrans_block
(identifier)
(filtered_value
(value
(variable_attribute
(identifier)))))
(content)
(plural_clause
(plural_block)
(content)))
(missing_endblocktrans_block)))
==================
Missing Endfor Before A Tag Without Spaces
==================
{% if a %}{% for x in y %}{%endif%}
---
(template
(content)
(if_group
(if_clause
(if_block
(predicate
(filtered_value
(value
(variable_attribute
(identifier))))))
(template
(for_group
(for_clause
(for_block
(identifier)
(filtered_value
(value
(variable_attribute
(identifier))))))
(missing_endfor_block))))
(endif_block))
(content))
==================
Missing Endfor Before A Tag Across Lines
==================
{% if a %}{% for x in y %}{%
endif %}
---
(template
(content)
(if_group
(if_clause
(if_block
(predicate
(filtered_value
(value
(variable_attribute
(identifier))))))
(template
(for_group
(for_clause
(for_block
(identifier)
(filtered_value
(value
(variable_attribute
(identifier))))))
(missing_endfor_block))))
(endif_block))
(content))
==================
Missing Endif After A Custom End Tag
==================
{% if a %}{% endmytag %}
---
(template
(content)
(if_group
(if_clause
(if_block
(predicate
(filtered_value
(value
(variable_attribute
(identifier))))))
(template
(custom_tag_block
(identifier))
(content)))
(missing_endif_block)))
==================
Stray End Tag After A Missing One
:error
==================
{% if a %}{% endfor %}
---
==================
Endblock Naming No Open Block
:error
==================
{% block a %}{% block b %}{% endblock c %}
---
==================
Second Else Is Not A Missing Tag
:error
==================
{% if a %}{% else %}{% else %}{% endif %}
---
==================
Plural Without Count Is Not A Missing Tag
:error
==================
{% blocktrans %}one{% plural %}many{% endblocktrans %}
---