enforce key uniqueness on with expression, fix scanner.c

This commit is contained in:
danhanson committed 2026-09-07 20:33:43 -05:00
1 parent 6105fd9002
commit 356cc3a885
10 files changed
+53165 -53162

No files matched your search

+16 -9
View File
@@ -57,8 +57,9 @@ that place it:
positional filter expressions, then `name=value` keywords, then `as name`. `args`: `true` (`*args`), a
count (at most N), or `false`. `kwargs`: `true` (`**kwargs`), `false`, or `{name: rule}` for a known
signature. Put the Python signature in a comment above the call site.
- `arrangements(items)` — every ordered subset of `items`, used by `simpleTag` so that known keyword
arguments may appear in any order but never twice. Grows as `Σ C(n,k)·k!` (65 for four names), which is
- `arrangements(items)` — every ordered subset of `items`, so that a tag's options may be written in any
order but never twice. Used by `translate` and `include`; keyword arguments are guarded by the scanner
instead. Grows as `Σ C(n,k)·k!` (65 for four names), which is
why it is only used for signatures that name their arguments.
Consequences worth knowing before editing a rule:
@@ -138,9 +139,10 @@ Twelve external tokens, for the constraints a context-free grammar cannot expres
never a second candidate name to try. Serialized as `<kind><name> ` per entry (`tag_kind_chars`).
- `verbatim_content`, `comment_content` — raw text up to the matching close tag.
- `_tag_open`, `_bt_option`, `_kwarg_name` — what a tag has already been given, so that a repeated
argument is refused: `_bt_option` for `{% blocktranslate %}`'s five options, `_kwarg_name` for the
`name=value` arguments of a tag registered with `simple_tag`, whose names are open-ended and so are kept
as names rather than as a mask. `_kwarg_name` returns nothing unless the name is followed by `=`, which
argument is refused: `_bt_option` for `{% blocktranslate %}`'s five options, `_kwarg_name` for `name=value`
arguments — those of a tag registered with `simple_tag`, and the `with` bindings of `{% with %}`,
`{% include %}` and `{% blocktranslate %}` — whose names are open-ended and so are kept as names rather
than as a mask. `_kwarg_name` returns nothing unless the name is followed by `=`, which
is also what tells an argument apart from the `as name` clause or the end of the tag. `_tag_open` clears
both, and every rule that uses either has one. Both are zero width and hidden, so no tree changes. The state is one byte of
flags, not a stack, because the tag cannot nest: its body admits no tags. It lives outside the
@@ -159,10 +161,12 @@ and the reason is worth keeping straight: its five options are a fixed set, so u
`arrangements` expresses exactly that — and the scanner is standing in for a grammar that would be correct
but takes minutes to build. Where the set is open, as with `{% querystring %}`'s `**kwargs`, no grammar
can express it at all, and the scanner remembers the names the tag has been given instead of a mask of
known ones. Which tags get that guard is itself a differential question: only `parse_bits` refuses a
repeat, so `{% with a=1 a=2 %}`, `{% include "t" with a=1 a=2 %}` and `{% blocktranslate with a=1 a=2 %}`
are all accepted by Django — `token_kwargs` builds a dict and the later value simply wins — and must not
be guarded.
known ones. Only `parse_bits` actually refuses a repeat: the `with` bindings of
`{% with %}`, `{% include %}` and `{% blocktranslate %}` all go through `token_kwargs`, which builds a dict
and lets the later value win. They are guarded here anyway, as a deliberate divergence — writing a name
twice has no use and is a mistake worth reporting. `{% blocktranslate %}`'s `count` is not part of that,
because Django keeps it in a dict of its own, so `{% blocktranslate with a=1 count a=2 %}` is accepted here
as it is there.
Zero-width guards carry one hazard: `recover_with_missing` can supply one without the scanner running. The
`if (valid_symbols[MatcherError]) return false;` near the top of `scan` is what stops a guard being scanned
@@ -216,6 +220,9 @@ documentation or memory. Both checks are worth repeating whenever tags, filters
- Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal,
so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here.
- A `with` binding written twice (`{% with a=1 a=2 %}`, and the same in `{% include %}` and
`{% blocktranslate %}`) is an error here and not in Django, where `token_kwargs` builds a dict and the
later value wins. It has no use, so it is treated as the mistake it almost certainly is.
- `{% csp_nonce_attr "a" media="b" %}` is accepted, matching Django, which only rejects it at render time.
- A misspelled tag or filter cannot be caught: `{{ x|lenght }}` is indistinguishable from a filter some
library registered. Cross-referencing names against `{% load %}` needs the project's Python, so it