support blocktrans

CLAUDE.md

@@ -130,15 +130,19 @@ is accepted: these names are common enough that shadowing them is the mistake.
 
 ### External scanner (`src/scanner.c`)
 
-Eleven external tokens, for the constraints a context-free grammar cannot express:
+Twelve external tokens, for the constraints a context-free grammar cannot express:
 
 - `push_block`/`push_partial`/`push_verbatim` and the matching `pop_*` — name matching for
   `{% block a %}…{% endblock a %}`. They share **one** stack of `{kind, name}` entries, not one stack per
   kind: the grammar already guarantees nesting, so only the innermost open tag can be closed, and there is
   never a second candidate name to try. Serialized as `<kind><name> ` per entry (`tag_kind_chars`).
 - `verbatim_content`, `comment_content` — raw text up to the matching close tag.
-- `_bt_open`, `_bt_option` — which options a `{% blocktranslate %}` has already been given, so that a
-  repeated one is refused. Both are zero width and hidden, so no tree changes. The state is one byte of
+- `_tag_open`, `_bt_option`, `_kwarg_name` — what a tag has already been given, so that a repeated
+  argument is refused: `_bt_option` for `{% blocktranslate %}`'s five options, `_kwarg_name` for the
+  `name=value` arguments of a tag registered with `simple_tag`, whose names are open-ended and so are kept
+  as names rather than as a mask. `_kwarg_name` returns nothing unless the name is followed by `=`, which
+  is also what tells an argument apart from the `as name` clause or the end of the tag. `_tag_open` clears
+  both, and every rule that uses either has one. Both are zero width and hidden, so no tree changes. The state is one byte of
   flags, not a stack, because the tag cannot nest: its body admits no tags. It lives outside the
   `stack`/`error` union, is cleared in both branches of `reset_scanner`, and is serialized after the
   status character, so a GLR stack split copies it like everything else.
@@ -154,9 +158,11 @@ Block-name matching does — it decides which close tag closes what.
 and the reason is worth keeping straight: its five options are a fixed set, so uniqueness _is_ expressible —
 `arrangements` expresses exactly that — and the scanner is standing in for a grammar that would be correct
 but takes minutes to build. Where the set is open, as with `{% querystring %}`'s `**kwargs`, no grammar
-can express it — but the scanner still could, by remembering the names a tag has been given rather than a
-fixed mask of them. That is not done, and it is what the `{% querystring page=2 page=3 %}` divergence below
-is waiting on.
+can express it at all, and the scanner remembers the names the tag has been given instead of a mask of
+known ones. Which tags get that guard is itself a differential question: only `parse_bits` refuses a
+repeat, so `{% with a=1 a=2 %}`, `{% include "t" with a=1 a=2 %}` and `{% blocktranslate with a=1 a=2 %}`
+are all accepted by Django — `token_kwargs` builds a dict and the later value simply wins — and must not
+be guarded.
 
 Zero-width guards carry one hazard: `recover_with_missing` can supply one without the scanner running. The
 `if (valid_symbols[MatcherError]) return false;` near the top of `scan` is what stops a guard being scanned
@@ -210,8 +216,6 @@ documentation or memory. Both checks are worth repeating whenever tags, filters
 
 - Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal,
   so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here.
-- `{% querystring page=2 page=3 %}` (a repeated `**kwargs` key) is a Django error but is not expressible in
-  a context-free grammar. Repeats _are_ rejected wherever the signature names its arguments.
 - `{% csp_nonce_attr "a" media="b" %}` is accepted, matching Django, which only rejects it at render time.
 - A misspelled tag or filter cannot be caught: `{{ x|lenght }}` is indistinguishable from a filter some
   library registered. Cross-referencing names against `{% load %}` needs the project's Python, so it

grammar.js

@@ -137,7 +137,12 @@ function simpleTag($, tag, args = true, kwargs = true) {
     parts.push(optional(rule));
   }
   if (kwargs === true) {
-    parts.push(repeat(part(seq($.identifier, "=", $.filtered_value))));
+    // parse_bits refuses a keyword argument the tag already holds, whatever
+    // **kwargs it takes, so the scanner keeps the names rather than the grammar
+    parts.unshift($._tag_open);
+    parts.push(
+      repeat(part(seq($._kwarg_name, $.identifier, "=", $.filtered_value))),
+    );
   } else if (kwargs) {
     // a known signature: any of the names it accepts, in any order, none twice
     const named = Object.entries(kwargs).map(([name, value]) =>
@@ -302,8 +307,9 @@ const django = grammar({
     $.push_verbatim,
     $.verbatim_content,
     $.comment_content,
-    $._bt_open,
+    $._tag_open,
     $._bt_option,
+    $._kwarg_name,
   ],
   reserved: {
     global: ($) => ["not", "if", "in", "is", "as", "for", "from"],
@@ -492,14 +498,14 @@ const django = grammar({
         ...["blocktrans", "blocktranslate"].map((name) =>
           choice(
             seq(
-              block(name, $._bt_open, repeat($._translate_option)),
+              block(name, $._tag_open, repeat($._translate_option)),
               optional($._translate_body),
               block("end" + name),
             ),
             seq(
               block(
                 name,
-                $._bt_open,
+                $._tag_open,
                 repeat($._translate_option),
                 $._translate_count,
                 repeat($._translate_option),

src/grammar.json

@@ -1383,7 +1383,7 @@
                     },
                     {
                       "type": "SYMBOL",
-                      "name": "_bt_open"
+                      "name": "_tag_open"
                     },
                     {
                       "type": "REPEAT",
@@ -1501,7 +1501,7 @@
                     },
                     {
                       "type": "SYMBOL",
-                      "name": "_bt_open"
+                      "name": "_tag_open"
                     },
                     {
                       "type": "REPEAT",
@@ -1684,7 +1684,7 @@
                     },
                     {
                       "type": "SYMBOL",
-                      "name": "_bt_open"
+                      "name": "_tag_open"
                     },
                     {
                       "type": "REPEAT",
@@ -1802,7 +1802,7 @@
                     },
                     {
                       "type": "SYMBOL",
-                      "name": "_bt_open"
+                      "name": "_tag_open"
                     },
                     {
                       "type": "REPEAT",
@@ -2688,6 +2688,10 @@
             "context_name": "tag_name"
           }
         },
+        {
+          "type": "SYMBOL",
+          "name": "_tag_open"
+        },
         {
           "type": "REPEAT",
           "content": {
@@ -2716,6 +2720,10 @@
               {
                 "type": "SEQ",
                 "members": [
+                  {
+                    "type": "SYMBOL",
+                    "name": "_kwarg_name"
+                  },
                   {
                     "type": "SYMBOL",
                     "name": "identifier"
@@ -5664,6 +5672,10 @@
             "value": "querystring"
           }
         },
+        {
+          "type": "SYMBOL",
+          "name": "_tag_open"
+        },
         {
           "type": "REPEAT",
           "content": {
@@ -5692,6 +5704,10 @@
               {
                 "type": "SEQ",
                 "members": [
+                  {
+                    "type": "SYMBOL",
+                    "name": "_kwarg_name"
+                  },
                   {
                     "type": "SYMBOL",
                     "name": "identifier"
@@ -7869,11 +7885,15 @@
     },
     {
       "type": "SYMBOL",
-      "name": "_bt_open"
+      "name": "_tag_open"
     },
     {
       "type": "SYMBOL",
       "name": "_bt_option"
+    },
+    {
+      "type": "SYMBOL",
+      "name": "_kwarg_name"
     }
   ],
   "inline": [],

src/scanner.c

@@ -21,11 +21,14 @@ enum TokenType {
   VerbatimContent,
   /* raw text inside a comment block, up through (but excluding) the endcomment tag */
   CommentContent,
-  /* zero width, at the start of a blocktranslate tag: forget the options seen */
-  TranslateOpen,
+  /* zero width, at the start of a tag: forget what the tag before it held */
+  TagOpen,
   /* zero width, before an option of a blocktranslate tag: record it, and fail
    * if that option has already been written once */
   TranslateOption,
+  /* zero width, before the name of a "name=value" argument: record the name,
+   * and fail if the tag has already been given one by that name */
+  KwargName,
 };
 
 /* The options blocktranslate takes, in the order of the bits recording them.
@@ -66,6 +69,11 @@ struct Scanner {
    * translate_options. Kept out of the union below, whose members share their
    * storage with each other. */
   uint8_t translate_seen;
+  /* the "name=value" argument names the tag being read has been given, each
+   * followed by NAME_SEP. parse_bits refuses a repeated keyword argument, and
+   * the set of names is open, so this remembers the names rather than a mask
+   * of known ones. Kept out of the union, as translate_seen is. */
+  Name seen_kwargs;
   union {
     Stack stack;
     char error [ERROR_SIZE];
@@ -82,6 +90,7 @@ static enum TokenType pop_token_for_kind(TagKind kind) {
 
 static void reset_scanner(struct Scanner *const scanner) {
   scanner->translate_seen = 0;
+  array_delete(&scanner->seen_kwargs);
   if (scanner->has_error) {
     scanner->error[0] = '\0';
     scanner->has_error = false;
@@ -160,6 +169,17 @@ write_serialization_error:;
   iter += write_code(scanner->has_error + '0', iter);
   // five options fit in five bits, which stays inside printable ASCII
   iter += write_code(scanner->translate_seen + '0', iter);
+  for (unsigned i = 0; i < scanner->seen_kwargs.size; ++i) {
+    int32_t code = *array_get(&scanner->seen_kwargs, i);
+    unsigned write_amt = write_code(code, iter);
+    if (write_amt == 0) {
+      scanner_error(scanner, "bad code from keyword argument name");
+      goto write_serialization_error;
+    }
+    iter += write_amt;
+  }
+  // a name holds no newline, so this ends the list without escaping anything
+  iter += write_code('\n', iter);
   if (scanner->has_error) {
     for (int i = 0; i < ERROR_SIZE; ++i) {
       char value = scanner->error[i];
@@ -266,6 +286,29 @@ void tree_sitter_django_external_scanner_deserialize(
   }
   scanner->translate_seen = (uint8_t) (seen - '0');
 
+  for (;;) {
+    int32_t item;
+    bytes_read = read_code(buffer, &item);
+    if (!bytes_read) {
+      scanner_error(scanner, "received invalid utf8 byte");
+      return;
+    }
+    consumed += bytes_read;
+    if (consumed > length) {
+      scanner_error(scanner, "keyword argument names truncated by length");
+      return;
+    }
+    buffer += bytes_read;
+    if (item == '\n') {
+      break;
+    }
+    if (item != NAME_SEP && !check_name_char(item)) {
+      scanner_error(scanner, "invalid character in keyword argument name");
+      return;
+    }
+    array_push(&scanner->seen_kwargs, item);
+  }
+
   switch (code) {
     default:
       scanner_error(scanner, "received unrecognized scanner status");
@@ -418,6 +461,54 @@ static int read_translate_option(TSLexer *const lexer) {
   return -1;
 }
 
+/* Reads the name of a "name=value" argument, and reports whether the tag can
+ * still take it. A name not followed by "=" is not one of these arguments at
+ * all, which is what tells this apart from the "as name" clause or the end of
+ * the tag; the name is only kept once the "=" has been seen.
+ *
+ * The names are held end to end, each followed by NAME_SEP, which no name
+ * holds. The candidate is appended first and dropped again if it turns out to
+ * be a repeat, so nothing is kept unless the token is returned. */
+static bool scan_kwarg_name(struct Scanner *const scanner, TSLexer *const lexer) {
+  if (!check_name_start_char(lexer->lookahead)) {
+    return false;
+  }
+  const unsigned start = scanner->seen_kwargs.size;
+  while (check_name_char(lexer->lookahead)) {
+    array_push(&scanner->seen_kwargs, lexer->lookahead);
+    lexer->advance(lexer, false);
+  }
+  const unsigned size = scanner->seen_kwargs.size - start;
+  if (lexer->lookahead != '=') {
+    scanner->seen_kwargs.size = start;
+    return false;
+  }
+  for (unsigned i = 0; i < start;) {
+    unsigned end = i;
+    while (end < start && *array_get(&scanner->seen_kwargs, end) != NAME_SEP) {
+      ++end;
+    }
+    if (end - i == size) {
+      bool same = true;
+      for (unsigned j = 0; j < size; ++j) {
+        if (*array_get(&scanner->seen_kwargs, i + j)
+            != *array_get(&scanner->seen_kwargs, start + j)) {
+          same = false;
+          break;
+        }
+      }
+      if (same) {
+        // the tag already holds an argument by this name
+        scanner->seen_kwargs.size = start;
+        return false;
+      }
+    }
+    i = end + 1;
+  }
+  array_push(&scanner->seen_kwargs, NAME_SEP);
+  return true;
+}
+
 const char endverbatim_chars[] = "endverbatim";
 
 /* Called right after consuming "{%"; checks whether what follows closes the
@@ -582,10 +673,19 @@ bool tree_sitter_django_external_scanner_scan(
    * read. They belong above the loop below, which skips the separator the
    * grammar still has to match, and below the MatcherError check, which is what
    * keeps them from being scanned during error recovery. */
-  if (valid_symbols[TranslateOpen]) {
+  if (valid_symbols[TagOpen]) {
     lexer->mark_end(lexer);
     scanner->translate_seen = 0;
-    lexer->result_symbol = TranslateOpen;
+    array_delete(&scanner->seen_kwargs);
+    lexer->result_symbol = TagOpen;
+    return true;
+  }
+  if (valid_symbols[KwargName]) {
+    lexer->mark_end(lexer);
+    if (!scan_kwarg_name(scanner, lexer)) {
+      return false;
+    }
+    lexer->result_symbol = KwargName;
     return true;
   }
   if (valid_symbols[TranslateOption]) {

test/corpus/custom.txt

@@ -284,3 +284,64 @@ A Name Reserved For A Tag Is Still An Argument
         (variable_attribute
           (identifier)))))
   (content))
+
+==================
+Custom Tag With A Keyword Argument Twice
+:error
+==================
+
+{% mytag a=1 a=2 %}
+
+---
+
+
+==================
+Custom Tag With Keyword Arguments And As
+==================
+
+{% mytag a=1 b=2 as out %}
+
+---
+
+(template
+  (content)
+  (custom_tag
+    tag: (identifier)
+    (identifier)
+    (filtered_value
+      (value
+        (literal
+          (number))))
+    (identifier)
+    (filtered_value
+      (value
+        (literal
+          (number))))
+    variable: (identifier))
+  (content))
+
+==================
+Custom Tag Repeating A Keyword Of The Tag Before It
+==================
+
+{% mytag page=1 %}{% mytag page=2 %}
+
+---
+
+(template
+  (content)
+  (custom_tag
+    tag: (identifier)
+    (identifier)
+    (filtered_value
+      (value
+        (literal
+          (number)))))
+  (custom_tag
+    tag: (identifier)
+    (identifier)
+    (filtered_value
+      (value
+        (literal
+          (number)))))
+  (content))

test/corpus/querystring.txt

@@ -74,3 +74,45 @@ Querystring With Expression Argument And As
           (number))))
     variable: (identifier))
   (content))
+
+==================
+Querystring With Two Keys
+==================
+
+{% querystring page=2 size=3 %}
+
+---
+
+(template
+  (content)
+  (query_string
+    (identifier)
+    (filtered_value
+      (value
+        (literal
+          (number))))
+    (identifier)
+    (filtered_value
+      (value
+        (literal
+          (number)))))
+  (content))
+
+==================
+Querystring With A Key Twice
+:error
+==================
+
+{% querystring page=2 page=3 %}
+
+---
+
+
+==================
+Querystring With A Key Repeated Later
+:error
+==================
+
+{% querystring a=1 b=2 a=3 %}
+
+---