add blocktrans

CLAUDE.md

@@ -106,8 +106,7 @@ which Django itself enforces while it parses the template ("add requires 2 argum
 ### First-party libraries
 
 Django ships `i18n`, `l10n`, `static`, `cache` and `tz` as libraries rather than builtins. Their tags are
-modelled as ordinary tags (so far: `static`, `l10n`, `tz`, `cache`, and `i18n` apart from
-`blocktrans`/`blocktranslate`), and the grammar **never requires the `{% load %}`**, because
+modelled as ordinary tags (so far: `static`, `l10n`, `tz`, `cache` and `i18n`), and the grammar **never requires the `{% load %}`**, because
 an engine can preload a library through `OPTIONS: {"builtins": [...]}`, which makes `{% static "a" %}` valid
 with no load at all.
 
@@ -131,29 +130,50 @@ is accepted: these names are common enough that shadowing them is the mistake.
 
 ### External scanner (`src/scanner.c`)
 
-Nine external tokens, for the constraints a context-free grammar cannot express:
+Eleven external tokens, for the constraints a context-free grammar cannot express:
 
 - `push_block`/`push_partial`/`push_verbatim` and the matching `pop_*` — name matching for
   `{% block a %}…{% endblock a %}`. They share **one** stack of `{kind, name}` entries, not one stack per
   kind: the grammar already guarantees nesting, so only the innermost open tag can be closed, and there is
   never a second candidate name to try. Serialized as `<kind><name> ` per entry (`tag_kind_chars`).
 - `verbatim_content`, `comment_content` — raw text up to the matching close tag.
+- `_bt_open`, `_bt_option` — which options a `{% blocktranslate %}` has already been given, so that a
+  repeated one is refused. Both are zero width and hidden, so no tree changes. The state is one byte of
+  flags, not a stack, because the tag cannot nest: its body admits no tags. It lives outside the
+  `stack`/`error` union, is cleared in both branches of `reset_scanner`, and is serialized after the
+  status character, so a GLR stack split copies it like everything else.
 - `matcher_error` — used by no rule; returned only if the scanner reaches an error state.
 
 `check_space()` defines whitespace for the scanner and **must stay in sync with `SEP`**, because the
 scanner skips whitespace itself while matching names.
 
 Prefer the grammar over the scanner. A useful test: does the constraint change how the input _parses_?
-Block-name matching does (it decides which close tag closes what). Things like "this keyword may not repeat"
-do not, and belong in a linter instead.
+Block-name matching does — it decides which close tag closes what.
+
+"This keyword may not repeat" normally does not, and belongs in a linter. `blocktranslate` is the exception,
+and the reason is worth keeping straight: its five options are a fixed set, so uniqueness _is_ expressible —
+`arrangements` expresses exactly that — and the scanner is standing in for a grammar that would be correct
+but takes minutes to build. Where the set is open, as with `{% querystring %}`'s `**kwargs`, no grammar
+can express it — but the scanner still could, by remembering the names a tag has been given rather than a
+fixed mask of them. That is not done, and it is what the `{% querystring page=2 page=3 %}` divergence below
+is waiting on.
+
+Zero-width guards carry one hazard: `recover_with_missing` can supply one without the scanner running. The
+`if (valid_symbols[MatcherError]) return false;` near the top of `scan` is what stops a guard being scanned
+during tree-sitter's mark-everything-valid recovery pass, so new guards go **below** it — and above the
+whitespace loop, which skips the separator the grammar still has to match. A guard must also never be valid
+where `content` is: a zero-width token at a content boundary preempts the internal lexer and `content` stops
+matching.
 
 ### Queries
 
 `queries/highlights.scm` lists tag and filter names explicitly, so adding either to the grammar means
 adding it there too; filter names appear there without their colon. A name the grammar does not know is
-captured through its field instead (`(filter name: (identifier))`, `(custom_tag tag: (identifier))`). `queries/libraries.scm` is not a query editors run themselves; it is data for a linter, and every tag
-or filter added from a library belongs in it. `queries/locals.scm` relies on the `variable:` field to tell a binding from a
-reference — another reason to keep `asVariable` uniform.
+captured through its field instead (`(filter name: (identifier))`, `(custom_tag tag: (identifier))`).
+
+`queries/libraries.scm` is not a query editors run themselves; it is data for a linter, and every tag or
+filter added from a library belongs in it. `queries/locals.scm` relies on the `variable:` field to tell a
+binding from a reference — another reason to keep `asVariable` uniform.
 
 ## Testing
 

grammar.js

@@ -255,6 +255,8 @@ const NAMES_INSIDE_A_TAG_GROUP = [
   "empty",
   "endautoescape",
   "endblock",
+  "endblocktrans",
+  "endblocktranslate",
   "endcache",
   "endcomment",
   "endfilter",
@@ -269,6 +271,7 @@ const NAMES_INSIDE_A_TAG_GROUP = [
   "endtimezone",
   "endverbatim",
   "endwith",
+  "plural",
 ];
 
 /** Django's string constant, which takes any escape but a line break. */
@@ -284,6 +287,7 @@ const django = grammar({
     [$.template],
     [$.predicate],
     [$.library, $.load],
+    [$._translate_option],
     [$._filtered_value_spaced],
     [$._filter_expression_spaced],
   ],
@@ -298,6 +302,8 @@ const django = grammar({
     $.push_verbatim,
     $.verbatim_content,
     $.comment_content,
+    $._bt_open,
+    $._bt_option,
   ],
   reserved: {
     global: ($) => ["not", "if", "in", "is", "as", "for", "from"],
@@ -388,6 +394,7 @@ const django = grammar({
       choice(
         $.autoescape_group,
         $.block_group,
+        $.blocktranslate_group,
         $.cache_group,
         $.comment_group,
         $.csp_nonce_attr,
@@ -480,6 +487,60 @@ const django = grammar({
         optional($.template),
         block("endblock", $.pop_block),
       ),
+    blocktranslate_group: ($) =>
+      choice(
+        ...["blocktrans", "blocktranslate"].map((name) =>
+          choice(
+            seq(
+              block(name, $._bt_open, repeat($._translate_option)),
+              optional($._translate_body),
+              block("end" + name),
+            ),
+            seq(
+              block(
+                name,
+                $._bt_open,
+                repeat($._translate_option),
+                $._translate_count,
+                repeat($._translate_option),
+              ),
+              $._translate_body,
+              block("plural"),
+              optional($._translate_body),
+              block("end" + name),
+            ),
+          ),
+        ),
+      ),
+    /**
+     * The scanner reads the name of the option before the grammar does, so it
+     * can refuse one the tag has already been given: do_block_translate keeps
+     * the options it has seen in a dict and raises on a repeat.
+     */
+    _translate_option: ($) =>
+      seq(
+        $._bt_option,
+        choice(
+          seq(part("with"), $._translate_kwargs),
+          part("context", $.filtered_value),
+          part("trimmed"),
+          part("asvar", field("variable", $.identifier)),
+        ),
+      ),
+    _translate_count: ($) =>
+      seq(
+        $._bt_option,
+        part(
+          "count",
+          seq(field("variable", $.identifier), "=", $.filtered_value),
+        ),
+      ),
+    _translate_body: ($) => repeat1(choice($.content, $.template_variable)),
+    /** The "with" option of blocktranslate takes at least one of these. */
+    _translate_kwargs: ($) =>
+      repeat1(
+        part(seq(field("variable", $.identifier), "=", $.filtered_value)),
+      ),
     cache_group: ($) =>
       seq(
         block(

queries/highlights.scm

@@ -50,6 +50,11 @@ tag: [
     ("endautoescape")
     ("block")
     ("endblock")
+    ("blocktrans")
+    ("endblocktrans")
+    ("blocktranslate")
+    ("endblocktranslate")
+    ("plural")
     ("cache")
     ("endcache")
     ("comment")

queries/libraries.scm

@@ -24,6 +24,11 @@
 (get_media_prefix
   tag: ("get_media_prefix") @library.static)
 
+(blocktranslate_group
+  tag: [
+    ("blocktrans")
+    ("blocktranslate")] @library.i18n)
+
 (translate
   tag: [
     ("trans")

src/node-types.json

@@ -11,6 +11,10 @@
         "type": "block_group",
         "named": true
       },
+      {
+        "type": "blocktranslate_group",
+        "named": true
+      },
       {
         "type": "cache_group",
         "named": true
@@ -278,6 +282,66 @@
       ]
     }
   },
+  {
+    "type": "blocktranslate_group",
+    "named": true,
+    "fields": {
+      "tag": {
+        "multiple": true,
+        "required": true,
+        "types": [
+          {
+            "type": "blocktrans",
+            "named": false
+          },
+          {
+            "type": "blocktranslate",
+            "named": false
+          },
+          {
+            "type": "endblocktrans",
+            "named": false
+          },
+          {
+            "type": "endblocktranslate",
+            "named": false
+          },
+          {
+            "type": "plural",
+            "named": false
+          }
+        ]
+      },
+      "variable": {
+        "multiple": true,
+        "required": false,
+        "types": [
+          {
+            "type": "identifier",
+            "named": true
+          }
+        ]
+      }
+    },
+    "children": {
+      "multiple": true,
+      "required": false,
+      "types": [
+        {
+          "type": "content",
+          "named": true
+        },
+        {
+          "type": "filtered_value",
+          "named": true
+        },
+        {
+          "type": "template_variable",
+          "named": true
+        }
+      ]
+    }
+  },
   {
     "type": "cache_group",
     "named": true,
@@ -2222,6 +2286,10 @@
     "type": "as",
     "named": false
   },
+  {
+    "type": "asvar",
+    "named": false
+  },
   {
     "type": "attribute",
     "named": true
@@ -2238,6 +2306,14 @@
     "type": "block",
     "named": false
   },
+  {
+    "type": "blocktrans",
+    "named": false
+  },
+  {
+    "type": "blocktranslate",
+    "named": false
+  },
   {
     "type": "by",
     "named": false
@@ -2286,6 +2362,10 @@
     "type": "context",
     "named": false
   },
+  {
+    "type": "count",
+    "named": false
+  },
   {
     "type": "csp_nonce_attr",
     "named": false
@@ -2350,6 +2430,14 @@
     "type": "endblock",
     "named": false
   },
+  {
+    "type": "endblocktrans",
+    "named": false
+  },
+  {
+    "type": "endblocktranslate",
+    "named": false
+  },
   {
     "type": "endcache",
     "named": false
@@ -2662,6 +2750,10 @@
     "type": "phone2numeric",
     "named": false
   },
+  {
+    "type": "plural",
+    "named": false
+  },
   {
     "type": "pluralize",
     "named": false
@@ -2786,6 +2878,10 @@
     "type": "translate",
     "named": false
   },
+  {
+    "type": "trimmed",
+    "named": false
+  },
   {
     "type": "truncatechars",
     "named": false

src/scanner.c

@@ -21,6 +21,18 @@ enum TokenType {
   VerbatimContent,
   /* raw text inside a comment block, up through (but excluding) the endcomment tag */
   CommentContent,
+  /* zero width, at the start of a blocktranslate tag: forget the options seen */
+  TranslateOpen,
+  /* zero width, before an option of a blocktranslate tag: record it, and fail
+   * if that option has already been written once */
+  TranslateOption,
+};
+
+/* The options blocktranslate takes, in the order of the bits recording them.
+ * do_block_translate keeps the ones it has seen in a dict and refuses a repeat,
+ * so a bit per option is the whole of the state this needs. */
+static const char *const translate_options[] = {
+  "with", "count", "context", "trimmed", "asvar",
 };
 
 typedef Array(int32_t) Name;
@@ -50,6 +62,10 @@ typedef Array(OpenTag) Stack;
 
 struct Scanner {
   bool has_error;
+  /* one bit per option of the blocktranslate tag being read, by the order of
+   * translate_options. Kept out of the union below, whose members share their
+   * storage with each other. */
+  uint8_t translate_seen;
   union {
     Stack stack;
     char error [ERROR_SIZE];
@@ -65,6 +81,7 @@ static enum TokenType pop_token_for_kind(TagKind kind) {
 }
 
 static void reset_scanner(struct Scanner *const scanner) {
+  scanner->translate_seen = 0;
   if (scanner->has_error) {
     scanner->error[0] = '\0';
     scanner->has_error = false;
@@ -141,6 +158,8 @@ write_serialization_error:;
   char *iter = buffer;
 
   iter += write_code(scanner->has_error + '0', iter);
+  // five options fit in five bits, which stays inside printable ASCII
+  iter += write_code(scanner->translate_seen + '0', iter);
   if (scanner->has_error) {
     for (int i = 0; i < ERROR_SIZE; ++i) {
       char value = scanner->error[i];
@@ -227,13 +246,33 @@ void tree_sitter_django_external_scanner_deserialize(
     return;
   }
   buffer += bytes_read;
+  unsigned consumed = bytes_read;
+
+  int32_t seen;
+  bytes_read = read_code(buffer, &seen);
+  if (!bytes_read) {
+    scanner_error(scanner, "received invalid utf8 byte");
+    return;
+  }
+  consumed += bytes_read;
+  if (consumed > length) {
+    scanner_error(scanner, "option mask truncated by length");
+    return;
+  }
+  buffer += bytes_read;
+  if (seen < '0' || seen > '0' + 0x1F) {
+    scanner_error(scanner, "received unrecognized option mask");
+    return;
+  }
+  scanner->translate_seen = (uint8_t) (seen - '0');
+
   switch (code) {
     default:
       scanner_error(scanner, "received unrecognized scanner status");
       return;
     case '1': {
       scanner->has_error = true;
-      for (unsigned i = 0; i < length - bytes_read; ++i) {
+      for (unsigned i = 0; i < length - consumed; ++i) {
         char value = *(buffer++);
         scanner->error[i] = value;
         if (value == '\0') {
@@ -244,7 +283,7 @@ void tree_sitter_django_external_scanner_deserialize(
     }
     case '0': {
       scanner->has_error = false;
-      const char *end = buffer + length - bytes_read;
+      const char *end = buffer + length - consumed;
       // NULL between tags, and the name being read inside one
       Name *name = NULL;
 
@@ -348,6 +387,37 @@ static void skip_whitespace(TSLexer *const lexer) {
   }
 }
 
+/* The longest name in translate_options, "context" and "trimmed", is 7. */
+#define TRANSLATE_OPTION_MAX 8
+
+/* Reads the word naming a blocktranslate option and returns its index, or -1
+ * for anything else, which includes the end of the tag. Consumes nothing the
+ * caller keeps: it is called after the end has been marked. */
+static int read_translate_option(TSLexer *const lexer) {
+  char word[TRANSLATE_OPTION_MAX + 1];
+  unsigned size = 0;
+  if (!check_name_start_char(lexer->lookahead)) {
+    return -1;
+  }
+  while (check_name_char(lexer->lookahead)) {
+    if (size < TRANSLATE_OPTION_MAX) {
+      word[size] = (char) lexer->lookahead;
+    }
+    ++size;
+    lexer->advance(lexer, false);
+  }
+  if (size > TRANSLATE_OPTION_MAX) {
+    return -1;
+  }
+  word[size] = '\0';
+  for (unsigned i = 0; i < sizeof(translate_options) / sizeof(*translate_options); ++i) {
+    if (strcmp(word, translate_options[i]) == 0) {
+      return (int) i;
+    }
+  }
+  return -1;
+}
+
 const char endverbatim_chars[] = "endverbatim";
 
 /* Called right after consuming "{%"; checks whether what follows closes the
@@ -508,6 +578,35 @@ bool tree_sitter_django_external_scanner_scan(
   if (valid_symbols[CommentContent]) {
     return scan_comment_content(lexer);
   }
+  /* Both of these are zero width, so the end is marked before anything is
+   * read. They belong above the loop below, which skips the separator the
+   * grammar still has to match, and below the MatcherError check, which is what
+   * keeps them from being scanned during error recovery. */
+  if (valid_symbols[TranslateOpen]) {
+    lexer->mark_end(lexer);
+    scanner->translate_seen = 0;
+    lexer->result_symbol = TranslateOpen;
+    return true;
+  }
+  if (valid_symbols[TranslateOption]) {
+    lexer->mark_end(lexer);
+    if (!check_space(lexer->lookahead)) {
+      return false;
+    }
+    skip_whitespace(lexer);
+    int option = read_translate_option(lexer);
+    if (option < 0) {
+      return false;
+    }
+    uint8_t bit = (uint8_t) (1 << option);
+    if (scanner->translate_seen & bit) {
+      // the option is already written once in this tag
+      return false;
+    }
+    scanner->translate_seen |= bit;
+    lexer->result_symbol = TranslateOption;
+    return true;
+  }
   while (check_space(lexer->lookahead)) {
     lexer->advance(lexer, true);
   }

test/corpus/blocktranslate.txt

@@ -0,0 +1,440 @@
+==================
+Blocktrans
+==================
+
+{% blocktrans %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (content))
+  (content))
+
+==================
+Blocktranslate
+==================
+
+{% blocktranslate %}hi{% endblocktranslate %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (content))
+  (content))
+
+==================
+Blocktrans Empty
+==================
+
+{% blocktrans %}{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group)
+  (content))
+
+==================
+Blocktrans With A Placeholder
+==================
+
+{% blocktrans %}hi {{ name }}{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (content)
+    (template_variable
+      (filtered_value
+        (value
+          (variable_attribute
+            (identifier))))))
+  (content))
+
+==================
+Blocktrans Trimmed
+==================
+
+{% blocktrans trimmed %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (content))
+  (content))
+
+==================
+Blocktrans With
+==================
+
+{% blocktrans with a=b %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (content))
+  (content))
+
+==================
+Blocktrans With Several Bindings
+==================
+
+{% blocktrans with a=b c=d %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (content))
+  (content))
+
+==================
+Blocktrans With A Filtered Binding
+==================
+
+{% blocktrans with a=b|upper %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier)))
+      (filter_expression
+        (filter)))
+    (content))
+  (content))
+
+==================
+Blocktrans With Context
+==================
+
+{% blocktrans context "menu" %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    (content))
+  (content))
+
+==================
+Blocktrans With Asvar
+==================
+
+{% blocktrans asvar out %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (content))
+  (content))
+
+==================
+Blocktrans With Count And Plural
+==================
+
+{% blocktrans count n=x %}one{% plural %}many{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (content)
+    (content))
+  (content))
+
+==================
+Blocktrans With Every Option
+==================
+
+{% blocktrans trimmed with a=b count n=x context "menu" asvar out %}s{% plural %}p{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    variable: (identifier)
+    (content)
+    (content))
+  (content))
+
+==================
+Blocktrans With Its Options Reordered
+==================
+
+{% blocktrans asvar out trimmed %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (content))
+  (content))
+
+==================
+Blocktrans Closed By The Other Spelling
+:error
+==================
+
+{% blocktrans %}hi{% endblocktranslate %}
+
+---
+
+
+==================
+Blocktranslate Closed By The Other Spelling
+:error
+==================
+
+{% blocktranslate %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Count And No Plural
+:error
+==================
+
+{% blocktrans count n=x %}one{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Plural And No Count
+:error
+==================
+
+{% blocktrans %}one{% plural %}many{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans Counting Two Bindings
+:error
+==================
+
+{% blocktrans count n=x c=d %}one{% plural %}many{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Nothing To Bind
+:error
+==================
+
+{% blocktrans with %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With An Unknown Option
+:error
+==================
+
+{% blocktrans bogus %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Asvar And No Name
+:error
+==================
+
+{% blocktrans asvar %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans Holding A Tag
+:error
+==================
+
+{% blocktrans %}a{% if x %}b{% endif %}{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans Holding A Comment
+:error
+==================
+
+{% blocktrans %}a{# c #}b{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans Holding Another Blocktrans
+:error
+==================
+
+{% blocktrans %}a{% blocktrans %}b{% endblocktrans %}{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Two Plurals
+:error
+==================
+
+{% blocktrans count n=x %}one{% plural %}many{% plural %}x{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With An Argument On Plural
+:error
+==================
+
+{% blocktrans count n=x %}one{% plural extra %}many{% endblocktrans %}
+
+---
+
+==================
+Blocktrans With Trimmed Twice
+:error
+==================
+
+{% blocktrans trimmed trimmed %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With With Twice
+:error
+==================
+
+{% blocktrans with a=b with c=d %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Context Twice
+:error
+==================
+
+{% blocktrans context "a" context "b" %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Asvar Twice
+:error
+==================
+
+{% blocktrans asvar x asvar y %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Count Twice
+:error
+==================
+
+{% blocktrans count n=x count m=y %}a{% plural %}b{% endblocktrans %}
+
+---
+
+
+==================
+Blocktranslate With Trimmed Twice
+:error
+==================
+
+{% blocktranslate trimmed trimmed %}hi{% endblocktranslate %}
+
+---
+
+
+==================
+Blocktrans Repeating An Option Of The Tag Before It
+==================
+
+{% blocktrans trimmed %}a{% endblocktrans %}{% blocktrans trimmed %}b{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (content))
+  (blocktranslate_group
+    (content))
+  (content))