add blocktrans

CLAUDE.md

@@ -106,7 +106,7 @@ which Django itself enforces while it parses the template ("add requires 2 argum
 ### First-party libraries
 
 Django ships `i18n`, `l10n`, `static`, `cache` and `tz` as libraries rather than builtins. Their tags are
-modelled as ordinary tags (so far: `static`, `l10n`, `tz`, `cache`), and the grammar **never requires the `{% load %}`**, because
+modelled as ordinary tags (so far: `static`, `l10n`, `tz`, `cache` and `i18n`), and the grammar **never requires the `{% load %}`**, because
 an engine can preload a library through `OPTIONS: {"builtins": [...]}`, which makes `{% static "a" %}` valid
 with no load at all.
 
@@ -130,29 +130,50 @@ is accepted: these names are common enough that shadowing them is the mistake.
 
 ### External scanner (`src/scanner.c`)
 
-Nine external tokens, for the constraints a context-free grammar cannot express:
+Eleven external tokens, for the constraints a context-free grammar cannot express:
 
 - `push_block`/`push_partial`/`push_verbatim` and the matching `pop_*` — name matching for
   `{% block a %}…{% endblock a %}`. They share **one** stack of `{kind, name}` entries, not one stack per
   kind: the grammar already guarantees nesting, so only the innermost open tag can be closed, and there is
   never a second candidate name to try. Serialized as `<kind><name> ` per entry (`tag_kind_chars`).
 - `verbatim_content`, `comment_content` — raw text up to the matching close tag.
+- `_bt_open`, `_bt_option` — which options a `{% blocktranslate %}` has already been given, so that a
+  repeated one is refused. Both are zero width and hidden, so no tree changes. The state is one byte of
+  flags, not a stack, because the tag cannot nest: its body admits no tags. It lives outside the
+  `stack`/`error` union, is cleared in both branches of `reset_scanner`, and is serialized after the
+  status character, so a GLR stack split copies it like everything else.
 - `matcher_error` — used by no rule; returned only if the scanner reaches an error state.
 
 `check_space()` defines whitespace for the scanner and **must stay in sync with `SEP`**, because the
 scanner skips whitespace itself while matching names.
 
 Prefer the grammar over the scanner. A useful test: does the constraint change how the input _parses_?
-Block-name matching does (it decides which close tag closes what). Things like "this keyword may not repeat"
+Block-name matching does — it decides which close tag closes what.
-do not, and belong in a linter instead.
+
+"This keyword may not repeat" normally does not, and belongs in a linter. `blocktranslate` is the exception,
+and the reason is worth keeping straight: its five options are a fixed set, so uniqueness _is_ expressible —
+`arrangements` expresses exactly that — and the scanner is standing in for a grammar that would be correct
+but takes minutes to build. Where the set is open, as with `{% querystring %}`'s `**kwargs`, no grammar
+can express it — but the scanner still could, by remembering the names a tag has been given rather than a
+fixed mask of them. That is not done, and it is what the `{% querystring page=2 page=3 %}` divergence below
+is waiting on.
+
+Zero-width guards carry one hazard: `recover_with_missing` can supply one without the scanner running. The
+`if (valid_symbols[MatcherError]) return false;` near the top of `scan` is what stops a guard being scanned
+during tree-sitter's mark-everything-valid recovery pass, so new guards go **below** it — and above the
+whitespace loop, which skips the separator the grammar still has to match. A guard must also never be valid
+where `content` is: a zero-width token at a content boundary preempts the internal lexer and `content` stops
+matching.
 
 ### Queries
 
 `queries/highlights.scm` lists tag and filter names explicitly, so adding either to the grammar means
 adding it there too; filter names appear there without their colon. A name the grammar does not know is
-captured through its field instead (`(filter name: (identifier))`, `(custom_tag tag: (identifier))`). `queries/libraries.scm` is not a query editors run themselves; it is data for a linter, and every tag
+captured through its field instead (`(filter name: (identifier))`, `(custom_tag tag: (identifier))`).
-or filter added from a library belongs in it. `queries/locals.scm` relies on the `variable:` field to tell a binding from a
+
-reference — another reason to keep `asVariable` uniform.
+`queries/libraries.scm` is not a query editors run themselves; it is data for a linter, and every tag or
+filter added from a library belongs in it. `queries/locals.scm` relies on the `variable:` field to tell a
+binding from a reference — another reason to keep `asVariable` uniform.
 
 ## Testing
 
@@ -210,5 +231,7 @@ documentation or memory. Both checks are worth repeating whenever tags, filters
   `{% cache 500 sidebar.name %}` names a fragment there, `{% cache 500 using="c" %}` names a fragment
   called `using="c"` rather than choosing a cache, and `{% cache 500 sidebar using= %}` takes an empty
   name. All three are errors here.
+- `do_translate` refuses `as` and `noop` as the value of its `context` option, so
+  `{% trans "hi" context noop %}` is an error there and a context named by the variable `noop` here.
 - `partial`/`partialdef` are Django builtins as of Django 6; `elif`/`else`/`empty` are modelled as parts of
   their enclosing tag rather than as separate tags.

grammar.js

@@ -233,6 +233,14 @@ const L10N_FILTERS_WITHOUT_ARGUMENT = ["localize", "unlocalize"];
 const TZ_FILTERS_WITHOUT_ARGUMENT = ["localtime", "utc"];
 const TZ_FILTERS_WITH_ARGUMENT = ["timezone"];
 
+/** django.templatetags.i18n. */
+const I18N_FILTERS_WITHOUT_ARGUMENT = [
+  "language_bidi",
+  "language_name",
+  "language_name_local",
+  "language_name_translated",
+];
+
 /**
  * The words that head an end tag or a part of a tag group. Each is a token
  * only inside the group it belongs to, so anywhere else the lexer reads it as
@@ -247,12 +255,15 @@ const NAMES_INSIDE_A_TAG_GROUP = [
   "empty",
   "endautoescape",
   "endblock",
+  "endblocktrans",
+  "endblocktranslate",
   "endcache",
   "endcomment",
   "endfilter",
   "endfor",
   "endif",
   "endifchanged",
+  "endlanguage",
   "endlocalize",
   "endlocaltime",
   "endpartialdef",
@@ -260,6 +271,7 @@ const NAMES_INSIDE_A_TAG_GROUP = [
   "endtimezone",
   "endverbatim",
   "endwith",
+  "plural",
 ];
 
 /** Django's string constant, which takes any escape but a line break. */
@@ -275,6 +287,7 @@ const django = grammar({
     [$.template],
     [$.predicate],
     [$.library, $.load],
+    [$._translate_option],
     [$._filtered_value_spaced],
     [$._filter_expression_spaced],
   ],
@@ -289,6 +302,8 @@ const django = grammar({
     $.push_verbatim,
     $.verbatim_content,
     $.comment_content,
+    $._bt_open,
+    $._bt_option,
   ],
   reserved: {
     global: ($) => ["not", "if", "in", "is", "as", "for", "from"],
@@ -379,6 +394,7 @@ const django = grammar({
       choice(
         $.autoescape_group,
         $.block_group,
+        $.blocktranslate_group,
         $.cache_group,
         $.comment_group,
         $.csp_nonce_attr,
@@ -390,12 +406,18 @@ const django = grammar({
         $.filter_group,
         $.firstof,
         $.for_group,
+        $.get_available_languages,
+        $.get_current_language,
+        $.get_current_language_bidi,
         $.get_current_timezone,
+        $.get_language_info,
+        $.get_language_info_list,
         $.get_media_prefix,
         $.get_static_prefix,
         $.if_group,
         $.ifchanged_group,
         $.include,
+        $.language_group,
         $.load,
         $.localize_group,
         $.localtime_group,
@@ -410,6 +432,7 @@ const django = grammar({
         $.static,
         $.template_tag_block,
         $.timezone_group,
+        $.translate,
         $.url_block,
         $.verbatim_group,
         $.width_ratio,
@@ -427,6 +450,7 @@ const django = grammar({
           "name",
           choice(
             ...FILTERS_WITHOUT_ARGUMENT,
+            ...I18N_FILTERS_WITHOUT_ARGUMENT,
             ...L10N_FILTERS_WITHOUT_ARGUMENT,
             ...TZ_FILTERS_WITHOUT_ARGUMENT,
           ),
@@ -463,6 +487,60 @@ const django = grammar({
         optional($.template),
         block("endblock", $.pop_block),
       ),
+    blocktranslate_group: ($) =>
+      choice(
+        ...["blocktrans", "blocktranslate"].map((name) =>
+          choice(
+            seq(
+              block(name, $._bt_open, repeat($._translate_option)),
+              optional($._translate_body),
+              block("end" + name),
+            ),
+            seq(
+              block(
+                name,
+                $._bt_open,
+                repeat($._translate_option),
+                $._translate_count,
+                repeat($._translate_option),
+              ),
+              $._translate_body,
+              block("plural"),
+              optional($._translate_body),
+              block("end" + name),
+            ),
+          ),
+        ),
+      ),
+    /**
+     * The scanner reads the name of the option before the grammar does, so it
+     * can refuse one the tag has already been given: do_block_translate keeps
+     * the options it has seen in a dict and raises on a repeat.
+     */
+    _translate_option: ($) =>
+      seq(
+        $._bt_option,
+        choice(
+          seq(part("with"), $._translate_kwargs),
+          part("context", $.filtered_value),
+          part("trimmed"),
+          part("asvar", field("variable", $.identifier)),
+        ),
+      ),
+    _translate_count: ($) =>
+      seq(
+        $._bt_option,
+        part(
+          "count",
+          seq(field("variable", $.identifier), "=", $.filtered_value),
+        ),
+      ),
+    _translate_body: ($) => repeat1(choice($.content, $.template_variable)),
+    /** The "with" option of blocktranslate takes at least one of these. */
+    _translate_kwargs: ($) =>
+      repeat1(
+        part(seq(field("variable", $.identifier), "=", $.filtered_value)),
+      ),
     cache_group: ($) =>
       seq(
         block(
@@ -549,6 +627,23 @@ const django = grammar({
         optional(seq(block("empty"), optional($.template))),
         block("endfor"),
       ),
+    /**
+     * django.templatetags.i18n. Each of these requires exactly "as <name>",
+     * or "for <expression> as <name>", and rejects anything further.
+     */
+    get_available_languages: ($) =>
+      block("get_available_languages", asVariable($)),
+    get_current_language: ($) => block("get_current_language", asVariable($)),
+    get_current_language_bidi: ($) =>
+      block("get_current_language_bidi", asVariable($)),
+    get_language_info: ($) =>
+      block("get_language_info", part("for", $.filtered_value), asVariable($)),
+    get_language_info_list: ($) =>
+      block(
+        "get_language_info_list",
+        part("for", $.filtered_value),
+        asVariable($),
+      ),
     /**
      * django.templatetags.tz. get_current_timezone_tag requires exactly
      * "as <name>", so unlike get_static_prefix the clause is not optional and
@@ -574,6 +669,13 @@ const django = grammar({
         block("endifchanged"),
       ),
     include: ($) => block("include", part($.filtered_value)),
+    /** django.templatetags.i18n. The language tag takes the one argument. */
+    language_group: ($) =>
+      seq(
+        block("language", part($.filtered_value)),
+        optional($.template),
+        block("endlanguage"),
+      ),
     library: ($) => seq($.identifier, optional(seq(".", $.identifier))),
     load: ($) =>
       block(
@@ -653,6 +755,23 @@ const django = grammar({
         optional($.template),
         block("endtimezone"),
       ),
+    /**
+     * django.templatetags.i18n, registered under both spellings. do_translate
+     * pops its options off one at a time and refuses one it has already seen,
+     * so they may be written in any order but none of them twice.
+     */
+    translate: ($) =>
+      block(
+        choice("trans", "translate"),
+        part($.filtered_value),
+        optional(
+          arrangements([
+            part("noop"),
+            part("context", $.filtered_value),
+            asVariable($),
+          ]),
+        ),
+      ),
     url_block: ($) =>
       block(
         "url",

queries/highlights.scm

@@ -50,6 +50,11 @@ tag: [
     ("endautoescape")
     ("block")
     ("endblock")
+    ("blocktrans")
+    ("endblocktrans")
+    ("blocktranslate")
+    ("endblocktranslate")
+    ("plural")
     ("cache")
     ("endcache")
     ("comment")
@@ -68,7 +73,12 @@ tag: [
     ("for")
     ("empty")
     ("endfor")
+    ("get_available_languages")
+    ("get_current_language")
+    ("get_current_language_bidi")
     ("get_current_timezone")
+    ("get_language_info")
+    ("get_language_info_list")
     ("get_media_prefix")
     ("get_static_prefix")
     ("if")
@@ -78,6 +88,8 @@ tag: [
     ("ifchanged")
     ("endifchanged")
     ("include")
+    ("language")
+    ("endlanguage")
     ("lorem")
     ("load")
     ("localize")
@@ -97,6 +109,8 @@ tag: [
     ("templatetag")
     ("timezone")
     ("endtimezone")
+    ("trans")
+    ("translate")
     ("url")
     ("verbatim")
     ("endverbatim")
@@ -127,6 +141,10 @@ tag: [
         ("iriencode")
         ("join")
         ("json_script")
+        ("language_bidi")
+        ("language_name")
+        ("language_name_local")
+        ("language_name_translated")
         ("last")
         ("length")
         ("linebreaks")

queries/libraries.scm

@@ -24,6 +24,41 @@
 (get_media_prefix
   tag: ("get_media_prefix") @library.static)
 
+(blocktranslate_group
+  tag: [
+    ("blocktrans")
+    ("blocktranslate")] @library.i18n)
+
+(translate
+  tag: [
+    ("trans")
+    ("translate")] @library.i18n)
+
+(language_group
+  tag: ("language") @library.i18n)
+
+(get_available_languages
+  tag: ("get_available_languages") @library.i18n)
+
+(get_current_language
+  tag: ("get_current_language") @library.i18n)
+
+(get_current_language_bidi
+  tag: ("get_current_language_bidi") @library.i18n)
+
+(get_language_info
+  tag: ("get_language_info") @library.i18n)
+
+(get_language_info_list
+  tag: ("get_language_info_list") @library.i18n)
+
+(filter
+  name: [
+    ("language_bidi")
+    ("language_name")
+    ("language_name_local")
+    ("language_name_translated")] @library.i18n)
+
 (cache_group
   tag: ("cache") @library.cache)
 

src/node-types.json

@@ -11,6 +11,10 @@
         "type": "block_group",
         "named": true
       },
+      {
+        "type": "blocktranslate_group",
+        "named": true
+      },
       {
         "type": "cache_group",
         "named": true
@@ -55,10 +59,30 @@
         "type": "for_group",
         "named": true
       },
+      {
+        "type": "get_available_languages",
+        "named": true
+      },
+      {
+        "type": "get_current_language",
+        "named": true
+      },
+      {
+        "type": "get_current_language_bidi",
+        "named": true
+      },
       {
         "type": "get_current_timezone",
         "named": true
       },
+      {
+        "type": "get_language_info",
+        "named": true
+      },
+      {
+        "type": "get_language_info_list",
+        "named": true
+      },
       {
         "type": "get_media_prefix",
         "named": true
@@ -79,6 +103,10 @@
         "type": "include",
         "named": true
       },
+      {
+        "type": "language_group",
+        "named": true
+      },
       {
         "type": "load",
         "named": true
@@ -135,6 +163,10 @@
         "type": "timezone_group",
         "named": true
       },
+      {
+        "type": "translate",
+        "named": true
+      },
       {
         "type": "url_block",
         "named": true
@@ -250,6 +282,66 @@
       ]
     }
   },
+  {
+    "type": "blocktranslate_group",
+    "named": true,
+    "fields": {
+      "tag": {
+        "multiple": true,
+        "required": true,
+        "types": [
+          {
+            "type": "blocktrans",
+            "named": false
+          },
+          {
+            "type": "blocktranslate",
+            "named": false
+          },
+          {
+            "type": "endblocktrans",
+            "named": false
+          },
+          {
+            "type": "endblocktranslate",
+            "named": false
+          },
+          {
+            "type": "plural",
+            "named": false
+          }
+        ]
+      },
+      "variable": {
+        "multiple": true,
+        "required": false,
+        "types": [
+          {
+            "type": "identifier",
+            "named": true
+          }
+        ]
+      }
+    },
+    "children": {
+      "multiple": true,
+      "required": false,
+      "types": [
+        {
+          "type": "content",
+          "named": true
+        },
+        {
+          "type": "filtered_value",
+          "named": true
+        },
+        {
+          "type": "template_variable",
+          "named": true
+        }
+      ]
+    }
+  },
   {
     "type": "cache_group",
     "named": true,
@@ -606,6 +698,22 @@
             "type": "json_script",
             "named": false
           },
+          {
+            "type": "language_bidi",
+            "named": false
+          },
+          {
+            "type": "language_name",
+            "named": false
+          },
+          {
+            "type": "language_name_local",
+            "named": false
+          },
+          {
+            "type": "language_name_translated",
+            "named": false
+          },
           {
             "type": "last",
             "named": false
@@ -948,6 +1056,84 @@
       ]
     }
   },
+  {
+    "type": "get_available_languages",
+    "named": true,
+    "fields": {
+      "tag": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "get_available_languages",
+            "named": false
+          }
+        ]
+      },
+      "variable": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "identifier",
+            "named": true
+          }
+        ]
+      }
+    }
+  },
+  {
+    "type": "get_current_language",
+    "named": true,
+    "fields": {
+      "tag": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "get_current_language",
+            "named": false
+          }
+        ]
+      },
+      "variable": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "identifier",
+            "named": true
+          }
+        ]
+      }
+    }
+  },
+  {
+    "type": "get_current_language_bidi",
+    "named": true,
+    "fields": {
+      "tag": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "get_current_language_bidi",
+            "named": false
+          }
+        ]
+      },
+      "variable": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "identifier",
+            "named": true
+          }
+        ]
+      }
+    }
+  },
   {
     "type": "get_current_timezone",
     "named": true,
@@ -974,6 +1160,78 @@
       }
     }
   },
+  {
+    "type": "get_language_info",
+    "named": true,
+    "fields": {
+      "tag": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "get_language_info",
+            "named": false
+          }
+        ]
+      },
+      "variable": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "identifier",
+            "named": true
+          }
+        ]
+      }
+    },
+    "children": {
+      "multiple": false,
+      "required": true,
+      "types": [
+        {
+          "type": "filtered_value",
+          "named": true
+        }
+      ]
+    }
+  },
+  {
+    "type": "get_language_info_list",
+    "named": true,
+    "fields": {
+      "tag": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "get_language_info_list",
+            "named": false
+          }
+        ]
+      },
+      "variable": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "identifier",
+            "named": true
+          }
+        ]
+      }
+    },
+    "children": {
+      "multiple": false,
+      "required": true,
+      "types": [
+        {
+          "type": "filtered_value",
+          "named": true
+        }
+      ]
+    }
+  },
   {
     "type": "get_media_prefix",
     "named": true,
@@ -1132,6 +1390,40 @@
       ]
     }
   },
+  {
+    "type": "language_group",
+    "named": true,
+    "fields": {
+      "tag": {
+        "multiple": true,
+        "required": true,
+        "types": [
+          {
+            "type": "endlanguage",
+            "named": false
+          },
+          {
+            "type": "language",
+            "named": false
+          }
+        ]
+      }
+    },
+    "children": {
+      "multiple": true,
+      "required": true,
+      "types": [
+        {
+          "type": "filtered_value",
+          "named": true
+        },
+        {
+          "type": "template",
+          "named": true
+        }
+      ]
+    }
+  },
   {
     "type": "library",
     "named": true,
@@ -1667,6 +1959,46 @@
       ]
     }
   },
+  {
+    "type": "translate",
+    "named": true,
+    "fields": {
+      "tag": {
+        "multiple": false,
+        "required": true,
+        "types": [
+          {
+            "type": "trans",
+            "named": false
+          },
+          {
+            "type": "translate",
+            "named": false
+          }
+        ]
+      },
+      "variable": {
+        "multiple": false,
+        "required": false,
+        "types": [
+          {
+            "type": "identifier",
+            "named": true
+          }
+        ]
+      }
+    },
+    "children": {
+      "multiple": true,
+      "required": true,
+      "types": [
+        {
+          "type": "filtered_value",
+          "named": true
+        }
+      ]
+    }
+  },
   {
     "type": "translated_string",
     "named": true,
@@ -1954,6 +2286,10 @@
     "type": "as",
     "named": false
   },
+  {
+    "type": "asvar",
+    "named": false
+  },
   {
     "type": "attribute",
     "named": true
@@ -1970,6 +2306,14 @@
     "type": "block",
     "named": false
   },
+  {
+    "type": "blocktrans",
+    "named": false
+  },
+  {
+    "type": "blocktranslate",
+    "named": false
+  },
   {
     "type": "by",
     "named": false
@@ -2014,6 +2358,14 @@
     "type": "content",
     "named": true
   },
+  {
+    "type": "context",
+    "named": false
+  },
+  {
+    "type": "count",
+    "named": false
+  },
   {
     "type": "csp_nonce_attr",
     "named": false
@@ -2078,6 +2430,14 @@
     "type": "endblock",
     "named": false
   },
+  {
+    "type": "endblocktrans",
+    "named": false
+  },
+  {
+    "type": "endblocktranslate",
+    "named": false
+  },
   {
     "type": "endcache",
     "named": false
@@ -2102,6 +2462,10 @@
     "type": "endifchanged",
     "named": false
   },
+  {
+    "type": "endlanguage",
+    "named": false
+  },
   {
     "type": "endlocalize",
     "named": false
@@ -2178,6 +2542,18 @@
     "type": "from",
     "named": false
   },
+  {
+    "type": "get_available_languages",
+    "named": false
+  },
+  {
+    "type": "get_current_language",
+    "named": false
+  },
+  {
+    "type": "get_current_language_bidi",
+    "named": false
+  },
   {
     "type": "get_current_timezone",
     "named": false
@@ -2186,6 +2562,14 @@
     "type": "get_digit",
     "named": false
   },
+  {
+    "type": "get_language_info",
+    "named": false
+  },
+  {
+    "type": "get_language_info_list",
+    "named": false
+  },
   {
     "type": "get_media_prefix",
     "named": false
@@ -2234,6 +2618,26 @@
     "type": "json_script",
     "named": false
   },
+  {
+    "type": "language",
+    "named": false
+  },
+  {
+    "type": "language_bidi",
+    "named": false
+  },
+  {
+    "type": "language_name",
+    "named": false
+  },
+  {
+    "type": "language_name_local",
+    "named": false
+  },
+  {
+    "type": "language_name_translated",
+    "named": false
+  },
   {
     "type": "last",
     "named": false
@@ -2286,6 +2690,10 @@
     "type": "media",
     "named": false
   },
+  {
+    "type": "noop",
+    "named": false
+  },
   {
     "type": "not",
     "named": false
@@ -2342,6 +2750,10 @@
     "type": "phone2numeric",
     "named": false
   },
+  {
+    "type": "plural",
+    "named": false
+  },
   {
     "type": "pluralize",
     "named": false
@@ -2458,6 +2870,18 @@
     "type": "title",
     "named": false
   },
+  {
+    "type": "trans",
+    "named": false
+  },
+  {
+    "type": "translate",
+    "named": false
+  },
+  {
+    "type": "trimmed",
+    "named": false
+  },
   {
     "type": "truncatechars",
     "named": false

src/scanner.c

@@ -21,6 +21,18 @@ enum TokenType {
   VerbatimContent,
   /* raw text inside a comment block, up through (but excluding) the endcomment tag */
   CommentContent,
+  /* zero width, at the start of a blocktranslate tag: forget the options seen */
+  TranslateOpen,
+  /* zero width, before an option of a blocktranslate tag: record it, and fail
+   * if that option has already been written once */
+  TranslateOption,
+};
+
+/* The options blocktranslate takes, in the order of the bits recording them.
+ * do_block_translate keeps the ones it has seen in a dict and refuses a repeat,
+ * so a bit per option is the whole of the state this needs. */
+static const char *const translate_options[] = {
+  "with", "count", "context", "trimmed", "asvar",
 };
 
 typedef Array(int32_t) Name;
@@ -50,6 +62,10 @@ typedef Array(OpenTag) Stack;
 
 struct Scanner {
   bool has_error;
+  /* one bit per option of the blocktranslate tag being read, by the order of
+   * translate_options. Kept out of the union below, whose members share their
+   * storage with each other. */
+  uint8_t translate_seen;
   union {
     Stack stack;
     char error [ERROR_SIZE];
@@ -65,6 +81,7 @@ static enum TokenType pop_token_for_kind(TagKind kind) {
 }
 
 static void reset_scanner(struct Scanner *const scanner) {
+  scanner->translate_seen = 0;
   if (scanner->has_error) {
     scanner->error[0] = '\0';
     scanner->has_error = false;
@@ -141,6 +158,8 @@ write_serialization_error:;
   char *iter = buffer;
 
   iter += write_code(scanner->has_error + '0', iter);
+  // five options fit in five bits, which stays inside printable ASCII
+  iter += write_code(scanner->translate_seen + '0', iter);
   if (scanner->has_error) {
     for (int i = 0; i < ERROR_SIZE; ++i) {
       char value = scanner->error[i];
@@ -227,13 +246,33 @@ void tree_sitter_django_external_scanner_deserialize(
     return;
   }
   buffer += bytes_read;
+  unsigned consumed = bytes_read;
+
+  int32_t seen;
+  bytes_read = read_code(buffer, &seen);
+  if (!bytes_read) {
+    scanner_error(scanner, "received invalid utf8 byte");
+    return;
+  }
+  consumed += bytes_read;
+  if (consumed > length) {
+    scanner_error(scanner, "option mask truncated by length");
+    return;
+  }
+  buffer += bytes_read;
+  if (seen < '0' || seen > '0' + 0x1F) {
+    scanner_error(scanner, "received unrecognized option mask");
+    return;
+  }
+  scanner->translate_seen = (uint8_t) (seen - '0');
+
   switch (code) {
     default:
       scanner_error(scanner, "received unrecognized scanner status");
       return;
     case '1': {
       scanner->has_error = true;
-      for (unsigned i = 0; i < length - bytes_read; ++i) {
+      for (unsigned i = 0; i < length - consumed; ++i) {
         char value = *(buffer++);
         scanner->error[i] = value;
         if (value == '\0') {
@@ -244,7 +283,7 @@ void tree_sitter_django_external_scanner_deserialize(
     }
     case '0': {
       scanner->has_error = false;
-      const char *end = buffer + length - bytes_read;
+      const char *end = buffer + length - consumed;
       // NULL between tags, and the name being read inside one
       Name *name = NULL;
 
@@ -348,6 +387,37 @@ static void skip_whitespace(TSLexer *const lexer) {
   }
 }
 
+/* The longest name in translate_options, "context" and "trimmed", is 7. */
+#define TRANSLATE_OPTION_MAX 8
+
+/* Reads the word naming a blocktranslate option and returns its index, or -1
+ * for anything else, which includes the end of the tag. Consumes nothing the
+ * caller keeps: it is called after the end has been marked. */
+static int read_translate_option(TSLexer *const lexer) {
+  char word[TRANSLATE_OPTION_MAX + 1];
+  unsigned size = 0;
+  if (!check_name_start_char(lexer->lookahead)) {
+    return -1;
+  }
+  while (check_name_char(lexer->lookahead)) {
+    if (size < TRANSLATE_OPTION_MAX) {
+      word[size] = (char) lexer->lookahead;
+    }
+    ++size;
+    lexer->advance(lexer, false);
+  }
+  if (size > TRANSLATE_OPTION_MAX) {
+    return -1;
+  }
+  word[size] = '\0';
+  for (unsigned i = 0; i < sizeof(translate_options) / sizeof(*translate_options); ++i) {
+    if (strcmp(word, translate_options[i]) == 0) {
+      return (int) i;
+    }
+  }
+  return -1;
+}
+
 const char endverbatim_chars[] = "endverbatim";
 
 /* Called right after consuming "{%"; checks whether what follows closes the
@@ -508,6 +578,35 @@ bool tree_sitter_django_external_scanner_scan(
   if (valid_symbols[CommentContent]) {
     return scan_comment_content(lexer);
   }
+  /* Both of these are zero width, so the end is marked before anything is
+   * read. They belong above the loop below, which skips the separator the
+   * grammar still has to match, and below the MatcherError check, which is what
+   * keeps them from being scanned during error recovery. */
+  if (valid_symbols[TranslateOpen]) {
+    lexer->mark_end(lexer);
+    scanner->translate_seen = 0;
+    lexer->result_symbol = TranslateOpen;
+    return true;
+  }
+  if (valid_symbols[TranslateOption]) {
+    lexer->mark_end(lexer);
+    if (!check_space(lexer->lookahead)) {
+      return false;
+    }
+    skip_whitespace(lexer);
+    int option = read_translate_option(lexer);
+    if (option < 0) {
+      return false;
+    }
+    uint8_t bit = (uint8_t) (1 << option);
+    if (scanner->translate_seen & bit) {
+      // the option is already written once in this tag
+      return false;
+    }
+    scanner->translate_seen |= bit;
+    lexer->result_symbol = TranslateOption;
+    return true;
+  }
   while (check_space(lexer->lookahead)) {
     lexer->advance(lexer, true);
   }

test/corpus/blocktranslate.txt

@@ -0,0 +1,440 @@
+==================
+Blocktrans
+==================
+
+{% blocktrans %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (content))
+  (content))
+
+==================
+Blocktranslate
+==================
+
+{% blocktranslate %}hi{% endblocktranslate %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (content))
+  (content))
+
+==================
+Blocktrans Empty
+==================
+
+{% blocktrans %}{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group)
+  (content))
+
+==================
+Blocktrans With A Placeholder
+==================
+
+{% blocktrans %}hi {{ name }}{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (content)
+    (template_variable
+      (filtered_value
+        (value
+          (variable_attribute
+            (identifier))))))
+  (content))
+
+==================
+Blocktrans Trimmed
+==================
+
+{% blocktrans trimmed %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (content))
+  (content))
+
+==================
+Blocktrans With
+==================
+
+{% blocktrans with a=b %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (content))
+  (content))
+
+==================
+Blocktrans With Several Bindings
+==================
+
+{% blocktrans with a=b c=d %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (content))
+  (content))
+
+==================
+Blocktrans With A Filtered Binding
+==================
+
+{% blocktrans with a=b|upper %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier)))
+      (filter_expression
+        (filter)))
+    (content))
+  (content))
+
+==================
+Blocktrans With Context
+==================
+
+{% blocktrans context "menu" %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    (content))
+  (content))
+
+==================
+Blocktrans With Asvar
+==================
+
+{% blocktrans asvar out %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (content))
+  (content))
+
+==================
+Blocktrans With Count And Plural
+==================
+
+{% blocktrans count n=x %}one{% plural %}many{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (content)
+    (content))
+  (content))
+
+==================
+Blocktrans With Every Option
+==================
+
+{% blocktrans trimmed with a=b count n=x context "menu" asvar out %}s{% plural %}p{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    variable: (identifier)
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    variable: (identifier)
+    (content)
+    (content))
+  (content))
+
+==================
+Blocktrans With Its Options Reordered
+==================
+
+{% blocktrans asvar out trimmed %}hi{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    variable: (identifier)
+    (content))
+  (content))
+
+==================
+Blocktrans Closed By The Other Spelling
+:error
+==================
+
+{% blocktrans %}hi{% endblocktranslate %}
+
+---
+
+
+==================
+Blocktranslate Closed By The Other Spelling
+:error
+==================
+
+{% blocktranslate %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Count And No Plural
+:error
+==================
+
+{% blocktrans count n=x %}one{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Plural And No Count
+:error
+==================
+
+{% blocktrans %}one{% plural %}many{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans Counting Two Bindings
+:error
+==================
+
+{% blocktrans count n=x c=d %}one{% plural %}many{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Nothing To Bind
+:error
+==================
+
+{% blocktrans with %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With An Unknown Option
+:error
+==================
+
+{% blocktrans bogus %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Asvar And No Name
+:error
+==================
+
+{% blocktrans asvar %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans Holding A Tag
+:error
+==================
+
+{% blocktrans %}a{% if x %}b{% endif %}{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans Holding A Comment
+:error
+==================
+
+{% blocktrans %}a{# c #}b{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans Holding Another Blocktrans
+:error
+==================
+
+{% blocktrans %}a{% blocktrans %}b{% endblocktrans %}{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Two Plurals
+:error
+==================
+
+{% blocktrans count n=x %}one{% plural %}many{% plural %}x{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With An Argument On Plural
+:error
+==================
+
+{% blocktrans count n=x %}one{% plural extra %}many{% endblocktrans %}
+
+---
+
+==================
+Blocktrans With Trimmed Twice
+:error
+==================
+
+{% blocktrans trimmed trimmed %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With With Twice
+:error
+==================
+
+{% blocktrans with a=b with c=d %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Context Twice
+:error
+==================
+
+{% blocktrans context "a" context "b" %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Asvar Twice
+:error
+==================
+
+{% blocktrans asvar x asvar y %}hi{% endblocktrans %}
+
+---
+
+
+==================
+Blocktrans With Count Twice
+:error
+==================
+
+{% blocktrans count n=x count m=y %}a{% plural %}b{% endblocktrans %}
+
+---
+
+
+==================
+Blocktranslate With Trimmed Twice
+:error
+==================
+
+{% blocktranslate trimmed trimmed %}hi{% endblocktranslate %}
+
+---
+
+
+==================
+Blocktrans Repeating An Option Of The Tag Before It
+==================
+
+{% blocktrans trimmed %}a{% endblocktrans %}{% blocktrans trimmed %}b{% endblocktrans %}
+
+---
+
+(template
+  (content)
+  (blocktranslate_group
+    (content))
+  (blocktranslate_group
+    (content))
+  (content))

test/corpus/i18n.txt

@@ -0,0 +1,488 @@
+==================
+Trans
+==================
+
+{% trans "hi" %}
+
+---
+
+(template
+  (content)
+  (translate
+    (filtered_value
+      (value
+        (literal
+          (string)))))
+  (content))
+
+==================
+Translate
+==================
+
+{% translate "hi" %}
+
+---
+
+(template
+  (content)
+  (translate
+    (filtered_value
+      (value
+        (literal
+          (string)))))
+  (content))
+
+==================
+Trans With A Variable
+==================
+
+{% trans greeting %}
+
+---
+
+(template
+  (content)
+  (translate
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier)))))
+  (content))
+
+==================
+Trans With A Filtered Message
+==================
+
+{% trans "hi"|upper %}
+
+---
+
+(template
+  (content)
+  (translate
+    (filtered_value
+      (value
+        (literal
+          (string)))
+      (filter_expression
+        (filter))))
+  (content))
+
+==================
+Trans Noop
+==================
+
+{% trans "hi" noop %}
+
+---
+
+(template
+  (content)
+  (translate
+    (filtered_value
+      (value
+        (literal
+          (string)))))
+  (content))
+
+==================
+Trans With As
+==================
+
+{% trans "hi" as greeting %}
+
+---
+
+(template
+  (content)
+  (translate
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    variable: (identifier))
+  (content))
+
+==================
+Trans With Context
+==================
+
+{% trans "hi" context "menu" %}
+
+---
+
+(template
+  (content)
+  (translate
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    (filtered_value
+      (value
+        (literal
+          (string)))))
+  (content))
+
+==================
+Trans With Every Option
+==================
+
+{% trans "hi" context "menu" noop as greeting %}
+
+---
+
+(template
+  (content)
+  (translate
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    variable: (identifier))
+  (content))
+
+==================
+Trans With Its Options Reordered
+==================
+
+{% trans "hi" as greeting noop context "menu" %}
+
+---
+
+(template
+  (content)
+  (translate
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    variable: (identifier)
+    (filtered_value
+      (value
+        (literal
+          (string)))))
+  (content))
+
+==================
+Trans Without A Message
+:error
+==================
+
+{% trans %}
+
+---
+
+
+==================
+Trans With An Option Twice
+:error
+==================
+
+{% trans "hi" noop noop %}
+
+---
+
+
+==================
+Trans With As Twice
+:error
+==================
+
+{% trans "hi" as x as y %}
+
+---
+
+
+==================
+Trans With An Unknown Option
+:error
+==================
+
+{% trans "hi" bogus %}
+
+---
+
+
+==================
+Trans With Context And No Value
+:error
+==================
+
+{% trans "hi" context %}
+
+---
+
+
+==================
+Get Available Languages
+==================
+
+{% get_available_languages as languages %}
+
+---
+
+(template
+  (content)
+  (get_available_languages
+    variable: (identifier))
+  (content))
+
+==================
+Get Available Languages Without As
+:error
+==================
+
+{% get_available_languages %}
+
+---
+
+
+==================
+Get Current Language
+==================
+
+{% get_current_language as language %}
+
+---
+
+(template
+  (content)
+  (get_current_language
+    variable: (identifier))
+  (content))
+
+==================
+Get Current Language Bidi
+==================
+
+{% get_current_language_bidi as bidi %}
+
+---
+
+(template
+  (content)
+  (get_current_language_bidi
+    variable: (identifier))
+  (content))
+
+==================
+Get Language Info
+==================
+
+{% get_language_info for "de" as info %}
+
+---
+
+(template
+  (content)
+  (get_language_info
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    variable: (identifier))
+  (content))
+
+==================
+Get Language Info For A Variable
+==================
+
+{% get_language_info for code as info %}
+
+---
+
+(template
+  (content)
+  (get_language_info
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    variable: (identifier))
+  (content))
+
+==================
+Get Language Info Without As
+:error
+==================
+
+{% get_language_info for "de" %}
+
+---
+
+
+==================
+Get Language Info Without For
+:error
+==================
+
+{% get_language_info "de" as info %}
+
+---
+
+
+==================
+Get Language Info List
+==================
+
+{% get_language_info_list for codes as info %}
+
+---
+
+(template
+  (content)
+  (get_language_info_list
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    variable: (identifier))
+  (content))
+
+==================
+Language
+==================
+
+{% language "de" %}x{% endlanguage %}
+
+---
+
+(template
+  (content)
+  (language_group
+    (filtered_value
+      (value
+        (literal
+          (string))))
+    (template
+      (content)))
+  (content))
+
+==================
+Language With A Variable
+==================
+
+{% language code %}x{% endlanguage %}
+
+---
+
+(template
+  (content)
+  (language_group
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (template
+      (content)))
+  (content))
+
+==================
+Language Without An Argument
+:error
+==================
+
+{% language %}x{% endlanguage %}
+
+---
+
+
+==================
+Language With Two Arguments
+:error
+==================
+
+{% language "a" "b" %}x{% endlanguage %}
+
+---
+
+
+==================
+Language Name Filter
+==================
+
+{{ "de"|language_name }}
+
+---
+
+(template
+  (content)
+  (template_variable
+    (filtered_value
+      (value
+        (literal
+          (string)))
+      (filter_expression
+        (filter))))
+  (content))
+
+==================
+Language Bidi Filter
+==================
+
+{{ "de"|language_bidi }}
+
+---
+
+(template
+  (content)
+  (template_variable
+    (filtered_value
+      (value
+        (literal
+          (string)))
+      (filter_expression
+        (filter))))
+  (content))
+
+==================
+Language Name Local Filter
+==================
+
+{{ "de"|language_name_local }}
+
+---
+
+(template
+  (content)
+  (template_variable
+    (filtered_value
+      (value
+        (literal
+          (string)))
+      (filter_expression
+        (filter))))
+  (content))
+
+==================
+Language Name Translated Filter
+==================
+
+{{ "de"|language_name_translated }}
+
+---
+
+(template
+  (content)
+  (template_variable
+    (filtered_value
+      (value
+        (literal
+          (string)))
+      (filter_expression
+        (filter))))
+  (content))
+
+==================
+Language Name Filter With An Argument
+:error
+==================
+
+{{ "de"|language_name:1 }}
+
+---