add _missing_variable_close

CLAUDE.md

@@ -284,7 +284,7 @@ README nothing checks at all. The library-contents differential below, run again
 
 ### External scanner (`src/scanner.c`)
 
-Nineteen external tokens, for the constraints a context-free grammar cannot express:
+Twenty external tokens, for the constraints a context-free grammar cannot express:
 
 - `push_block`/`push_partial`/`push_verbatim` and the matching `pop_*` — the names in
   `{% block a %}…{% endblock a %}`. A push token reads the name into `pending_name`, which the group takes
@@ -346,6 +346,11 @@ Nineteen external tokens, for the constraints a context-free grammar cannot expr
   brace. `content` is a right-associative run of `_text` and `_text_brace`, so it stays one node. Inside a
   group's body the missing-tag check reads past `{` first, so it scans the brace itself
   (`scan_text_brace_rest`) rather than give up the position.
+- `_missing_variable_close` — the marker where a variable's `}}` is missing, aliased to
+  `missing_variable_close`. It is placed only where the variable cannot go on — at the end of input, or
+  before a `}` that is not `}}`, a `%` or a `{` — so `{{ hello }{% if x %}` closes the variable, reads the
+  stray `}` as text and parses the `if` normally, where error recovery used to take the rest of the file.
+  It is valid only after a variable's value, never where `content` is. It added 18 states.
 - `matcher_error` — used by no rule; returned only if the scanner reaches an error state.
 
 `check_space()` defines whitespace for the scanner and **must stay in sync with `_sep`**, because the
@@ -409,8 +414,9 @@ A branch that is also a `@local.scope` confines its bindings whichever way the b
 `context.push()` as the loop body, so `empty_block` is a scope alongside `for_block`.
 
 `queries/errors.scm` collects what an editor should report: `(ERROR)` as `@error.syntax`, `(MISSING)` as
-`@error.missing`, every missing-tag marker as `@error.missing_tag`, and `unexpected_tag` and
+`@error.missing`, every missing-tag marker as `@error.missing_tag`, `unexpected_tag` and
-`unexpected_argument` as `@error.unexpected_tag` and `@error.unexpected_argument`. A tree holding only markers has
+`unexpected_argument` as `@error.unexpected_tag` and `@error.unexpected_argument`, and
+`missing_variable_close` as `@error.missing_variable_close`. A tree holding only markers has
 no `has_error`, so a tool that checks that flag alone misses them. Adding a tag group means adding its
 marker there, since no supertype matches them all.
 
@@ -492,3 +498,8 @@ unavoidable limitation, or a case where this grammar is the more permissive one
   `{% for %}` closed while an `{% if %}` inside it is still open, holds a `missing_endif_tag` marker
   where the end tag belongs. The marker is the error, reported through `queries/errors.scm`; a check
   for `ERROR` nodes alone accepts the tree.
+- A `{{` that does not close is text in Django, whose lexer only takes a variable written `{{ … }}` on one
+  line, so `{{ hello }` renders as written. Here it is a `template_variable` closed by a
+  `missing_variable_close` marker, reported through `queries/errors.scm`, since a half-written variable is
+  what it almost always is. The same lexer makes `{{ hello\n}}` text too, which this grammar accepts as a
+  variable, because its separators allow newlines.

README.md

@@ -70,7 +70,8 @@ Five of the files in `queries/` are query files an editor loads:
   belongs to an enclosing group, or before an `{% endblock %}` naming a block further out.
   A group's tag written where no open group can hold it, such as `{% else %}` inside a `{% for %}`, is an
   `unexpected_tag`, captured as `@error.unexpected_tag`. Words a tag does not take, such as the `x` in
-  `{% else x %}`, are an `unexpected_argument` inside that tag, captured as `@error.unexpected_argument`.
+  `{% else x %}`, are an `unexpected_argument` inside that tag, captured as `@error.unexpected_argument`. A `{{` whose `}}` is missing closes with a zero-width
+  `missing_variable_close`, captured as `@error.missing_variable_close`, so the rest of the template still parses.
 
 The other three are data for a tool rather than queries an editor runs. Each captures the two halves of a
 relation a tree-sitter query cannot express, and the tool does the join:

grammar.js

@@ -566,6 +566,7 @@ const django = grammar({
     $._group_close,
     $._group_held,
     $._text_brace,
+    $._missing_variable_close,
   ],
   reserved: {
     global: ($) => ["not", "if", "in", "is", "as", "for", "from"],
@@ -677,13 +678,22 @@ const django = grammar({
           ),
         ),
       ),
+    /**
+     * A variable whose "}}" is missing closes with a zero-width marker instead,
+     * so that error recovery does not take the rest of the file with it. The
+     * scanner places it only where the variable cannot go on: at the end of
+     * input, or before a "}" that is not "}}", a "%" or a "{".
+     */
     template_variable: ($) =>
       seq(
         "{{",
         optional(SEP),
         alias($._filtered_value_spaced, $.filtered_value),
         optional(SEP),
-        "}}",
+        choice(
+          "}}",
+          alias($._missing_variable_close, $.missing_variable_close),
+        ),
       ),
     /**
      * Everything between "{{" and "}}" reaches Django as a single expression,

queries/errors.scm

@@ -43,3 +43,5 @@
 (unexpected_tag) @error.unexpected_tag
 
 (unexpected_argument) @error.unexpected_argument
+
+(missing_variable_close) @error.missing_variable_close

src/grammar.json

@@ -986,8 +986,22 @@
           ]
         },
         {
-          "type": "STRING",
+          "type": "CHOICE",
-          "value": "}}"
+          "members": [
+            {
+              "type": "STRING",
+              "value": "}}"
+            },
+            {
+              "type": "ALIAS",
+              "content": {
+                "type": "SYMBOL",
+                "name": "_missing_variable_close"
+              },
+              "named": true,
+              "value": "missing_variable_close"
+            }
+          ]
         }
       ]
     },
@@ -8131,6 +8145,10 @@
     {
       "type": "SYMBOL",
       "name": "_text_brace"
+    },
+    {
+      "type": "SYMBOL",
+      "name": "_missing_variable_close"
     }
   ],
   "inline": [

src/node-types.json

@@ -3340,12 +3340,16 @@
     "named": true,
     "fields": {},
     "children": {
-      "multiple": false,
+      "multiple": true,
       "required": true,
       "types": [
         {
           "type": "filtered_value",
           "named": true
+        },
+        {
+          "type": "missing_variable_close",
+          "named": true
         }
       ]
     }
@@ -4458,6 +4462,10 @@
     "type": "missing_plural_tag",
     "named": true
   },
+  {
+    "type": "missing_variable_close",
+    "named": true
+  },
   {
     "type": "none",
     "named": true

src/scanner.c

@@ -60,6 +60,8 @@ enum TokenType {
   /* a "{" that is text although another "{" follows it, because the second
    * one opens a tag or a comment, or although nothing follows it at all */
   TextBrace,
+  /* zero width, where a variable's "}}" is missing */
+  MissingVariableClose,
 };
 
 /* The options blocktranslate takes, in the order of the bits recording them.
@@ -956,6 +958,27 @@ static bool scan_raw_content(
   }
 }
 
+/* Scans the marker for a variable whose "}}" is missing: at the end of input,
+ * or before a "}" that is not "}}", a "%" or a "{", none of which can continue
+ * the variable. Django would read such a "{{" as text, since its lexer only
+ * takes a variable that closes on the same line; the marker reports it as the
+ * unfinished variable it almost certainly is. */
+static bool scan_missing_variable_close(TSLexer *const lexer) {
+  lexer->mark_end(lexer);
+  if (!lexer->eof(lexer)) {
+    if (lexer->lookahead == '}') {
+      lexer->advance(lexer, false);
+      if (lexer->lookahead == '}') {
+        return false;
+      }
+    } else if (lexer->lookahead != '%' && lexer->lookahead != '{') {
+      return false;
+    }
+  }
+  lexer->result_symbol = MissingVariableClose;
+  return true;
+}
+
 bool tree_sitter_django_external_scanner_scan(
   void *payload,
   TSLexer *lexer,
@@ -974,6 +997,9 @@ bool tree_sitter_django_external_scanner_scan(
   if (valid_symbols[MatcherError]) {
     return false;
   }
+  if (valid_symbols[MissingVariableClose]) {
+    return scan_missing_variable_close(lexer);
+  }
   /* The zero-width tokens below belong here: above the raw-text scanners,
    * which find nothing at the end of an empty body, above the loop further
    * down, which skips the separator the grammar still has to match, and below

test/corpus/template_variable.txt

@@ -211,7 +211,6 @@ Template Variable Literal Adjacent To Name
 ---
 
 
-
 ==================
 Template Variable With Escaped String
 ==================
@@ -273,7 +272,6 @@ Template Variable Attribute Beginning With Underscore
 ---
 
 
-
 ==================
 Template Variable Translated String
 ==================
@@ -322,7 +320,6 @@ Template Variable Translated String With Spaces
 ---
 
 
-
 ==================
 Template Variable Spanning Lines
 ==================
@@ -465,7 +462,6 @@ Template Variable Attribute Of A Literal
 ---
 
 
-
 ==================
 Template Variable Trailing Dot With An Exponent
 ==================
@@ -493,7 +489,6 @@ Template Variable Trailing Dot
 ---
 
 
-
 ==================
 Template Variable Signed Trailing Dot
 :error
@@ -504,7 +499,6 @@ Template Variable Signed Trailing Dot
 ---
 
 
-
 ==================
 Braces That Open Nothing Are Text
 ==================
@@ -608,7 +602,6 @@ a{{{ x }}
 ---
 
 
-
 ==================
 A Filter Pipe With No Filter Stays In Its Variable
 ==================
@@ -646,3 +639,49 @@ hello
         (content)))
     (endif_tag))
   (content))
+
+==================
+A Variable Missing Its Close Ends Where It Cannot Continue
+==================
+
+{{ hello }{% if display %}body {% endif %}
+
+---
+
+(template
+  (content)
+  (template_variable
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (missing_variable_close))
+  (content)
+  (if_group
+    (if_block
+      (if_tag
+        (predicate
+          (filtered_value
+            (value
+              (variable_attribute
+                (identifier))))))
+      (template
+        (content)))
+    (endif_tag))
+  (content))
+
+==================
+A Variable Missing Its Close At The End
+==================
+
+{{ hello
+---
+
+(template
+  (content)
+  (template_variable
+    (filtered_value
+      (value
+        (variable_attribute
+          (identifier))))
+    (missing_variable_close)))