add license, review README.md, fix found bugs

CLAUDE.md

@@ -138,7 +138,10 @@ arity, or whether a tag is a block tag. Two fallback rules accept them:
 
 - `custom_tag` — `simpleTag($, $.identifier)`, i.e. what `simple_tag`/`inclusion_tag` accept, which is how
   a tag is registered unless it needs the parser itself.
-- the last alternative of `filter`, whose name is `$.identifier` rather than one of the builtin literals.
+- `_custom_filter`, the last alternative of `filter`, whose name is `$.identifier` rather than one of the
+  builtin literals. Hidden, so the tree is what an inline alternative would give, but named so that a
+  grammar extending this one can filter it out of `filter`'s `members` the way it can `custom_tag` — see
+  the extension section in `README.md`.
 
 **A name that only exists inside a tag group has to be reserved instead.** `endif`, `else`, `empty` and
 the rest are tokens only in the state their group opens, so where a tag is named the lexer reads them as an
@@ -277,8 +280,8 @@ Corpus tests live in `test/corpus/*.txt`, one file per tag.
 
 ### Checking against Django itself
 
-The `Pipfile` pins Django so the grammar can be diffed against the real implementation instead of against
-documentation or memory. Both checks are worth repeating whenever tags, filters or literals change:
+`Pipfile.lock` pins Django (`Pipfile` itself says `django = "*"`) so the grammar can be diffed against the
+real implementation instead of against documentation or memory. Both checks are worth repeating whenever tags, filters or literals change:
 
 - **Builtin parity** — enumerate `Engine.default_builtins`, then compare `register.tags` and
   `register.filters` against `grammar.js`. For filters also compare argument arity via
@@ -302,7 +305,11 @@ rule it states. Anything added here that fits that rule belongs there too; anyth
 unavoidable limitation, or a case where this grammar is the more permissive one — stays here.
 
 - Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal,
-  so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here.
+  so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here. `1.` is the
+  one that needs care: `Variable` calls `float()` and then rejects what it parsed if the last character is
+  a dot (`# "2." is invalid`), so a trailing dot is a number only with an exponent after it — `1.e5` is a
+  float and `1.` is not. The `number` regex is spelled to say that, and `test/corpus/template_variable.txt`
+  pins all three.
 - Django has no boolean or `None` token either. `True`, `False` and `None` are lookups, which resolve
   because `BaseContext._reset_dicts` seeds every context with `{"True": True, "False": False, "None": None}`.
   They are `boolean` and `none` literals here, so that a tool reading `queries/locals.scm` does not count

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Daniel Hanson
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,6 +1,35 @@
 # tree-sitter-django
 
-A [tree-sitter](https://tree-sitter.github.io/tree-sitter/) grammar for the Django Template Language (DTL) — the `{% %}` tags and `{{ }}` variables used in Django templates.
+A [tree-sitter](https://tree-sitter.github.io/tree-sitter/) grammar for the Django Template Language (DTL) — the `{% %}` tags, `{{ }}` variables and `{# #}` comments used in Django templates.
+
+## Coverage
+
+Every tag and filter Django ships as a builtin is modelled — 28 tags and 57 filters as of Django 6.1 —
+and so is every tag and filter of the five libraries below. Arguments are part of the grammar rather than
+something a linter checks later, so a builtin written with the wrong ones is a parse error: `{{ x|length:2 }}`
+and `{% now %}` do not parse, because `length` takes no argument and `now` requires one.
+
+The target is Django 6.1, including the builtins Django 6 added (`{% partial %}`, `{% partialdef %}`,
+`{% querystring %}`, `{% csp_nonce_attr %}`). Rather than being written from the documentation, the grammar
+is checked against a real Django — `Pipfile.lock` pins the version — by enumerating `Engine.default_builtins`
+for the tag and filter lists and their arities, and by running candidate snippets through both
+`django.template.Template()` and this parser to compare what each accepts. Where the two deliberately
+disagree is listed under [Divergences](#divergences-from-django).
+
+## Using it
+
+There is no published package yet, and no language bindings — those are coming. What is here today:
+
+- **`src/parser.c` and `src/scanner.c` are generated and committed**, so any tree-sitter host can compile
+  them directly without running the CLI. The scanner is required: block-name matching
+  (`{% block a %}…{% endblock a %}`) and the repeated-argument guards live in it.
+- **`npm run parser-build`** produces `tree-sitter-django.wasm` for a `web-tree-sitter` host or an editor
+  that loads Wasm grammars. It is a build artifact rather than a checked-in one, so it has to be built
+  (or shipped by a release) before anything can load it.
+- **`queries/`** holds the highlight, locals, tags and injection queries an editor loads, plus three files
+  meant for a linter. See [Queries](#queries).
+- **`tree-sitter.json`** names the grammar `django`, scopes it `source.django`, and points at the query
+  files, which is what a tree-sitter host reads to wire all of the above together.
 
 ## Development
 
@@ -12,6 +41,45 @@ npm run parser-build     # build the WASM binary
 npm run playground       # interactively explore the grammar
 ```
 
+`src/parser.c`, `src/grammar.json` and `src/node-types.json` are generated **and committed**, so a change
+to `grammar.js` or `src/scanner.c` is only half made until `parser-generate` has run and the result is
+committed alongside it. `tree-sitter test` compiles the parser itself, so a stale `src/` shows up as tests
+that pass against the wrong grammar.
+
+## Queries
+
+Four of the files in `queries/` are the standard query files an editor loads:
+
+- **`highlights.scm`** — syntax highlighting. Tag and filter names are listed one by one, so a name the
+  grammar knows is captured as `@function` and a name it is only tolerating as `@function.call`.
+- **`locals.scm`** — scopes, definitions and references, so an editor can resolve a variable to the tag
+  that bound it. A definition is a `variable:` field; the scopes are the clauses whose bindings do not
+  outlive them (`{% with %}`, `{% for %}`, `{% empty %}`, `{% block %}`, `{% partialdef %}`,
+  `{% blocktranslate %}`).
+- **`tags.scm`** — the symbol index behind `tree-sitter tag`. A `{% partialdef %}` and the names a
+  `{% load %}` brings in are definitions; every tag and filter used is a reference.
+- **`injections.scm`** — marks template text and `{% verbatim %}` bodies as another language's content, so
+  the HTML around the tags can be parsed by its own grammar. It names no language, leaving the editor to
+  supply one.
+
+The other three are data for a tool rather than queries an editor runs. Each captures the two halves of a
+relation a tree-sitter query cannot express, and the tool does the join:
+
+- **`libraries.scm`** — which library each tag or filter is registered in, and what each `{% load %}`
+  brings in. The grammar never requires the load, so reporting a missing one is a linter's job; a query
+  has no ordering predicate to compare the positions itself.
+- **`exports.scm`** — a name a tag binds in the scope _around_ it rather than the scope it opens. Only
+  `{% blocktranslate %}`'s `asvar` does that, and a locals query cannot say so: it places a definition in
+  the innermost scope containing it and offers no way out again.
+- **`conditionals.scm`** — which bodies render on only some passes, as `@conditional.group` /
+  `@conditional.branch` / `@conditional.default`, so a checker can tell that
+  `{% if a %}{% now "Y" as n %}{% endif %}{{ n }}` may reach `{{ n }}` with nothing bound. A group is
+  exhaustive exactly when one of its branches is a default. A locals query has no notion of a path not
+  taken.
+
+`tree-sitter test` loads every file in the directory, so a pattern naming a node that no longer exists
+fails the suite rather than quietly matching nothing.
+
 ## Django libraries
 
 Five of Django's template libraries are not builtins: their names have to be `{% load %}`ed, or preloaded
@@ -107,7 +175,7 @@ export default grammar(django, {
 });
 ```
 
-Four things to know:
+Five things to know:
 
 - **Whitespace is explicit.** `extras` is empty, because Django splits a tag's contents before parsing any
   argument, so every separator has to be written out. The `part`/`block`/`simpleTag` helpers the base
@@ -116,13 +184,13 @@ Four things to know:
   alternative and your arity applies — `{{ x|shout }}` is now an error, while names you have not modelled
   still fall through to `(filter name: (identifier))`.
 - **A tag with a body needs a `conflicts` entry**, as every clause in the base grammar does:
-  `conflicts: ($, previous) => [...previous, [$.map_clause]]`. The generator offers an associativity
-  instead; taking it silently discards the parse in which the body continues. Its end tag is not reserved
-  either, so a stray `{% endmap %}` parses as a `custom_tag` rather than an error — `reserved` takes no
-  `previous`, so the `tag_name` list cannot be added to a name at a time.
+  `conflicts: ($, previous) => [...previous, [$.map_clause]]` for a `map_clause` holding the body. The
+  generator offers an associativity
+  instead; taking it silently discards the parse in which the body continues.
+- **End tags are not reserved**, a stray `{% endmap %}` parses as a `custom_tag` rather than an error — `reserved` takes no
+  `previous`, so the `tag_name` list cannot be added to a name at a time. Instead you may drop the `custom_tag` rule to avoid parsing end tags as custom tags.
 - **The external scanner has to be re-exported under your grammar's name**, or block-name matching and the
-  repeated-argument guards will not link. `exports` cannot deliver it — that is Node's resolver, and the
-  scanner is C — so give your grammar a `src/scanner.c` that renames the base's five entry points to the
+  repeated-argument guards will not link. Give your grammar a `src/scanner.c` that renames the base's five entry points to the
   ones your generated `parser.c` calls, and includes it:
 
 ```c
@@ -141,6 +209,26 @@ path, hence the relative `#include`; the package also exports `tree-sitter-djang
 build step can hand the compiler an absolute path from `import.meta.resolve` where a package manager's
 layout makes the relative one unreliable.
 
+**Turning the fallbacks off.** A grammar that models every tag and filter its project uses probably does
+not want `custom_tag` and `_custom_filter` accepting unrecognized names as they allow typos to parse. `previous` is the base rule whose `members` are filterable. We can filter out these rules to make the grammar more strict:
+
+```js
+template_block_groups: ($, previous) =>
+  choice(
+    ...previous.members.filter((m) => m.name !== "custom_tag"),
+    $.map_tag,
+  ),
+
+filter: ($, previous) =>
+  choice(
+    ...previous.members.filter((m) => m.name !== "_custom_filter"),
+    seq(field("name", "shout"), ":", field("argument", $.value)),
+  ),
+```
+
+`{% mytag %}` and `{{ x|unknown }}` are then errors, while the builtins, the libraries and your own rules
+carry on as before.
+
 ## Divergences from Django
 
 The grammar aims to accept what Django accepts, with one deliberate exception. Where Django takes an
@@ -150,7 +238,8 @@ Every case below is input Django itself accepts:
 
 - **Number-shaped names.** Django has no number token: a word is a literal if `int()` or `float()` parses
   it and a variable lookup otherwise, so `0x1f`, `1a`, `1.`, `1.2.3` and `1__0` look up a variable of that
-  name and render nothing.
+  name and render nothing. (`1.` really is a lookup there and not a float — `Variable` rejects a trailing
+  dot after parsing it — while `1.e5` is a float, and both are treated that way here.)
 - **An attribute of `True`, `False` or `None`.** Every Django context is seeded with those three names, so
   `{{ True.x }}` is an ordinary two-part lookup that can never resolve.
 - **A keyword argument written twice.** `{% with a=1 a=2 %}`, and the same in `{% include %}` and
@@ -164,3 +253,7 @@ Every case below is input Django itself accepts:
 
 In the other direction the grammar is more permissive only where it cannot know better, which is what the
 fallbacks above are for.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

grammar.js

@@ -395,10 +395,17 @@ const django = grammar({
     value: ($) => choice($.literal, $.variable_attribute),
     literal: ($) =>
       choice($.number, $.string, $.translated_string, $.boolean, $.none),
-    // digits, with single underscores allowed between them, as Python's
-    // int() and float() accept ("1_000") and Django's Variable relies on
+    /**
+     * Digits, with single underscores allowed between them, as Python's int()
+     * and float() accept ("1_000") and Django's Variable relies on.
+     *
+     * A trailing dot needs an exponent after it. Variable calls float() and
+     * then rejects what it just parsed if the last character is a dot — the
+     * comment in Django reads `# "2." is invalid` — so "1." is a two-part
+     * lookup there rather than a number, while "1.e5" is a float.
+     */
     number: ($) =>
-      /[-+]?(?:[0-9](?:_?[0-9])*(?:\.(?:[0-9](?:_?[0-9])*)?)?|\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?/,
+      /[-+]?(?:(?:[0-9](?:_?[0-9])*(?:\.[0-9](?:_?[0-9])*)?|\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?|[0-9](?:_?[0-9])*\.[eE][0-9](?:_?[0-9])*)/,
     string: ($) => STRING,
     /**
      * A string to translate at render time. Django matches "_(" and ")" as
@@ -549,12 +556,19 @@ const django = grammar({
           field("name", choice(...FILTERS_WITH_OPTIONAL_ARGUMENT)),
           optional(seq(":", field("argument", $.value))),
         ),
-        // a filter a "load" brought in: its name is not known here, and so
-        // neither is whether it takes an argument
-        seq(
-          field("name", $.identifier),
-          optional(seq(":", field("argument", $.value))),
-        ),
+        $._custom_filter,
+      ),
+    /**
+     * A filter a "load" brought in: its name is not known here, and so neither
+     * is whether it takes an argument. Hidden, so the tree is the same as if
+     * it were spelled inline, but named so that a grammar extending this one
+     * can drop it from filter's alternatives the way it can drop custom_tag
+     * from template_block_groups.
+     */
+    _custom_filter: ($) =>
+      seq(
+        field("name", $.identifier),
+        optional(seq(":", field("argument", $.value))),
       ),
     autoescape_clause: ($) =>
       seq(block("autoescape", part(choice("on", "off"))), optional($.template)),

src/grammar.json

@@ -135,7 +135,7 @@
     },
     "number": {
       "type": "PATTERN",
-      "value": "[-+]?(?:[0-9](?:_?[0-9])*(?:\\.(?:[0-9](?:_?[0-9])*)?)?|\\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?"
+      "value": "[-+]?(?:(?:[0-9](?:_?[0-9])*(?:\\.[0-9](?:_?[0-9])*)?|\\.[0-9](?:_?[0-9])*)(?:[eE][0-9](?:_?[0-9])*)?|[0-9](?:_?[0-9])*\\.[eE][0-9](?:_?[0-9])*)"
     },
     "string": {
       "type": "PATTERN",
@@ -1313,40 +1313,44 @@
           ]
         },
         {
-          "type": "SEQ",
+          "type": "SYMBOL",
+          "name": "_custom_filter"
+        }
+      ]
+    },
+    "_custom_filter": {
+      "type": "SEQ",
+      "members": [
+        {
+          "type": "FIELD",
+          "name": "name",
+          "content": {
+            "type": "SYMBOL",
+            "name": "identifier"
+          }
+        },
+        {
+          "type": "CHOICE",
           "members": [
             {
-              "type": "FIELD",
-              "name": "name",
-              "content": {
-                "type": "SYMBOL",
-                "name": "identifier"
-              }
-            },
-            {
-              "type": "CHOICE",
+              "type": "SEQ",
               "members": [
                 {
-                  "type": "SEQ",
-                  "members": [
-                    {
-                      "type": "STRING",
-                      "value": ":"
-                    },
-                    {
-                      "type": "FIELD",
-                      "name": "argument",
-                      "content": {
-                        "type": "SYMBOL",
-                        "name": "value"
-                      }
-                    }
-                  ]
+                  "type": "STRING",
+                  "value": ":"
                 },
                 {
-                  "type": "BLANK"
+                  "type": "FIELD",
+                  "name": "argument",
+                  "content": {
+                    "type": "SYMBOL",
+                    "name": "value"
+                  }
                 }
               ]
+            },
+            {
+              "type": "BLANK"
             }
           ]
         }

test/corpus/template_variable.txt

@@ -461,3 +461,40 @@ Template Variable Attribute Of A Literal
 
 ---
 
+
+==================
+Template Variable Trailing Dot With An Exponent
+==================
+
+{{ 1.e5 }}
+
+---
+
+(template
+  (content)
+  (template_variable
+    (filtered_value
+      (value
+        (literal
+          (number)))))
+  (content))
+
+==================
+Template Variable Trailing Dot
+:error
+==================
+
+{{ 1. }}
+
+---
+
+
+==================
+Template Variable Signed Trailing Dot
+:error
+==================
+
+{{ -1. }}
+
+---
+