diff --git a/CLAUDE.md b/CLAUDE.md index e7667ab..d6ff871 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -181,6 +181,14 @@ Modelling a library name makes it keyword-extracted, so it no longer reaches `cu registers its own tag under one of these names and a different signature therefore gets a parse error. That is accepted: these names are common enough that shadowing them is the mistake. +**Adding a library tag or filter touches four files**: `grammar.js`, `queries/libraries.scm` (which library +it came from), `queries/highlights.scm` (the name, spelled out) and the table in `README.md`, which lists +every library name the grammar models. All four are hand-maintained, and **no test catches a name left out +of any of them**: `tree-sitter test` loads every `.scm` under `queries/`, so a pattern naming a node that +no longer exists aborts the whole run, but a name that was never added passes 304/304 in silence, and the +README nothing checks at all. The library-contents differential below, run against +`django.templatetags..register`, is what finds a gap. + ### External scanner (`src/scanner.c`) Twelve external tokens, for the constraints a context-free grammar cannot express: @@ -288,6 +296,11 @@ documentation or memory. Both checks are worth repeating whenever tags, filters ### Deliberate divergences from Django +`README.md` carries a shorter list of its own, and the two are not the same set on purpose: the README +lists only the divergences where Django accepts an input that is almost certainly a mistake, which is the +rule it states. Anything added here that fits that rule belongs there too; anything that does not — an +unavoidable limitation, or a case where this grammar is the more permissive one — stays here. + - Django has no number token: `[\w.]+` is one lexeme and `int()`/`float()` decides whether it is a literal, so `1a`, `1e`, `0x1f`, `1.2.3`, `1__0` and `1.` are variable _lookups_ there and errors here. - Django has no boolean or `None` token either. `True`, `False` and `None` are lookups, which resolve diff --git a/README.md b/README.md index 1d25857..6e93695 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,135 @@ npm run parser-build # build the WASM binary npm run playground # interactively explore the grammar ``` +## Django libraries + +Five of Django's template libraries are not builtins: their names have to be `{% load %}`ed, or preloaded +through the engine's `OPTIONS: {"builtins": [...]}`. Every tag and filter all five register is modelled, +with the same argument and arity rules as a builtin: + +| library | tags | filters | +| -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | +| `i18n` | `trans`, `translate`, `blocktrans`, `blocktranslate`, `language`, `get_available_languages`, `get_current_language`, `get_current_language_bidi`, `get_language_info`, `get_language_info_list` | `language_bidi`, `language_name`, `language_name_local`, `language_name_translated` | +| `l10n` | `localize` | `localize`, `unlocalize` | +| `static` | `static`, `get_static_prefix`, `get_media_prefix` | — | +| `cache` | `cache` | — | +| `tz` | `localtime`, `timezone`, `get_current_timezone` | `localtime`, `timezone`, `utc` | + +**The `{% load %}` is never required**, because an engine may have preloaded the library. Whether one is +present is a question about the order of nodes rather than about how the template parses, so it is left to +a tool: `queries/libraries.scm` marks each name with the library it came from and each `{% load %}` with +what it brings in, and the tool joins the two. Such a tool needs to know that loading is positional +(`{% trans "x" %}{% load i18n %}` is an error), that it is per file rather than inherited through +`{% extends %}` or `{% include %}`, and that `{% load trans from i18n %}` brings in only the names it lists. + +Because these names are modelled, they win over the fallbacks below: a project that registers its own tag +under one of them with a different signature gets a parse error. They are common enough that shadowing +them is treated as the mistake. + +## Custom tags and filters + +A library's tags and filters are registered in Python, so the grammar cannot know their names, their +arity, or whether a tag opens a body. Two fallback rules accept them: + +- **Tags.** Any tag the grammar does not recognise parses as `custom_tag`, with the shape `simple_tag` and + `inclusion_tag` give: positional filter expressions, then `name=value` keyword arguments, then an + optional `as name`. So `{% mytag a b|upper "s" k=1 as out %}` parses, with the name on the `tag:` field. + A keyword argument written twice is refused, as `parse_bits` refuses it. +- **Filters.** An unknown filter parses as `(filter name: (identifier))` with at most one `:argument`, + which is all Django's expression regex allows. No arity check is possible. + +Both are captured as `@function.call` rather than `@function`, so an editor can distinguish a name the +grammar knows from one it is only tolerating. + +What the fallbacks cannot do: + +- **A block tag is not nested.** `{% mytag %}…{% endmytag %}` is two sibling `custom_tag`s; pairing them + would mean guessing that an unknown tag opens a body. +- **A tag registered with `@register.tag` may parse anything**, since it receives the raw token, so one + whose arguments are not the `simple_tag` shape — `{% mytag <<>> %}` — is an error here and not in Django. +- **A misspelled name cannot be caught.** `{{ x|lenght }}` is indistinguishable from a filter some library + registered. Cross-referencing names against `{% load %}` needs the project's Python, so it belongs in a + linter; `queries/libraries.scm` and the `load` rule supply what one needs. +- **Names that only exist inside a tag group are reserved**, so `{% endif %}` or `{% empty %}` standing on + its own is an error rather than a call to a custom tag of that name. + +### Adding rules for your own tags and filters + +For a project whose custom tags deserve real rules rather than the fallbacks, use this grammar as a base +grammar. `grammar(base, {...})` merges your rules into it, and a rule that redefines one the base already +has receives it as `previous`. A tag is an alternative of `template_block_groups`; a filter is an +alternative of `filter`. + +```js +import django from "tree-sitter-django/grammar"; + +const SEP = /[ \t\r\n]+/; + +export default grammar(django, { + name: "djangox", + + rules: { + // {% map items over rows %} + template_block_groups: ($, previous) => choice(previous, $.map_tag), + map_tag: ($) => + seq( + "{%", + optional(SEP), + "map", + SEP, + $.filtered_value, + SEP, + "over", + SEP, + $.filtered_value, + optional(SEP), + "%}", + ), + + // {{ value|shout:"!" }} + filter: ($, previous) => + choice( + previous, + seq(field("name", "shout"), ":", field("argument", $.value)), + ), + }, +}); +``` + +Four things to know: + +- **Whitespace is explicit.** `extras` is empty, because Django splits a tag's contents before parsing any + argument, so every separator has to be written out. The `part`/`block`/`simpleTag` helpers the base + grammar is built from are private to it, hence the `SEP` above. +- **A new name wins over the fallback.** `shout` becomes a token of its own, so it commits to your + alternative and your arity applies — `{{ x|shout }}` is now an error, while names you have not modelled + still fall through to `(filter name: (identifier))`. +- **A tag with a body needs a `conflicts` entry**, as every clause in the base grammar does: + `conflicts: ($, previous) => [...previous, [$.map_clause]]`. The generator offers an associativity + instead; taking it silently discards the parse in which the body continues. Its end tag is not reserved + either, so a stray `{% endmap %}` parses as a `custom_tag` rather than an error — `reserved` takes no + `previous`, so the `tag_name` list cannot be added to a name at a time. +- **The external scanner has to be re-exported under your grammar's name**, or block-name matching and the + repeated-argument guards will not link. `exports` cannot deliver it — that is Node's resolver, and the + scanner is C — so give your grammar a `src/scanner.c` that renames the base's five entry points to the + ones your generated `parser.c` calls, and includes it: + +```c +#define tree_sitter_django_external_scanner_create tree_sitter_djangox_external_scanner_create +#define tree_sitter_django_external_scanner_destroy tree_sitter_djangox_external_scanner_destroy +#define tree_sitter_django_external_scanner_scan tree_sitter_djangox_external_scanner_scan +#define tree_sitter_django_external_scanner_serialize tree_sitter_djangox_external_scanner_serialize +#define tree_sitter_django_external_scanner_deserialize tree_sitter_djangox_external_scanner_deserialize + +#include "../node_modules/tree-sitter-django/src/scanner.c" +``` + +The base's `tree_sitter/parser.h` then resolves to your own generated copy, and the externals keep the +order they have here, so the scanner's token enum stays valid. `tree-sitter build` takes no include +path, hence the relative `#include`; the package also exports `tree-sitter-django/src/scanner.c`, so a +build step can hand the compiler an absolute path from `import.meta.resolve` where a package manager's +layout makes the relative one unreliable. + ## Divergences from Django The grammar aims to accept what Django accepts, with one deliberate exception. Where Django takes an @@ -33,6 +162,5 @@ Every case below is input Django itself accepts: in that position, so `{% cache 500 using="c" %}` names a fragment called `using="c"` rather than choosing a cache, and `{% cache 500 sidebar.name %}` names one with a dot in it. -In the other direction the grammar is more permissive only where it cannot know better: the names a -`{% load %}` brings in live in Python, so `{{ x|lenght }}` cannot be told from a filter some library -registered, and an unknown tag's arguments are assumed to have the shape `simple_tag` gives them. +In the other direction the grammar is more permissive only where it cannot know better, which is what the +fallbacks above are for. diff --git a/package.json b/package.json index 1cc9d62..e7af9c6 100644 --- a/package.json +++ b/package.json @@ -4,6 +4,12 @@ "private": true, "license": "MIT", "type": "module", + "exports": { + "./grammar": "./grammar.js", + "./src/scanner.c": "./src/scanner.c", + "./queries/*": "./queries/*", + "./package.json": "./package.json" + }, "scripts": { "lint": "eslint", "lint:fix": "eslint --fix",