Skip to content

grammarPatch: deep / keys throw TypeError when the path crosses a string inside (e.g. inside: 'markup'); silently no-ops on bare-RegExp tokensΒ #4176

Open

Description

@rahul05ranjan

Description

Two problems in the v2 grammar patcher (src/util/grammar-patch.js, reached via Language.resolvedGrammar, src/core/classes/language.js:239) when the grammar contains values that are perfectly legal according to types.d.ts but are not plain objects.

1. Deep $insertBefore/$insertAfter keys crash when the path crosses a string inside

// src/util/grammar-patch.js:17-28 ($insertBefore; same shape at :42-52 for $insertAfter)
if (key?.includes('/')) {
    let path = key.split('/');
    const lastKey = path.pop();
    path = path.flatMap(key => [key, 'inside']);
    const obj = path.reduce((acc, key) => acc?.[key], grammar);

    if (obj) {
        insertBefore(obj, lastKey, tokens);
    }
}

The walk's guard is if (obj) β€” a plain truthiness check. But at grammarPatch time the grammar is still raw β€” token insides are commonly stored as language-id strings (inside?: string | Grammar | null, types.d.ts:219; used throughout src/languages/, e.g. actionscript.js:25, apex.js:44). If a deep key walks into such a token, obj is a string:

const grammar = {
    'code': { pattern: /`[^`]*`/, inside: 'markup' },  // type-legal GrammarRef
    $insertBefore: {
        'code/comment': { 'my-token': { pattern: /Y/ } },
    },
};
grammarPatch(grammar);
// TypeError: Cannot use 'in' operator to search for 'comment' in markup

The TypeError comes from insert() (src/util/insert.js:57: if (!(atKey in grammar))) being handed the string 'markup' as its "grammar". Since grammarPatch is what resolvedGrammar runs, this exception escapes to whoever touches the language β€” the language simply fails to load, with an error message that doesn't point anywhere near the user's $insertBefore key.

Two neighbouring shapes misbehave too:

  • intermediate inside is a bare RegExp token (RegExpLike = RegExp, types.d.ts:183): no TypeError, but a misleading Error: "comment" has to be a key of grammar. β€” thrown about a RegExp object the user never named.
  • intermediate inside is a function (GrammarRef explicitly allows "a function returning one"): 'x' in fn is legal, and if lastKey matches an own property of the function (name, length, prototype), insert() happily deletes and redefines the function's own properties and glues the new tokens onto the function object.

Repro output (running the real grammarPatch under node, current master):

--- Case 1: intermediate inside is a string (language id) ---
THROWS: TypeError: Cannot use 'in' operator to search for 'comment' in markup

--- Case 2: intermediate inside is a bare RegExp token ---
THROWS: Error: "comment" has to be a key of grammar.

--- Case 3: control, plain-object inside (works) ---
code.inside keys now: [ 'my-token', 'comment' ]

Expected: the walk should either treat non-object insides as "not patchable" (skip, like if (obj) does for undefined/null), or throw a descriptive error mentioning the $insertBefore/$insertAfter key. Ideally string insides would be resolved (like resolve() in src/core/tokenize/util.js does at tokenize time) so 'code/comment' can patch into the referenced language β€” but at minimum it must not crash with an internal TypeError on type-legal input.

2. $merge silently does nothing when the target token is a bare RegExp

// src/util/grammar-patch.js:76-81
if (grammar[key]) {
    deepMerge(grammar[key], tokens);
}
else {
    grammar[key] = tokens;
}

deepMerge (src/util/objects.js:104-125) only mutates when both sides are plain Object/EventTarget containers; otherwise it returns source ?? target β€” and the caller discards the return value. So if the existing token is a bare RegExp (type-legal: GrammarTokens = Partial<Record<TokenName, RegExpLike | GrammarToken | ...>>), the whole $merge entry is silently dropped:

const grammar = {
    'string': /"[^"]*"/,        // type-legal RegExpLike token
    $merge: { 'string': { greedy: true } },
};
grammarPatch(grammar);
grammar['string'];  // still /"[^"]*"/ β€” greedy lost, no error, no warning
--- Case 4: $merge silently no-ops when target token is a bare RegExp ---
after $merge, string token = /"[^"]*"/ (greedy lost)

--- Case 5: control, $merge into object token (works) ---
after $merge, string token = { pattern: /"[^"]*"/, greedy: true }

Expected: merge into a RegExp token by replacing it with { ...tokens, pattern: <the RegExp> } (or at least overwrite the token), not silently ignore the patch.

Environment

Prism 2.0 (current master, src/util/grammar-patch.js at commit of today). Both cases reproduced by importing grammarPatch directly β€” no browser needed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions