"TypeError: chunks[startIndex].slice is not a function" during call to postprocess when used with micromark-extension-directive

[DavidAnson]

Initial checklist

• I read the support docs
• I read the contributing guide
• I agree to follow the code of conduct
• I searched issues and couldn’t find anything (or linked relevant results below)

Affected packages and versions

[email protected], [email protected]

Link to runnable example

No response

Steps to reproduce

user@HOST mmi % npm ls
mmi@ /Users/user/Documents/mmi
├── [email protected]
└── [email protected]

user@HOST mmi % cat issue.mjs 
import { parse, postprocess, preprocess } from "micromark";
import { directive } from "micromark-extension-directive";

function invokeMicromark(markdown) {
  console.log("trying...");
  const encoding = undefined;
  const end = true;
  const options = {
    "extensions": [ directive() ]
  };
  const chunks = preprocess()(markdown, encoding, end);
  const parseContext = parse(options).document().write(chunks);
  const events = postprocess(parseContext);
  for (const event of events) {
    const [ _, token, context ] = event;
    context.sliceSerialize(token);
  }
  console.log("ok");
}

invokeMicromark(
`* some text
   :::note

a`
);

// invokeMicromark(
// `* some text
//    :::note
//
// inner text
// :::`
// );

// invokeMicromark(
// `- some text
//    :::note
//
// inner text
//
// :::
//
// - some more text`
// );

// invokeMicromark(
// `1. some text
//    :::note
//
// inner text
//
// :::
//
// 2. some more text`
// );
user@HOST mmi % node issue.mjs 
trying...
file:///Users/user/Documents/mmi/node_modules/micromark/lib/create-tokenizer.js:512
    view = [chunks[startIndex].slice(startBufferIndex, endBufferIndex)]
                               ^

TypeError: chunks[startIndex].slice is not a function
    at sliceChunks (file:///Users/user/Documents/mmi/node_modules/micromark/lib/create-tokenizer.js:512:32)
    at Object.sliceStream (file:///Users/user/Documents/mmi/node_modules/micromark/lib/create-tokenizer.js:153:12)
    at subcontent (file:///Users/user/Documents/mmi/node_modules/micromark-util-subtokenize/index.js:150:24)
    at subtokenize (file:///Users/user/Documents/mmi/node_modules/micromark-util-subtokenize/index.js:71:30)
    at postprocess (file:///Users/user/Documents/mmi/node_modules/micromark/lib/postprocess.js:12:11)
    at invokeMicromark (file:///Users/user/Documents/mmi/issue.mjs:13:18)
    at file:///Users/user/Documents/mmi/issue.mjs:21:1
    at ModuleJob.run (node:internal/modules/esm/module_job:222:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:323:24)
    at async loadESM (node:internal/process/esm_loader:28:7)

Node.js v20.12.1
user@HOST mmi %

Expected behavior

No Error thrown.

Please see very related #131 for a similar micromark issue that was raised and fixed in January of 2023.

Please see DavidAnson/markdownlint-cli2#325 for the user report of this problem happening with the markdownlint library. I've included all 4 scenarios in the sample code here, though I expect they're all the same underlying issue.

Note that un-indenting the ::: line avoids the problem.

Actual behavior

Error, see above.

Runtime

Node v16

Package manager

npm v7

OS

macOS

Build and bundle tools

Other (please specify in steps to reproduce)

[DavidAnson]

Please note that unlike the previous/similar issue, it is not necessary for the caller to walk events. The relevant part of the repro can be simplified to the following which is more typical micromark API use, I think:

  const chunks = preprocess()(markdown, encoding, end);
  const parseContext = parse(options).document().write(chunks);
  postprocess(parseContext); // crashes in this call

[DavidAnson]

And actually, the same crash can be reproduced via remark/remark-directive directly:

user@HOST mmi % npm ls
mmi@ /Users/user/Documents/mmi
├── [email protected]
├── [email protected]
├── [email protected]
└── [email protected]

user@HOST mmi % cat bug.md
* some text
   :::note

a%
user@HOST mmi % ./node_modules/.bin/remark bug.md --use remark-directive
bug.md
 error Cannot process file
  [cause]:
    TypeError: chunks[startIndex].slice is not a function
    at sliceChunks (file:///Users/user/Documents/mmi/node_modules/micromark/lib/create-tokenizer.js:512:32)
    at Object.sliceStream (file:///Users/user/Documents/mmi/node_modules/micromark/lib/create-tokenizer.js:153:12)
    at subcontent (file:///Users/user/Documents/mmi/node_modules/micromark-util-subtokenize/index.js:150:24)
    at subtokenize (file:///Users/user/Documents/mmi/node_modules/micromark-util-subtokenize/index.js:71:30)
    at postprocess (file:///Users/user/Documents/mmi/node_modules/micromark/lib/postprocess.js:12:11)
    at fromMarkdown (file:///Users/user/Documents/mmi/node_modules/mdast-util-from-markdown/lib/index.js:174:5)
    at parser (file:///Users/user/Documents/mmi/node_modules/remark-parse/lib/index.js:33:12)
    at Function.parse (file:///Users/user/Documents/mmi/node_modules/unified/lib/index.js:668:12)
    at parse (file:///Users/user/Documents/mmi/node_modules/unified-engine/lib/file-pipeline/parse.js:59:36)
    at wrapped (file:///Users/user/Documents/mmi/node_modules/trough/lib/index.js:160:27)

✖ 1 error
user@HOST mmi %

[ChristianMurphy]

Possibly a duplicate of the continuation error from micromark/micromark-extension-directive#16