Add math support #366

digitalmoksha · 2024-02-18T22:23:43Z

See #366 (comment) for updated details.

This will add support for math syntax, both the dollar math and the newer GitLab/GFM syntax. Relates to #23

Display math is mostly working, $$ and ```math syntax. Inline not at all yet.

A couple notes:

there is no inline math support yet
Currently the $$ syntax follows the rules of code blocks. Meaning that
```
This is a paragraph
$$
1 + 2
$$
and this is a separate paragraph.
```
generates a paragraph followed by a block and then another paragraph. I've seen some implementations where it is all in one paragraph. I'm torn which way it should be. At the moment this is the easiest.
The $$ math blocks are generated the same way that ```math blocks are generated, as a <pre><code> block. This keeps the code and the output consistent. It also means that if math rendering is not enabled on a site, the math will display as a code block, rather than a bunch of run-together text, without any extra styling needed.
I'm using data attributes rather than classes. So it's <pre data-math-style="display"><code>... rather than <pre class="math"><code>....

I'm flexible on these decisions, this is just where it's at right now.

digitalmoksha · 2024-03-23T02:40:24Z

@kivikakk I think I have this finished 🤞

It's similar to pandoc and commonmark-hs for dollar math, including using <span>. But it uses a data-math-style attribute to determine whether it's inline or display.

Code math (the GitLab syntax), uses <code> instead of <span>, which I think is correct given the markdown syntax. And it gives a way to tell the two syntaxes apart if the HTML needs to be converted back into markdown (which we do).

If you get some time, would you mind reviewing?

`cargo run -- -e math-dollars` enables dollar math.

Inline math

Calc $x^2$ and $$y^2$$

<p>Calc <span data-math-style="inline">x^2</span> and <span data-math-style="display">y^2</span></p>

block math

$$
z^2
$$

<p><span data-math-style="display">
z^2
</span></p>

`cargo run -- -e math-code` enables code/GitLab math.

Inline math

Calc $`x^2`$

<p>Calc <code data-math-style="inline">x^2</code></p>

block math

```math
z^2
```

<pre><code class="language-math" data-math-style="display">z^2
</code></pre>

kivikakk · 2024-03-23T10:02:57Z

heya @digitalmoksha, thank you so much for this! i'm currently preparing for an imminent overseas move, but i'll try to squeeze this in in the next couple days :)

digitalmoksha · 2024-03-24T19:44:19Z

And good luck on your move! Exciting!

kivikakk

I've read through everything closely with the exception of inlines.rs; will look at that more carefully tonight! I did a quick check that "stopping" on $ unilaterally in the parsing loop doesn't affect e.g. autolinks. I'll also do some fuzzing.

src/nodes.rs

src/parser/math.rs

src/tests/math.rs

kivikakk

never mind that, I'm having a flare up and will be out of action for a bit again! please use your best judgment and run fuzzing for a while, and merge when ready. ❤️

digitalmoksha · 2024-03-26T16:16:46Z

Oh I'm sorry to hear that. I hope you feel better.

Thank you for allowing me to move forward on this 🙇. I will definitely run fuzzing and take another look at the commented out tests.

kivikakk · 2024-03-27T07:19:41Z

Just fwiw, this branch did hit a crash on cargo fuzz run all_options -j4 within a minute or two, so it'd be worth going through a few fuzz–fix cycles!

kivikakk · 2024-03-27T07:24:49Z

Minimised repro of the one I hit first:

> printf '$`$' | RUST_BACKTRACE=1 cargo run -- -e math-dollars -e math-code
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `target/debug/comrak -e math-dollars -e math-code`
thread 'main' panicked at src/parser/inlines.rs:726:42:
slice index starts at 2 but ends at 1
stack backtrace:
   0: rust_begin_unwind
             at /rustc/2f1bd0729b74787f55d4cbc7818cfd787cd43a99/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/2f1bd0729b74787f55d4cbc7818cfd787cd43a99/library/core/src/panicking.rs:72:14
   2: core::slice::index::slice_index_order_fail_rt
             at /rustc/2f1bd0729b74787f55d4cbc7818cfd787cd43a99/library/core/src/slice/index.rs:98:5
   3: core::slice::index::slice_index_order_fail
             at /rustc/2f1bd0729b74787f55d4cbc7818cfd787cd43a99/library/core/src/slice/index.rs:91:14
   4: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::index
             at /rustc/2f1bd0729b74787f55d4cbc7818cfd787cd43a99/library/core/src/slice/index.rs:406:13
   5: core::slice::index::<impl core::ops::index::Index<I> for [T]>::index
             at /rustc/2f1bd0729b74787f55d4cbc7818cfd787cd43a99/library/core/src/slice/index.rs:18:9
   6: comrak::parser::inlines::Subject::handle_dollars
             at ./src/parser/inlines.rs:726:42
   7: comrak::parser::inlines::Subject::parse_inline
             at ./src/parser/inlines.rs:219:25
   8: comrak::parser::Parser::parse_inlines
             at ./src/parser/mod.rs:2014:15
   9: comrak::parser::Parser::process_inlines_node
             at ./src/parser/mod.rs:1994:17
  10: comrak::parser::Parser::process_inlines
             at ./src/parser/mod.rs:1988:9
  11: comrak::parser::Parser::finalize_document
             at ./src/parser/mod.rs:1827:9
  12: comrak::parser::Parser::finish
             at ./src/parser/mod.rs:1809:9
  13: comrak::parser::parse_document_with_broken_link_callback
             at ./src/parser/mod.rs:123:5
  14: comrak::parser::parse_document
             at ./src/parser/mod.rs:62:5
  15: comrak::main
             at ./src/main.rs:286:16
  16: core::ops::function::FnOnce::call_once
             at /rustc/2f1bd0729b74787f55d4cbc7818cfd787cd43a99/library/core/src/ops/function.rs:250:5

to match closer with other renderers, such as pandoc.

for math

Inline parsing of $$ handles the block style while providing better compatability with other parsers.

digitalmoksha · 2024-03-29T03:10:54Z

@kivikakk I've been running fuzzing for about 5 hours without any problems, and will probably let it run overnight. I feel this is ready to go.

Let me know if you feel like giving it another once-over before I merge it.

digitalmoksha · 2024-03-29T14:26:26Z

I've been running fuzzing for about 15 hours without problems.

I'm going to go ahead and merge this.

digitalmoksha · 2024-03-29T14:28:21Z

@kivikakk before you become completely unavailable with your move, would you have time to cut a new release? I'm particularly hoping to get the escaped chars option released.

kivikakk · 2024-03-29T20:30:12Z

Yes, of course! Will do so in the next day or two! Nice work on the release 🥳El 30 mar 2024, a la(s) 01:28, digitalMoksha ***@***.***> escribió: @kivikakk before you become completely unavailable with your move, would you have time to cut a new release? I'm particularly hoping to get the escaped chars option released. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

kivikakk · 2024-03-29T23:07:42Z

Release underway …

kivikakk · 2024-03-29T23:22:07Z

Done! https://crates.io/crates/comrak/0.22.0 / https://github.com/kivikakk/comrak/releases/tag/0.22.0

Thank you all. 😊

gjtorikian · 2024-03-30T21:52:24Z

BTW @kivikakk if it is at all helpful I can help cut releases too—I just don’t know how or whether I have the access 😅

kivikakk · 2024-03-30T22:21:19Z

@gjtorikian Thank you very much! Invite sent; bus factor halved! ✅

gjtorikian · 2024-03-31T18:47:39Z

Thanks for that! Do you just cargo publish locally or is there any automation for git tags and GitHub releases?

kivikakk · 2024-03-31T21:06:59Z

No automation, just RELEASE_CHECKLIST.md in the root! (It'd be lovely to automate but frankly it's never become worth doing.)

digitalmoksha · 2024-04-01T14:48:22Z

I updated the dingus to allow the enabling of math (and the enabling of most other options, though I'm still missing a couple)

gjtorikian · 2024-04-01T16:36:34Z

No automation, just RELEASE_CHECKLIST.md in the root! (It'd be lovely to automate but frankly it's never become worth doing.)

I am a lazy, lazy person, so I will see how much a robot can do for us. 😆

digitalmoksha mentioned this pull request Feb 18, 2024

Mathematics #23

Closed

digitalmoksha force-pushed the bw-add-math branch 2 times, most recently from 7606d03 to 2f6d586 Compare March 22, 2024 19:15

kivikakk reviewed Mar 26, 2024

View reviewed changes

src/nodes.rs Show resolved Hide resolved

src/parser/math.rs Outdated Show resolved Hide resolved

src/tests/math.rs Outdated Show resolved Hide resolved

kivikakk approved these changes Mar 26, 2024

View reviewed changes

digitalmoksha self-assigned this Mar 26, 2024

digitalmoksha and others added 9 commits March 28, 2024 14:16

Add support for dollar math and code math

f03d99b

Add extension options

ad0b6ed

Add additional tests

252aebd

More tests and refactoring

f3c42d7

Render spans for dollar math

182386d

to match closer with other renderers, such as pandoc.

Update README

cdb7cdc

cargo run --example update-readme

e9896f9

Fix edge case panic

9e6c9ae

Fix problem with cm conversion adding extra padding

5fa3106

for math

digitalmoksha force-pushed the bw-add-math branch from 2aab7b9 to 52a35cf Compare March 28, 2024 20:38

Removed support for redundant NodeMathBlock

003f777

Inline parsing of $$ handles the block style while providing better compatability with other parsers.

digitalmoksha force-pushed the bw-add-math branch from 52a35cf to 003f777 Compare March 28, 2024 20:40

Minor fix to math spec, change to .md extension

971bf34

digitalmoksha force-pushed the bw-add-math branch from f3a65dc to 971bf34 Compare March 28, 2024 22:27

digitalmoksha merged commit 05a2532 into kivikakk:main Mar 29, 2024
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add math support #366

Add math support #366

digitalmoksha commented Feb 18, 2024 •

edited

Loading

digitalmoksha commented Mar 23, 2024 •

edited

Loading

kivikakk commented Mar 23, 2024

digitalmoksha commented Mar 24, 2024

kivikakk left a comment

kivikakk left a comment

digitalmoksha commented Mar 26, 2024

kivikakk commented Mar 27, 2024

kivikakk commented Mar 27, 2024

digitalmoksha commented Mar 29, 2024

digitalmoksha commented Mar 29, 2024

digitalmoksha commented Mar 29, 2024

kivikakk commented Mar 29, 2024 via email

kivikakk commented Mar 29, 2024

kivikakk commented Mar 29, 2024

gjtorikian commented Mar 30, 2024

kivikakk commented Mar 30, 2024

gjtorikian commented Mar 31, 2024

kivikakk commented Mar 31, 2024

digitalmoksha commented Apr 1, 2024

gjtorikian commented Apr 1, 2024

Add math support #366

Add math support #366

Conversation

digitalmoksha commented Feb 18, 2024 • edited Loading

digitalmoksha commented Mar 23, 2024 • edited Loading

cargo run -- -e math-dollars enables dollar math.

cargo run -- -e math-code enables code/GitLab math.

kivikakk commented Mar 23, 2024

digitalmoksha commented Mar 24, 2024

kivikakk left a comment

Choose a reason for hiding this comment

kivikakk left a comment

Choose a reason for hiding this comment

digitalmoksha commented Mar 26, 2024

kivikakk commented Mar 27, 2024

kivikakk commented Mar 27, 2024

digitalmoksha commented Mar 29, 2024

digitalmoksha commented Mar 29, 2024

digitalmoksha commented Mar 29, 2024

kivikakk commented Mar 29, 2024 via email

kivikakk commented Mar 29, 2024

kivikakk commented Mar 29, 2024

gjtorikian commented Mar 30, 2024

kivikakk commented Mar 30, 2024

gjtorikian commented Mar 31, 2024

kivikakk commented Mar 31, 2024

digitalmoksha commented Apr 1, 2024

gjtorikian commented Apr 1, 2024

digitalmoksha commented Feb 18, 2024 •

edited

Loading

digitalmoksha commented Mar 23, 2024 •

edited

Loading

`cargo run -- -e math-dollars` enables dollar math.

`cargo run -- -e math-code` enables code/GitLab math.