Title: Demonstration of jmarkdown Date: 24 January 2025 Author: J. McKenzie Alexander Address: Department of Philosophy, Logic and Scientific Method
London School of Economics and Political Science
London, United Kingdom
WC2A 2AE Biblify defer: false Biblify website: Biblify CSS: test.css CSS: test2.css Script: test.js Body classes: class1 Body classes: class2 Highlight theme: atom-one-light script: <script> Custom element: that-container

The start of my container

The end of my container

Load extensions: reverseText from my-extensions.js Optionals: secret[false] top-secret[false] Optionals: answer[false] HTML header: HTML header: HTML footer: HTML footer: Extension 1: [: :] [true,false,true] 3 ${content3} ${content1} ${content2} Extension 2: @mark( ) true ${content1} Extension 3: "" "" ${content1} Extension 4: @explain( ) [false,true] 2 ${content1} gets interpreted as ${content2} Extension 5: ⌜ ⌝ false ${content1} homepage: https://jmckalex.org/home/ Extension 6: ↑ ↑ false ${content1} Extension 7: /[a-z]+-->/ /([a-z]+)-->([a-z]+)/ false 2 \(${content1}\rightarrow ${content2}\) Extension 8: @dothis( ) true Here is the content: ${content1} Extension 9: /x[0-9]+/ /x([0-9]+)/ false 1 $x_{${content1}}$ Extension linebreak: /↵/ /(↵)/ false 1
Extension Q1: /Q1\./ /(Q1\.)/ false 1

This is the text of the first question.

Extension Q2: /Q2\./ /(Q2\.)/ false 1

This is the text of the second question.

Extension mark: /\[\d\d\]/ /\[(\d\d)\]/ false 1 Grade ${content1} Extension candidate: /Candidate:\s*[0-9]+/ /Candidate:\s*([0-9]+)/ false 1 Candidate: ${content1}
Extension subscripts: /[pxyz][0-9k]+/ /^([pxyz])([0-9k]+)/ false 2 \(${content1}_{${content2}}\) Extension tuples: /\(\s*x([0-9a-z])\s*,\s*y([0-9a-z])\s*,\s*z([0-9a-z])\)/ /\(\s*x([0-9a-z])\s*,\s*y([0-9a-z])\s*,\s*z([0-9a-z])\)/ false 3 \((x_${content1}, y_${content2}, z_${content3})\) ------------------------------------ :div[*JMarkdown: JavaScript empowered markdown with dynamic syntax extensions*]{.big .left} Created by [J. McKenzie Alexander](mailto:jalex@lse.ac.uk)
:today >> “Let’s go crazy” {.class1 .class2} ↵ >> /* — Prince*/ `jmarkdown` is a full-featured variant of markdown which provides the ability to define new syntax extensions /in the markdown file itself/. The extensions can be /simple/ — where you specify only the starting- and ending- delimiters — or /complex/ — where you specify a regular expression, possibly with groups. The text identified by the extension can then be manipulated, processed by the markdown interpreter, or inserted into arbitrary HTML. In addition, `jmarkdown` supports the evaluation of JavaScript included in the markdown file /at the time the markdown code is processed/. The JavaScript code can be used to define syntax extensions or to generate HTML output, or to use Cheerio as a post-processor to manipulate the HTML output generated by `jmarkdown`. The syntax extensions can take advantage of the full set of Unicode characters, allowing you to define intuitive markup for certain effects. For example, this markdown file defines extensions using Unicode delimiters so that `↑Small caps↑` becomes ↑Small caps↑
`⌜strings like in the TeXBook⌝ ` becomes ⌜strings like in the TeXBook⌝ Extensions can also be specified using regular expressions, allowing you to configure things so that `Consider points p1, p2, p3,... where pk is (xk, yk, zk)` becomes consider points p1, p2, p3,... where pk is (xk, yk, zk) As for embedding JavaScript in the markdown, here’s a fun example. With a suitable JavaScript function defined: `rainbow("It's time to be awesome!")` becomes rainbow("It's time to be awesome!") `rainbow("And options can be specified!", {duration: 10})` becomes rainbow("And options can be specified!", {duration: 10}) ```md rainbow("And the colours can vary, too!", { duration: 3, delay: 0.08, colors: ['#000000', '#666666', '#ffffff', '#666666', '#000000'] }) ``` rainbow("And the colours can vary, too!", { duration: 3, delay: 0.08, colors: ['#000000', '#666666', '#ffffff', '#666666', '#000000'] }) [View the `jmarkdown` source which generated this page!](jmarkdown.md) [Download the source from GitHub!](https://github.com/jmckalex/jmarkdown) *Table of Contents* {{TOC}} # Motivation This project started out small, and then grew in its ambition. The beginning was this: I’ve always found the default syntax of markdown to be a little annoying. Why aren't italics defined `/like this/` rather than `*like this*`? It also seemed like boldface was better indicated `*like this*` rather than `**like this**`. And, finally, shouldn’t underlining be indicated `__like this__`? That just seems more… /rational/, you know. From that initial goal — to create a custom markdown interpreter which catered to my own peculiar syntax preferences — this project has since grown into a full-featured markdown interpreter which allows you to define /custom syntax extensions/ in the markdown file itself! :fa-champagne-glasses: There’s also a lot more. Anyway, `jmarkdown` uses [marked.js](https://marked.js.org/) to adjust and extend the syntax of markdown in a number of ways. It also preloads several additional extensions from the marked.js extension library to make it behave more like multimarkdown, with some additional features multimarkdown doesn’t have. Here’s a list, in no particular order, of all the features `jmarkdown` offers: * Better syntax for italics, boldface, and underline as per above * Better syntax for inline subscripts and superscripts, so that `H_2O` yields H_2O and `E=mc^2` yields E=mc^2. * A syntax extension for right-aligned and center-aligned text. * A syntax extension for adding classes and an id to the parent element. * A syntax extension for inline comments. This defaults to the `%` character, as in $\TeX$, but it can be changed. * A syntax extension for description lists. * Emojis (all [octocat supported](https://gist.github.com/rxaviers/7360908) emojis) :beers: :smile: I :heart: marked! :tada: * [FontAwesome](fontawesome.com) icons :fa-thumbs-up: * GFM tables * GFM alerts * Footnotes[^a] via `marked-footnotes` * [Directives syntax](https://talk.commonmark.org/t/generic-directives-plugins-syntax/444). If you don’t know what this is, it’s an incredibly useful extension. I use this to provide: * The ability to comment out parts of the markdown file, i.e., multiline comments providing the functionality of C's `/*...*/` syntax. * The ability to define “optionals” for including/excluding portions of the file (e.g., answers to a quiz) * Support for math via MathJax * Support for Mermaid. * Support for alphabetical and roman numeral lists via `marked-more-lists`. * Code highlighting using `highlight.js` via `marked-highlight`. * In-browser bibliographies generated via [Biblify](https://jmckalex.org/software/bibtex-in-webpages.html). These are handled via JavaScript and are processed entirely within the browser, so there is no need to use `pandoc` as part of the workflow. * Auto-generated table of contents. * Typesetting strategic-form games. * A useful environment for showing markdown code in its raw form, and processed form, side-by-side. * A metadata header whose content is used for configuring jmarkdown and also for defining variables/parameters to be inserted into the document by a pre-processor. Given the header for this file, by simply typing this `{{homepage}}` I get this: {{homepage}} * The ability to define new markdown syntax, simply, in the metadata header. * The ability to insert arbitrary JavaScript code in ` ``` :::: :::::source{target='container5'} ::::markdown-demo :::that-container Some text /inside/ a *container* that has $\alpha$ math, too. ::: :::: ::::: :::source{target='bar'} >> *The file `definitions.js`* << ```js export const foobar = { level: 'container', marker: ":::", label: "foobar", renderer(token) { if (token.meta.name === "foobar") { return `FOOBAR`; } return false; } }; export const barfoo = { level: 'container', marker: ":::", label: "barfoo", renderer(token) { if (token.meta.name === "barfoo") { return `BARFOO`; } return false; } }; export default [ foobar, barfoo ]; ``` ::: :::source{target='foo'} >> *The file `my-extensions.js`* << ```js export const jmckalex = { name: 'jmckalex', level: 'inline', start(src) { return src.match(/jmckalex/)?.index }, tokenizer(src) { const match = src.match(/^jmckalex/); if (match) { console.log(match); const token = { type: 'jmckalex', raw: match[0], mermaid: true, text: match[0], tokens: [] }; return token; } }, renderer(token) { return `J. McKenzie Alexander`; } }; ``` ::: That’s it for /most/ of the special keys in the metadata header. There is one more type which requires explanations of greater length. More on that in a moment. What can you do with other definable keys in the metadata header? `jmarkdown` defines a mustache-like syntax where the expression `{{variable name}}` gets replaced by whatever the corresponding value is from the metadata header, if it's defined. This provides a very simple form of variable interpolation. So, for example, if the header contained: ```yaml Address: Department of Philosophy, Logic and Scientific Method
London School of Economics and Political Science
London, United Kingdom
WC2A 2AE ``` Then you can use that as follows: :::markdown-demo You can contact me at:
{{Address}} ::: If the same key has been used multiple times, then `{{key}}` will return all of the values concatenated into a single string. ## Defining new markdown syntax (simple) In the metadata header, you can define new markdown syntax by simply specifying the starting delimiter, the ending delimiter, and the replacement text as HTML. (With an optional argument, you can also indicated whether the contents of the new delimiter should be parsed as markdown text.) Anything between the delimiters is then inserted into the defined replacement text every place that `${content}` appears. Here are three examples. Suppose the metadata header includes the following: ```yaml Extension 1: [: :] [true,false,true] 3 ${content3} ${content1} ${content2} Extension 2: @mark( ) ${content1} Extension 3: ⌜ ⌝ false ${content1} Extension 4: ↑ ↑ false ${content1} ``` Each new defined extension must begin with the keyword `Extension` followed by a space and then a following unique identifier, which could be a short sequence of words naming the extension. In the simple case, we specify the opening and closing delimiters. These can be any normal text characters, in sequence, provided that there are no spaces in the delimiters. Since `jmarkdown` supports unicode, you have a large set of symbols you can use which provide good syntactic clues as to what the extension does. The third argument is optional, but if it is `true` then the contents surrounded by the delimiter will be parsed as markdown input, otherwise it will be assumed to be `false` and the contents surrounded will not be processed as markdown input. There is an optional fourth argument. If it is an integer, it specifies how many /arguments/ are given to the extension. The arguments are comma delimited. If you need to specify an argument that contains a comma, you can give it as a "string", where quotes can be included using the normal escape syntax of \\". You refer to the arguments in the replacement text via `${content1}` or `${content2}` or so on, where the integer refers to the number of the argument. If more than one argument is specified, then the third argument can take a special form. Instead of just /true/ or /false/, it can be an array of values indicating whether the i^{th} argument should be pushed through the markdown processor. If the i^{th} entry in the array is true, that argument will be processed as markdown text. If the i^{th} entry is false, that argument will be treated as verbatim text. The first defined extension converts `[:First *argument*, /second/ argument, __third__ argument:]` into [:First *argument*, /second/ argument, __third__ argument:] (The spaces between the `` elements appears because the definition text contains a tab before each span.) The second defined extension converts `@mark(68)` into @mark(68) The third defined extension allows ""⌜⌝"" to serve as delimiters for verbatim text, allowing me to type arbitary markdown code. But I also enclose the code in a `` element, and then target it with a little bit of JavaScript, allowing the verbatim code to be typeset like in Knuth’s TeXBook, where spaces are indicated with little square caps. The result? `⌜Text *like* /this/⌝` ends up transformed into ⌜Text *like* /this/⌝. The fourth extension defines `↑ ↑` to function as delimiters which convert the enclosed text to small caps, so `↑Small caps↑` becomes ↑Small caps↑. ## Defining new markdown syntax (complex) Syntax extensions can also be defined via regular expressions. To do this, you need to specify two regular expressions: the first should be a quick and easy regular expression which indicates the start of the new syntax element, and the second is the full regular expression to match the entire construct. (Why two? Because that's how `marked.js` defines syntax extensions.) Here are three examples, taken from my own personal use which are discussed in more detail below: ```yaml Extension mark: /\[\d\d\]/ /\[(\d\d)\]/ false 1 Grade ${content1} Extension candidate: /Candidate:\s*[0-9]+/ /Candidate:\s*([0-9]+)/ false 1 Candidate: ${content1}
Extension subscripts: /[pxyz][0-9]+/ /^([pxyz])([0-9]+)/ false 2 \(${content1}_{${content2}}\) ``` The first extension matches any two digits surrounded by brackets. It then takes the numeric mark and wraps it in some HTML which causes it to float to the right, so that `[59]` becomes the construct you see to the right.[59] Why do I have this defined? Because when I mark student essays, I write the feedback in markdown, and it is nice to be able to simply include the mark using that syntax and have it formatted in a more readable way. Moreover, by automatically adding a class to the second inner span, we can do automated data extraction during the post-processing phase, as we’ll see later. The second extension matches the expression `Candidate:` followed by any amount of optional space, and then a sequence of digits. This then gets formatted as follows: :::markdown-demo Candidate: 123456 ::: Again, why do this? If you look, you'll see that a class has been added to the `strong` element, which means we can target that element during the post-processing phase. The third expression is more interesting. On old internet math groups, it used to be the case that people would write `x1, x2, x3,...` to refer to subscripted variables. The use of regular expression in defining this extensions means that you can use that syntax directly in your markdown documents. :::markdown-demo Consider points p1, p2, p3 in three-dimensional space, where the respective components of p1 are denoted by x1, y1, and z1 respectively. ::: In short, using regular expressions to define syntax extensions means that /patterned data/ can be automatically recognised by `jmarkdown` and formatted into something more aesthetically pleasing. In addition, classes can be added to the HTML markup so that the generated document has suitable structure for being able to automate certain tasks. However, automating certain tasks requires using JavaScript, so let me now explain why `jmarkdown` as a `j` in its name! # Scripting capabilities One additional feature of `jmarkdown` is the ability to define JavaScript functions which can be embedded in the markdown code and invoked on pieces of markdown to perform additional transformations. The JavaScript functions can either return raw HTML, to be inserted into the output, or markdown code, to be processed in either inline- or block- mode. ## Inline scripts (text generation and manipulation) Here's a simple example (we’ll see more complicated examples below). Suppose the markdown code contains the following: ```html ``` The `data-type='jmarkdown'` tells `jmarkdown` that the script code should actually be processed rather than simply passed through quietly to the output HTML file. What `jmarkdown` does behind-the-scenes is to extract the code contained in the {{script}} tag and evaluate it in a Node virtual machine. The context of that virtual machine is shared across all jmarkdown script environments, so variables and functions defined in one script block are accessible in all the others. Normally, the script block containing the jmarkdown code is simply removed from the output. (We will later see how to put arbitrary HTML code in place of the script block.) The `export_to_markdown()` function takes, as its first argument, the name of the function to “export” to the `jmarkdown` interpreter. What happens when a function is exported is the following: 1. A new inline extension is created and registered with the markdown interpreter. By default, the extension scans the markdown code looking for text of the form `functionName(...text not containing a left parenthesis...)`. So, in this case, the markdown interpreter will look for text of the form `reverser(...)`. 2. When such a construct is found, the defined function will be evaluated with the text enclosed in parentheses passed as a string. So, in this example, jmarkdown code `reverser(Text to reverse)` will be evaluated in the Node virtual machine as `reverser("Text to reverse")`. Notice that in the simple, default use case that you don’t need to quote the argument. 3. If the function returns a string, that string will be inserted into the HTML output replacing the the original text, as shown below. You can basically think of this as though you’ve embedded a JavaScript function in markdown source code, where the return value of the function replaces the function call. :::markdown-demo{type=html} And here is how it is used: reverser(Text to reverse). ::: What happens if you attempt to include markdown text in the argument to the function? Unless you specifically request that the text should be interpreted, it will simply be passed through as uninterpreted text. For example: :::markdown-demo reverser(Text /containing/ *markdown* code.) ::: In order to request that the text be interpreted, you need to pass a second argument to `export_to_jmarkdown()`, telling it what to do. For example: :::markdown-demo{type=html} reverser2(Text /containing/ *markdown* code.). ::: The default way of handling embedded JavaScript functions works as long as the function arguments aren’t too complicated. You could, for example, interpret the string passed to the function as containing multiple arguments, as follows: :::markdown-demo{type=html} repeater(I :heart: /Jmarkdown/
, 3) ::: Since everything between the opening and closing parentheses is interpreted as a single string, this means that you have the freedom to create your own syntax for passing data to a function. For example, here’s a function which creates a button with an attached function using something akin to javascript’s `=>` notation: :::markdown-demo{type=html} Press here for a secret message: button(Click me => hello) ::: However, if you need nested function calls, that won’t work because of the requirement that the argument text cannot contain a `)`. If you need more complicated arguments, you need to tell `export_to_jmarkdown` to not assume that the argument structure is simple. When you export a function in this way, you are promising `jmarkdown` that you will properly handled quoted strings, etc., so that the tokenizer can just scan the source text looking for balanced parentheses `(...)` where the start and end paretheses do not appear in strings. This allows us to handle situations like the following: :::markdown-demo{type=html} Original string: Hello world!
ROT-13 encoded: rot13("Hello world!")
ROT-13 twice: rot13(rot13("Hello world!")) ::: Complex functions should only return HTML to be inserted as replacement text, because attempted anything too complicated while requesting the output be parsed can confuse the markdown interpreter. If you /really/ need to, though, you can use `marked.parse()` as follows: :::markdown-demo{type=html} colorize("lightblue") ::: The `marked` object accessible in `jmarkdown` scripts is a separate markdown processor configured exactly the same as the processor which is interpreting the `jmarkdown` file. However, because it is a separate processor that means that if you attempt to do anything with footnotes, headings, and the like, the results probably won’t be what you expect because those things will be processed by a separate markdown processor whose state isn’t connected to the one processing the file. What if you want to replace the jmarkdown {{script}} tag with some HTML, in addition to evaluating all the JavaScript inside it? A global variable named `output` is defined which allows you to specify the html which should be inserted in place of the {{script}} element. The value of `output` is set to the empty string before each jmarkdown script element is processed. Here is an example which also shows how variables and function definitions declared in one script element are available to another: :::markdown-demo{type=html} ::: Here’s a fun example showing what you can do with this. Here’s a function which, when given the name of an image file, embeds a base64 encoded version of the image into the HTML output from `jmarkdown`. ```html ``` :::markdown-demo Here is a paragraph containing some math $\alpha+\beta$ and an embedded image: embedImage('./nature.jpg', "width: 100px;") More text after it. ::: And what about the special command used at the start of this page, which generated a rainbow effect for text? That was defined by including the following code in the markdown file: ```html ``` And then we have the following: :::markdown-demo rainbow("And away we go!", { duration: 3, delay: 0.08, colors: ['#FF0000', '#00FF00', '#0000FF', '#FF0000'] }) rainbow("Here's a very slow version...", { duration: 20, colors: ['#000000', '#666666', '#ffffff', '#666666', '#000000'] }) ::: ## Post-processing scripts In addition, `jmarkdown` supports the inclusion of JavaScript which should be invoked once the final document has been assembled. In this case, once `jmarkdown` has finished constructing the HTML for the entire document, it will parse the document using [cheerio](https://cheerio.js.org/), which provides jQuery-like functionality for manipulating the DOM. (Note that cheerio provides a lightweight DOM implementation and only supports a subset of the jQuery syntax.) The cheerio object can be accessed via `$`, and access to the filesystem (using the `fs` module provided by Node) is via the variable `fs`. The HTML generated by `jmarkdown` is already loaded into cheerio. If you want to inspect the raw HTML source, that is available in the global variable `html`. If you make changes to the DOM that you want to be reflected in the HTML file exported by `jmarkdown`, you will need set `html` to the new content. More than one `jmarkdown-postprocess` script tag can be included in the markdown file: `jmarkdown` will simply collect the contents of each of those script tags and will evaluate them in order. For example, the following script will open a file named `output.txt` in the same directory as the markdown file being processed, then it will convert each of the `

` headings to upper-case, writing each new heading to `output.txt`. Once that’s done, the HTML contents to be exported by `jmarkdown` are set to the new state of the DOM and the file `output.txt` is closed. ```html ``` Because all of the markdown-magic has already been performed by the post-processing state, the above changes will not, of course, be reflected in the table of contents. While post-processing scripts don’t do anything which couldn’t already be done by including normal {{script}} tags to be evaluated in a browser, there are several things which are nice about them: 1. Post-processing scripts can permanently change the HTML source which is exported, whereas a script tag evaluated in the browser will leave the original source state visible to anyone who checks. 2. A post-processing script has access to the file system, so it becomes possible to extract information from the document and create auxiliary files based on the structured content. ## A detailed example I use `jmarkdown` for writing feedback on exam questions and storing the marks. The markdown file is structured as follows (ignoring the metadata header), with each student’s feedback and grade noted. ::::markdown-demo Candidate: 43786 File: ⌜student exam file name.pdf⌝ :::feedback Q1. ...text containing feedback on question 1... [68] Q2. ...text containing feedback on question 2... [75] ::: :::: In the metadata header, I've written several extensions which use regular expressions to do pattern matching on the candidate number, file name, question indicators, and the mark on each question. For example, here are the extensions which format the candidate number, the question text, and the mark for each question: ```yaml extension candidate: /Candidate:\s*[0-9]+/ /Candidate:\s*([0-9]+)/ false 1 Candidate: ${content1}
extension Q1: /Q1\./ /(Q1\.)/ false 1

This is the text of the first question.

extension Q2: /Q2\./ /(Q2\.)/ false 1

This is the text of the second question.

extension mark: /\[\d\d\]/ /\[(\d\d)\]/ false 1 ${content1} ``` I’ve defined the `feedback` container directive to add text and wrap the content in a `