Parser pipeline

The base parser exports to JSON objects, and does not run Markdown at all!

First pass - strings

Do things like fix the unindent, etc. Result should be a string that is pre-processed.

No line break processing happens here, because we need to split them up into scenes and fragments first!

Second pass - scenes

Separate scenes (since all content needs to be in a scene) into a scene object with a string interior. Updates the metadata of the scene object.

Creates a bunch of SceneObject:

[{
	type: 'scene',
	id: 'blah' // this is special, and is pulled out
	meta: {...}
	raw: "Blah blah blah",
},
...
]

Third pass - fragments

Further splits the content into fragments within a scene, and creates objects that are sub-sets of the scenes. Updates the metadata of the fragments objects.

Text that aren't in fragments are put instead into an empty fragment object, which should not be counted as a fragment.

[{
	type: 'scene',
	meta: {...},
	fragments: [
		{
			type: 'empty' // this is not a fragment!
			meta: {...}
			raw: "blah blah blah"
		},
		{
			type: 'fragment',
			meta: {...}
		}
	]
}]

Fourth pass - tokens

Performs all the string operations on the raw fragment strings, then splits them out.

Splits content into paragraphs per fragment, and creates text tokens out of them. Commands get command tokens, comments get comment tokens. End command tokens get their own as well.

These are slotted in as an array of objects into each fragment object.

[{
	type: 'scene',
	meta: {...},
	fragments: [
		{
			type: 'empty' // this is not a fragment!
			meta: {...}
			tokens: [
				{
					type: 'paragraph',
					text: 'blah',
					meta: 'raw meta',
					id: 'id'
				},
				{
					type: 'comment',
					text: 'blah'
				},
				{
					type: 'command',
					name: 'at',
					args: ['arguments']
				}
			]
		},
		{
			type: 'fragment',
			meta: {...}
		}
	]
}]

Token Types

{
	type: 'paragraph',
	text: <string>,
	meta?: <string>,
	id?: <string>,
	primary: <string>,
	classes: <string | string[]>
}

{
	type: 'comment',
	text: 'blah'
}

{
	type: 'command',
	name: <string>
	args?: <string[]>
	open?: <bool> // if true, we're looking for an `endCommand` token later on.
}

{
	type: 'endCommand'
	name?: <string> // if we can determine what kind of command it's supposed to end
}

Fifth pass - flattening

Takes the nested scene / fragment object, and creates start / end scene and start / end fragment objects out of them.

Flattening can be an extra step in the process, after all else is done, because sometimes we might want fragments and scenes nested.