Strata

Canonical form

The exact rules by which a document becomes the bytes that get hashed. Version 10.

A hash of a .docx is a hash of a zip file: it changes when the software that wrote it changes, and says nothing about the writing. So the text is first reduced to a canonical form — everything about how a document looks is discarded, everything about what it says is kept — and the hash is taken over that.

The reduction is lossy on purpose. Margins, fonts, alignment, bullet glyphs and page size are all absent, which is why restyling a manuscript cannot change its identity.

The rules

  1. XML entities are decoded first. " becomes a quotation mark before anything else happens.
  2. Trailing whitespace is stripped per line. Including a trailing soft break: U+2028 is whitespace, so a line ending in one loses it. A Shift+Enter with nothing after it is padding rather than a break between two lines.
  3. A blank paragraph is a scene break. Not noise — one of the oldest conventions in fiction. A run of blanks is someone reaching the next page, which is spacing; blanks touching a heading belong to the template.
  4. NFC normalisation is applied last, here and never in the hashing layer.
  5. No trailing newline. Lines are joined, not terminated.
  6. A soft line break is U+2028, never \n, which separates paragraphs.
  7. Emphasis is markdown-style — *italic*, **bold**, ***both*** — with \* and \\ for literals. Emphasis is content: he said nothing and he said nothing are different sentences.
  8. An image is ![alt](sha256:…); the bytes live in the content-addressed store.
  9. A page break is U+000C alone on a line. The character Unicode already defines as exactly this.
  10. A figure is ![description](sha256:… "caption"). A description is announced to someone who cannot see the image and never printed; a caption is printed and read by everyone. Both are the author’s words, so both are hashed.
  11. A list item is - text or 1. text, indented two spaces per level. Structure only.
  12. Strikethrough is ~~struck~~ and underline is __underlined__, toggling like emphasis. A lone ~ or _ is ordinary text.

What a list keeps, and what it drops

That a paragraph is an item, how deeply it nests, and whether its list is ordered are content — an ordered list asserts that its items have a sequence, which is a claim about the writing.

Which glyph draws the bullet, and whether numbers run in arabic, roman or letters, is how the list is set. None of it is hashed. - and 1. encode ordered-ness in the same way #  encodes a heading level without asserting that headings are drawn with a hash.

The ordinal is derived from position rather than read from the file: the sequence is the content, the number is that sequence made visible, and a word processor stores no per-paragraph number to read anyway.

Until version 11 the importer recognised italic and bold and nothing else. A struck-through sentence in a Word document arrived as ordinary prose: the words survived, and the fact that the author had struck them did not. Nothing in the application said so. A draft where a deleted line comes back looking like a kept one is the exact failure this project exists to prevent, and it was happening on the way in.

Both marks are content by the same test as everything else here. “He said nothing” and “he said nothing” are different sentences. What is not content is which of Word’s eighteen underline styles was used — dotted, wavy, dash-long-heavy. That is how the page is set, and it is no more hashed than a bullet glyph is.

A lone ~ or _ stays ordinary text, and only the doubled form is a delimiter. It is the difference between a plain-text mirror you can read and one where snake_case and every file path is full of backslashes.

The version moved, so the question was whether anything already written would be read differently. Every canonical byte in every project, every frozen vector and all forty-one corpus tabs were scanned for a bare ~~ or __. There were none, so for everything that exists, version 11 and version 10 are the same function. The scan is kept as code rather than as a memory of an afternoon.

Format version and importer version

The format version describes a pure function: given paragraphs, produce canonical text. It is frozen, and a change to it is a change to what every historical hash means.

The importer version describes how faithfully a .docx is turned into those paragraphs. Bugs there are found for years — soft breaks, literal tabs, page breaks and list structure were each read wrongly at some point, and every fix changed the text a given file yields while no rule of the form moved.

One number cannot express both. Freeze extraction and its bugs become permanent, because correcting them changes hashes; version it with the format and every bug fix falsely announces a format change.

Currently: format v10, importer v3.

Conformance

A second implementation is conforming when it reproduces the published vectors: synthetic cases covering each rule, and forty-one drawn from real documents. Cases are tagged with the contract they pin, so a format case and an importer case are never confused.

The reference implementation is deliberately plain Python with no dependencies, so it can be read and checked rather than trusted. A second implementation in Rust exists precisely so the two can be diffed against each other on real manuscripts.

Provenance leaves follow the separate DAON wire format, which is normative and has its own vectors. This project reproduces those too.