Canonical form
The exact rules by which a document becomes the bytes that get hashed. Version 10.
A hash of a .docx is a hash of a zip file: it changes when
the software that wrote it changes, and says nothing about the writing. So the text is first
reduced to a canonical form — everything about how a document looks is
discarded, everything about what it says is kept — and the hash is taken over
that.
The reduction is lossy on purpose. Margins, fonts, alignment, bullet glyphs and page size are all absent, which is why restyling a manuscript cannot change its identity.
The rules
- XML entities are decoded first.
"becomes a quotation mark before anything else happens. - Trailing whitespace is stripped per line. Including a trailing soft break: U+2028 is whitespace, so a line ending in one loses it. A Shift+Enter with nothing after it is padding rather than a break between two lines.
- A blank paragraph is a scene break. Not noise — one of the oldest conventions in fiction. A run of blanks is someone reaching the next page, which is spacing; blanks touching a heading belong to the template.
- NFC normalisation is applied last, here and never in the hashing layer.
- No trailing newline. Lines are joined, not terminated.
- A soft line break is U+2028, never
\n, which separates paragraphs. - Emphasis is markdown-style —
*italic*,**bold**,***both***— with\*and\\for literals. Emphasis is content: he said nothing and he said nothing are different sentences. - An image is
; the bytes live in the content-addressed store. - A page break is U+000C alone on a line. The character Unicode already defines as exactly this.
- A figure is
. A description is announced to someone who cannot see the image and never printed; a caption is printed and read by everyone. Both are the author’s words, so both are hashed. - A list item is
- textor1. text, indented two spaces per level. Structure only.
What a list keeps, and what it drops
That a paragraph is an item, how deeply it nests, and whether its list is ordered are content — an ordered list asserts that its items have a sequence, which is a claim about the writing.
Which glyph draws the bullet, and whether numbers run in arabic, roman or letters, is
how the list is set. None of it is hashed. - and 1. encode
ordered-ness in the same way # encodes a heading level without asserting
that headings are drawn with a hash.
The ordinal is derived from position rather than read from the file: the sequence is the content, the number is that sequence made visible, and a word processor stores no per-paragraph number to read anyway.
Format version and importer version
The format version describes a pure function: given paragraphs, produce canonical text. It is frozen, and a change to it is a change to what every historical hash means.
The importer version describes how faithfully a .docx is turned into
those paragraphs. Bugs there are found for years — soft breaks, literal tabs, page
breaks and list structure were each read wrongly at some point, and every fix changed the
text a given file yields while no rule of the form moved.
One number cannot express both. Freeze extraction and its bugs become permanent, because correcting them changes hashes; version it with the format and every bug fix falsely announces a format change.
Currently: format v10, importer v3.
Conformance
A second implementation is conforming when it reproduces the published vectors: synthetic cases covering each rule, and forty-one drawn from real documents. Cases are tagged with the contract they pin, so a format case and an importer case are never confused.
The reference implementation is deliberately plain Python with no dependencies, so it can be read and checked rather than trusted. A second implementation in Rust exists precisely so the two can be diffed against each other on real manuscripts.
Provenance leaves follow the separate DAON wire format, which is normative and has its own vectors. This project reproduces those too.