Citation verification record

A per-citation record of how each source behind the agreement and accuracy study was checked, kept because a reference is a factual claim about a document somebody else wrote, and a plausible-looking citation is not evidence that the document says what it is cited for.

The distinction it enforces

Existence and content are separate checks, and the record grades them separately. Existence verified means the identifier resolves and the title, authors and date match. Content verified means the primary record was read and it states the specific thing being relied on. A third level covers the case where an abstract supports the claim but a narrower sub-claim about the body was not read, and a fourth marks sources whose bodies were never obtained, where any content claim rests on a secondary report and is flagged as such in the text.

Citing a real document for a claim it does not make is the more common failure, and it is invisible unless the two checks are kept apart.

What it also records

Claims that were discarded, because a rejected claim is part of the audit trail. Three cases are logged: a quotation attributed to a survey that is not in that survey’s abstract, which is used nowhere; an absence claim about the same survey’s reference list, which was downgraded because the reference list was not read; and a set of identifiers whose existence was confirmed but whose content was not, which are relied on for nothing.

One paywalled source is recorded as existence-only, with an accompanying note that the specific result it might have supplied was derived independently and proved in the repository instead.

Where to read it

evaltrust at v0.2, documents CITATIONS.md and writeup/REFERENCES.md. The latter carries a correction to the source brief, which had bundled two distinct papers under one identifier.