← Blog

The same document twice: duplicates by signature and by text

Janium Collect can recognize that a document is already cataloged even when it arrives under another name, from another folder or scanned again, and it does so with two checks: the file’s signature for the exact copy and the similarity of the text for the second scan. Both are turned on per institution, and by default they only warn: the copy is cataloged and the batch’s quality report says which record it is a copy of.

The exact copy, by its signature

Before sending a file to the model, Collect calculates its SHA-256 signature and looks for it among the records of the same institution. An identical PDF has the same signature whether it is named differently or sits in another folder, so the copy is recognized without reading it. In one document collection, that check found 68 pairs of the same PDF stored in two different folders, which until then had been cataloged twice. This check is separate from the one the channels that receive ready-made PDFs already make, where a PDF identical to one already delivered is set aside in a duplicates folder and stays out of the batch, as described in Where material enters Collect.

The second scan, by its text

Two scans of the same document, or the signed original and its printed edition, are different files with almost the same text. For those cases, Collect calculates a fingerprint of the extracted text from its five-word phrases and compares it with those of the other records of the institution. Two records are flagged when the phrases they share reach 85% of all the phrases in the two texts, a threshold the institution can adjust.

The measure compares the two complete texts rather than checking whether one is contained in the other. With the second approach, an official gazette came out as similar to each of the regulations it publishes, and in a single institution it produced 253 false pairs. Measured across four institutions, the full comparison found 40 pairs, all of them of the same title.

What the institution receives

The copy is cataloged and goes out in its batch like any other record. The batch carries a list of duplicates with the original record, the reason (signature or text) and, when it is by text, the degree of similarity, and the quality report adds the duplicate warning to the copy without changing its score. The weekly report adds up the number of copies for the week.

Skipping the copy, that is, neither cataloging nor delivering it, is an explicit option the institution chooses. Records cataloged before the check was turned on can also be signed and measured, so the comparison covers what was already in the catalog.

Limits of the comparison

Short text. In a library, the text read from a book is usually the cover and the copyright page, and two volumes of the same series look as alike as two copies of the same book. For that reason the comparison by text requires a minimum of 300 phrases in each document, and a book with less text is compared only by its signature.

The exact signature. It is enough for a PDF to be saved again or cropped for its signature to change. That copy is no longer recognized by signature, and only the comparison by text detects it, if it has enough text.

The decision. The comparison flags the copy and its original record, and whoever knows the collection decides whether they are the same document, two versions worth keeping or a false resemblance.

To continue the conversation

If you suspect your collection has documents repeated across folders or scanned more than once, write to us at info@janium.com. We would like to know how your folders are organized and whether you prefer to see the duplicates in the report or leave them out of the batch.