Almost any AI cataloging tool looks good in a demo. It takes a document and in seconds returns a complete record, with the fields filled and well formed. But that first impression does not answer the decisive question for an institution with a collection: is that record reliable inside your catalog, where a wrong datum stays invisible and permanent? The questions that follow are the ones that settle it, and they work for evaluating any tool, ours included.
Where does each datum come from? An AI model does something more than read the document: it fills fields the document never carried, from what it already knows. That is useful and at the same time risky, because a datum that looks correct can be false. Ask whether the tool distinguishes what it verified from what it inferred, and whether it flags it. In a good answer, the names and subjects that were validated against a source come out with that source noted in the record, and what comes only from the model stays without that mark, so you know what is worth reviewing.
Against which authorities does it verify, and do they work with your systems? Authority control is what keeps one and the same author from entering the catalog with three different spellings and ending up as three people. Ask against which sources it checks names and subjects, from universal vocabularies to your own authority catalog. And if your material cannot leave the institution, ask whether that verification also works without an outside connection, against your systems.
Does it compute what has a rule, or guess it? Some fields have a correct answer, not an estimate: the call number, certain subfields, the archival level of description. Ask whether the tool computes them following the rule or asks the model for them. When a public rule exists, approximating is not enough.
Is the record correct against the standard, not just in form? A standard like MARC21, ISAD-G or CDWA is not just a field structure: it carries rules about provenance, hierarchy, field repetition and vocabulary. Ask for a real record in your standard and check it against the standard, not against its appearance. A record can have the correct MARC form and still be wrong.
Does it evaluate and flag each record? At scale nobody reviews record by record. Ask whether the tool scores each one and flags the ones that do not reach a minimum, so review concentrates on what is signaled. Without that, you receive a full batch without knowing which ones to look at.
Does it fit your collection, or is it one size fits all? The same document, under the same standard, is cataloged differently at each institution, according to its classification scheme, its authorities and the language it describes in. Ask to see it configured for your scheme, not a generic demo. If it cannot be adjusted to your collection, it will hardly be correct for you.
If your material cannot leave, does it process on your infrastructure? Some collections cannot leave the institution: sensitive or regulated files, and also, in a library, theses with restricted intellectual property. For those cases, ask whether the tool runs on your own servers, even on a network isolated from the internet, and what information leaves in each mode of operation: whether everything is processed locally, or whether something is sent out already anonymized.
Does it tell you where it fails? A vendor who promises never to be wrong is hiding the risk, not eliminating it. The serious answer explains how it detects and flags its errors: what the quality of the result depends on, above all the quality of the source material, in which cases it performs worse and how it warns you. A tool acknowledging its limits says more about its reliability than any figure.
If you are evaluating AI cataloging tools and want to test these questions against your collection and your standard, write to us at info@janium.com and we will tell you where Collect fits and where it does not.