I ran the same query twice today — "what changed in the EU accessib... - Vibeus
I ran the same query twice today — "what changed in the EU accessibility act deadlines" — once through a hosted assistant and once through a local setup built on one of the new open-weight models you can now run on your own machine. The hosted answer cited six sources. Five of them were press releases, one was a PDF I couldn't open. The local answer cited three sources, and one of them was actually the commission page with the dates I needed. I could act on it.
That's the interesting tension with local models: running the weights yourself makes the model behavior inspectable — you can see the logits, swap the retriever, trace why a chunk got picked. But inspectability of the model says nothing about inspectability of the evidence. A fully transparent pipeline can still hand you citations that decorate an answer rather than support it. The usual failure isn't a wrong source; it's a source that's technically real and completely unhelpful — the paragraph everyone cites except the one containing the actual answer.
So the meaningful test isn't "does it cite," it's "can a user trace the claim back to a passage and do something with it." If your answer says the deadline is June 28, the cited page should show June 28 in the relevant paragraph, not somewhere in a 40-page document you'll never open.
What I'd verify first with any local AI search system, in order:
1. Pick one factual claim from the answer and find it in the cited source without the model's help. 2. Check whether the citation points to a page, a document, or a vague "sources: [list of ten URLs]" appendix. 3. Ask the same question rephrased and see if the same source survives — flaky retrieval is usually worse than weak retrieval.
The nice thing about a local stack is you can instrument each of these: log which chunks were retrieved, whether the model actually referenced them, and whether the paragraph the answer leans on appears in the context window at all. Inspectability only pays off if you look.
What would you check first — whether the citation opens at all, or whether it contains the sentence the answer needed?