Chatting with your documents is one of the genuinely good ideas of the AI era: instead of skimming a 40-page contract or a dense report, you ask it a question and get the answer. The catch is the failure mode. Ask a generic chatbot about your file and you will sometimes get an answer the document never said, delivered in exactly the same confident tone as the answers it got right.
This guide explains why that happens, what grounding and retrieval actually do about it, and how to set up a document chat workflow where every claim can be checked in seconds rather than taken on faith.
Why generic chatbots hallucinate on your files
A large language model doesn't look up facts; it predicts plausible text. That's fine for brainstorming and terrible for "what does clause 14.2 say", because a plausible-sounding clause and the actual clause are different things.
When you paste or upload a document into a general-purpose chatbot, three specific problems stack up:
- The model blends your document with its training data. It has read a million contracts, so when yours is ambiguous or the relevant passage didn't make it into context, it quietly answers from the average contract instead of your contract.
- Long documents get lossy. Even models with huge context windows attend unevenly across it. Details buried in the middle of a long file are the ones most likely to be paraphrased loosely or missed entirely, and files that exceed the window get silently truncated.
- There is no penalty for guessing. A generic chatbot is built to answer. If the document doesn't contain the answer, filling the gap with something plausible looks, to the model, like doing its job.
None of this means the model is broken. It means "paste the file into a chatbot" is the wrong architecture for questions where being wrong costs you something. We've written a longer piece on the mechanics in how source-grounding stops AI hallucination.
Grounding and retrieval, in plain English
The fix has an unglamorous name, retrieval-augmented generation, but the idea is simple:
- Split the document into passages so each chunk is small enough to be handled precisely.
- When you ask a question, retrieve the passages that are actually relevant to it, using meaning-based search rather than hoping the right paragraph happens to be in context.
- Make the model answer from those passages only, not from its general knowledge of the topic.
The result is called source-grounding: the answer is constructed from text that was really retrieved from your document at answer time. If the retrieved passages don't contain an answer, a well-built system says so instead of improvising.
Grounding doesn't make a model smarter. It narrows what the model is allowed to say, which is exactly what you want when the question is about your file rather than the world in general.
What "verified citations" actually means
Plenty of tools now print little citation markers, so it's worth being precise about what a citation has to do before it earns your trust:
- It must point at a real passage. A citation to "the document" is decoration. A citation to the exact sentence is evidence.
- You must be able to open it in one click. If verifying a claim means re-reading the section yourself, the citation hasn't saved you anything.
- Unsupported claims should be dropped, not decorated. The strongest systems check each claim against the retrieved passages before showing it, and remove anything that can't be matched. A confident sentence with no citation is the tell.
This is the standard we build to: every answer in Sidenote carries a citation you can click, the page scrolls to the exact source sentence and highlights it, and claims that can't be tied to a retrieved passage are dropped server-side before you ever see them.
Ungrounded vs grounded document chat
| Question | Generic chatbot | Grounded document chat |
|---|---|---|
| Where does the answer come from? | Training data plus whatever fit in context | Passages retrieved from your document at answer time |
| What happens when the answer isn't in the file? | A plausible guess, confidently phrased | "Not found in this document" (or the claim is dropped) |
| How do you verify a claim? | Re-read the document yourself | Click the citation, land on the exact sentence |
| Biggest risk | Fluent answers the document never said | Retrieval missing a relevant passage |
That last row matters: grounding is a large reduction in risk, not a guarantee. A grounded system can still misread a passage it retrieved. The difference is that its mistakes are visible, because every claim points at the text it came from.
A workflow for document chat you can trust
Whatever tool you use, the same four habits apply:
- Keep the document open next to the chat. Answers you can't see the source of are answers you can't audit. Reading the document in place, rather than uploading it into a silo, keeps verification one glance away.
- Ask specific questions. "What are the termination conditions?" retrieves better passages than "summarize this", and gives you claims that are easy to check.
- Click the citation on anything that matters. Not every answer needs auditing, but any number, date, obligation, or conclusion you'll act on does. If a claim has no citation, treat it as unverified.
- Treat "I couldn't find that" as a feature. A tool that admits the document doesn't say something is more useful than one that never does.
With Sidenote, this workflow runs on the page you already have open: a PDF, a Confluence or Notion page, a Google Doc, a SharePoint file, or any web page, read in place and read-only. Ask a question in the sidebar, get an answer built only from the document's text, and click any citation to jump to the exact sentence.
Frequently asked questions
Why does ChatGPT make things up about my PDF?
Because it's a general-purpose model doing its default job: producing plausible text. When your PDF is long, the relevant passage may not be in context at answer time, and the model fills the gap from training data without flagging that it did. Uploading the file helps but doesn't change the underlying behavior; only retrieval plus grounding constrains the answer to what the document actually says.
Do citations guarantee the answer is correct?
No. A citation guarantees the answer is checkable, which is different and, in practice, more valuable. A grounded system can still misinterpret a passage it retrieved, but because the claim points at its source sentence, you can catch the misreading in one click instead of discovering it weeks later.
Is chatting with documents free in Sidenote?
Chat is a Pro feature. The free plan gives you standard-length summaries, explanations, and glossaries, all with the same verified citations. To chat, you can start a 7-day Pro trial (opt-in, no card required) or subscribe. That split is deliberate: the free tier covers reading faster, and the trial lets you test whether cited chat earns its keep on your real documents.
Does grounded chat work on scanned PDFs?
Only once there's real text to ground in. A scanned PDF is a stack of images, so it needs OCR before any retrieval system can quote it. Sidenote runs OCR on scanned PDFs as a paid feature and then cites the recognized text like any other document; for a quick one-off conversion there are free tools, including our own PDF to text converter.