You ask for a source and get a perfectly formatted citation to a paper that does not exist. This is called hallucination, and it follows directly from how language models work.

Why it happens

• The goal is plausible text, not true text. A model predicts what is likely to come next. A realistic-looking citation is very “likely text”, whether or not it is real.
• Facts are stored fuzzily. Knowledge is spread across billions of numbers, not kept in a database. Rare facts (a small company’s founding year, an obscure SAP table) are recalled worst.
• Models are trained to be helpful. An answer usually scores better than “I don’t know”, so guessing can be rewarded.
• Tokens hide details. Letters, digits and exact spellings live inside tokens, so precise counting and long arithmetic are weak spots.

Six habits that catch it

1. Ask for sources, then open them. A link that 404s or says something else is a red flag.
2. Test formulas and code on a few rows where you know the answer before trusting them on 10,000.
3. Give it the document instead of asking from memory — answers grounded in text you provide (RAG) are far more reliable.
4. Allow “I don’t know”: If you are not sure, say so. It genuinely reduces invented answers.
5. Ask twice, differently. If two phrasings give different facts, neither is safe.
6. Be extra careful with numbers, names, dates, law, medicine and money.