Asking questions of your own notes: how AI answers with sources work, and where they fail

You type "what did we decide about the pricing test?" into your notes app and get a tidy paragraph back, with a little [1] after the main claim. It feels like the notes understood you. They didn't, quite. Something searched them, picked a few pages, and handed those pages to a language model with the instruction to answer from them.

That design has a name, retrieval-augmented generation, and it is how most "ask your notes" features work, including ours. Knowing the two steps helps you trust the right answers and catch the wrong ones.

Step one: find the pages

A language model can't read your whole workspace for every question. Even with large context windows, sending thousands of pages per question would be slow and expensive. So the system first picks a small set of pages that probably contain the answer. This is retrieval, and it decides more about answer quality than the model does.

There are two common ways to do it.

Keyword search. The system pulls the meaningful words out of your question and looks for pages that contain them. Pages that match more words, or match in the title, rank higher. It is fast, cheap and predictable: if a page contains the words, it will be found.

Semantic search. Every page, or every chunk of a page, is turned into an embedding, a list of numbers that captures roughly what the text is about. Your question gets an embedding too, and the system returns the chunks whose numbers are closest. This finds pages that use different words for the same idea, like "invoice" when you asked about "bills". It is also harder to predict, because "close in meaning" is fuzzy.

Many systems mix both. Whatever the method, the output is the same: a short list of pages or excerpts that go to the next step.

Step two: write the answer

The model gets your question, the retrieved excerpts, and instructions along the lines of "answer only from these sources and cite each claim by number". It writes a paragraph, adds the citation numbers, and the app turns those numbers into links.

Citations are the part that makes this useful. An answer without sources asks you to trust it. An answer with sources lets you check it in one click. That is a big difference when the answer goes into a client email.

How Prompt Notebook does it

Here is exactly what happens when you use Ask your workspace in Prompt Notebook, because vague descriptions of AI features help nobody.

1. Your question goes to our server. The server pulls up to eight meaningful words out of it and skips common words like "the" and "what".
2. It runs a keyword search over the titles and text of the pages you're allowed to read. A match in the title counts double. The eight best pages go forward.
3. Claude, by Anthropic, gets your question plus the title and up to 3,500 characters of text from each of those pages, numbered 1 to 8. It is told to cite every claim with its source number and to say so plainly when the sources don't contain the answer.
4. You get the answer with numbered links to the pages it used.

We chose keyword retrieval on purpose for the first version. It is predictable, it needs no extra index to keep in sync, and when it misses, you can usually see why. It also has clear limits, which brings us to the failures.

Where it fails

The words don't match. With keyword search, a page that says "reimbursement" won't be found for a question about "expenses". Semantic search reduces this problem without removing it. The fix on your side is to ask with the words you used when you wrote the note, or to use the team's real vocabulary in page titles.

The answer is spread across too many pages. "What did we decide in every planning meeting this year?" needs dozens of pages. A system that reads eight will answer from eight, and the answer will look complete when it isn't. Narrow questions work better: one project, one month, one decision.

The page is long and the answer is near the end. Systems send excerpts, not whole books. Ours sends the first 3,500 characters of each page. If the decision sits at the bottom of a 30-minute meeting transcript, it may never reach the model. Put decisions near the top of a page, or in their own section that starts early.

The notes contradict each other. If an old page says refunds over €200 need approval and a newer one says €500, the model sees both. It may pick one, mention both, or blend them. Citations help here: two numbers on one claim is a hint to open both pages.

The notes are wrong. Retrieval finds what you wrote, not what is true. A stale page produces a confident, well-cited, wrong answer. No model fixes that. Owners and review dates do, and we wrote about that in how to build a team wiki people actually use.

The model misreads a source. Models still make mistakes. They can attach a claim to the wrong source, or merge two facts that sit next to each other. This is rarer when the instructions demand a citation for every claim, but it happens. Treat the answer as a pointer to the page, and read the page for anything that matters.

There is nothing to find. A good system says so instead of guessing. Ours is instructed to answer in one sentence that the notes don't cover the question, and to suggest what to search for instead. If you get an answer with no citations, be suspicious.

Access control matters more than it sounds

An "ask your notes" feature is only safe if retrieval respects permissions. If a colleague keeps salary notes in a private page, those must never become an excerpt in your answer, even if your question matches them perfectly.

In Prompt Notebook the search only runs over pages you own or pages someone added you to. A private page of someone else's is outside the search before the model ever sees anything. When you evaluate any tool with this feature, ask the vendor how they enforce this. "The AI is told not to reveal private pages" is not an answer. The private page should never reach the AI at all.

How to write notes that answer well

You don't need to change how you write much. A few habits help any system, keyword or semantic:

• Put the answer near the top. Decisions first, discussion after.
• Use specific titles. "Pricing test plan, November" beats "Notes 14".
• Write names, dates and numbers in full. "Lea owns the test plan, due 24 October" can be found and cited. "She'll do it by next week" can't.
• One topic per page. Long catch-all pages dilute retrieval, because they match everything a little.
• Delete or update stale pages. An outdated page is the most common source of a wrong answer that looks right.

What to expect

Used well, asking your notes is a faster search with a summary on top. It is good at "where did we write down X?" and "what did we decide about Y?", and it saves the time of opening five pages to find one sentence. It is weak at questions that need everything you've ever written, and it can't know what you never wrote down.

The citations are the feature. Click them. If you want to try it on your own notes, Prompt Notebook is free and includes 30 AI actions a day. Import a few Markdown pages, ask a question you already know the answer to, and see whether the sources hold up.