AI chatbot knowledge readiness starts with deciding which questions the business can answer reliably from approved material. Before connecting documents to a chatbot, identify their owners, resolve contradictions and define when the system should hand a question to a person. A large document collection is not the same as a usable source of answers.
Start with questions, then find their evidence
Collect a representative set of real questions from enquiries, support conversations and internal requests, removing personal details from test material. Group them by the decision the reader needs to make. “Can you help with my project?” may need a service explanation; “Where is my order?” requires current account-specific information and a different workflow.
For each question, record the approved source, expected answer, information that must not be inferred and fallback route. This creates a question-to-evidence map. If staff cannot identify an authoritative answer, assign a content task rather than expecting the chatbot to reconcile an unresolved business decision.
Choose a narrow first scope, such as explaining published service options. Keep live account changes and individual commitments outside that initial scope unless separately designed and tested. The point is to establish an answerable boundary that can be evaluated, then expand it deliberately.
Build a source register with owners and expiry decisions
The source register should identify each document, its owner, intended audience, version, effective date and review date. Record whether it is approved for chatbot use. Distinguish current guidance from archived examples, unfinished drafts and customer-specific agreements. A file's recent modification date does not prove that its advice is current.
Resolve overlaps explicitly. If the website says one thing and a sales presentation says another, ask the responsible team which applies. Record the decision and update the conflicting material where appropriate. Ranking one document above another in retrieval settings cannot substitute for resolving the underlying disagreement.
Keep enough context with each section to explain its scope. A paragraph saying “available on request” is unhelpful when separated from the service it describes. Preserve headings, relevant qualifications and source references when preparing content. Test the resulting material using the questions it is supposed to answer.
Define useful answers and useful refusals
Write an expected response for each test question, including acceptable variations. A good answer should address the question, stay within the evidence and identify the source where the interface supports it. A citation is something to verify, not proof that every sentence in the answer is supported.
Define fallback behaviour for missing, contradictory or outdated material. The chatbot might explain that it cannot confirm a detail, offer the relevant contact route and capture a concise question for staff. Avoid asking the visitor to repeat the entire conversation when handing over.
Separate “not in the knowledge base” from “not appropriate to answer here”. A public service bot may possess a document that is unsuitable for its audience. Test those boundaries independently of whether retrieval finds a plausible paragraph.
Keep document content separate from operating instructions
OWASP describes indirect prompt injection as instructions embedded in external material processed by an AI system. Its prompt injection prevention guidance recommends treating retrieved content as untrusted and enforcing tool permissions outside the model. For this pilot, use approved sources and avoid giving the answer workflow unnecessary action-taking tools.
Include a harmless test document containing an instruction unrelated to its subject, such as a request to change the answer format or disclose an unrelated test record. The expected behaviour is that source text does not acquire authority to change the workflow. Test with dummy information and record both the answer and any attempted tool activity.
The knowledge readiness review
- Every in-scope question has an approved source or an explicit fallback.
- Each source has a named owner and a clear audience.
- Contradictory guidance has been resolved rather than merely hidden.
- Archived versions cannot appear as current guidance.
- Answers can be checked against the exact source version used.
- Content changes trigger a proportionate repeat of affected question tests.
- Staff can withdraw a source and verify that it is no longer used.
Assign acceptance to the business owner of the information, supported by the implementation team. A developer can demonstrate that retrieval works, but the service owner must decide whether the resulting explanation is accurate and useful.
Rehearse a changed service and a missing answer
In a fictional pilot, a company has two service guides with different onboarding descriptions. Resolve the conflict, approve the current guide and create questions covering both the standard process and an exception. Include one question about a service the company does not offer.
Run the set before and after replacing the approved guide. Expect the updated answer to use the new evidence, the exception to retain its qualification and the unsupported service question to reach the defined fallback. Remove the guide entirely and confirm that the bot does not continue presenting its old answer as verified.
Review unsupported statements, wrong-source answers and unnecessary handovers separately. A single overall success percentage can hide which part of the knowledge process needs attention. Keep the test set and source versions together so later comparisons mean something.
If the proposed chatbot will also initiate work, read our human approval workflow guide before widening its scope. A staff-facing knowledge workspace may also need application development for source review and ownership. To discuss a bounded knowledge pilot, explore Visuosofts AI automation and contact the team with sample questions and their approved answers.