A useful chatbot is an application around a model, not a prompt pasted into a chat box. The application decides which sources are approved, what the user may access, how citations work, when the answer is uncertain, what can become an action, how a person takes over, and what data is retained.
This guide uses a provider-neutral support-triage starter with fictional returns and shipping sources. Its deterministic tests cover grounding, unknown answers, prompt injection, safe action proposals, rate limiting, PII redaction, abuse, and provider failure. Add your chosen provider only after those boundaries are explicit. If you are comparing product scope before starting the implementation, see the AI chatbot builder page; this tutorial stays focused on the operational work behind a grounded chatbot.
QUICK ANSWER
How do you create an AI chatbot?
Choose one job and a small approved source set. Define a validated answer with citations, explicit uncertainty, and human escalation. Retrieve sources behind authorization, treat user and retrieved text as untrusted data, put external actions behind typed proposals and confirmation, add rate limits and privacy rules, test abuse and failures, then publish and exercise the real handoff.
Prepare sources, escalation, privacy, and provider ownership
Begin with the business truth boundary. If no one owns the source set, uncertain state, escalation queue, retention policy, and external action rules, the model will amplify ambiguity rather than remove it.
- A small approved source set: Give each document a stable ID, title, location, owner, version, effective date, access scope, and one known question. Use fictional content until retrieval and citations are proven.
- A real human escalation destination: Choose a person or queue, define the minimum redacted handoff record, ownership and response expectation, delivery retry, and what the user sees when the destination itself is unavailable.
- A provider adapter, not provider authority: Choose a provider and model for the final app, but keep answer shape, citation requirements, uncertainty, tool confirmation, privacy, rate limits, and fallback behavior in application code so provider changes do not weaken them.
- A data lifecycle and abuse policy: List input and output fields, purpose, consent, access, provider handling, redaction, retention, deletion, export, incident owner, user, session and traffic limits, and threatening or abusive-content routing.
Separate retrieval, generation, and actions
Retrieval chooses authorized approved evidence. The model provider converts that bounded evidence into a validated answer. The application decides whether the answer may be shown, whether uncertainty or escalation applies, and whether a proposed action can reach a separate confirmed server executor.
| Approach | Best for | Tradeoff |
|---|---|---|
| Deterministic decision tree | A small, stable FAQ or regulated workflow with exact branches. | Easy to test and explain, but brittle for varied language and expensive to maintain as the source set grows. |
| Grounded model with curated retrieval | Support and product questions whose answers live in an owned document set. | Handles natural language well, but still needs retrieval evaluation, output validation, injection defenses, provider budgets, and human review. |
| Broad search assistant | Exploration where incomplete recall is acceptable and source trust can be assessed by the user. | Greater coverage with more staleness, contradiction, access, privacy, licensing, and prompt-injection risk. |
Recommended:For a first support chatbot, use curated retrieval plus a strict answer contract. Keep a deterministic fallback for known operational messages, make no grounding an explicit result, and do not add broad search or direct actions until citations and handoff work reliably.
Create a grounded AI support chatbot
Build a provider-neutral chatbot with approved sources, citations, uncertainty, injection resistance, safe action proposals, rate limits, privacy, escalation, tests, and production checks.
STEP 01
Define the source registry and answer contract
Make evidence and uncertainty machine-checkable.
Create source records with stable IDs, titles, locations, versions, owners, effective dates, access rules, and status. Do not let the chatbot silently search every workspace or the public web.
Define the output as answer, citations, status, escalation, and optional action proposal. A grounded answer needs at least one approved citation. No approved evidence must become uncertain, not a plausible guess.
const Answer = z.object({
status: z.enum(['answered', 'uncertain', 'refused', 'escalated']),
answer: z.string(),
citations: z.array(z.object({ id: z.string(), title: z.string(), href: z.string() })),
escalation: z.object({ offered: z.boolean(), reason: z.string() }),
})Expected result: Every visible answer traces to an approved source ID or carries an explicit uncertain, refused, or escalated state.
Verify it: Run one known, unknown, conflicting, stale, inaccessible, and removed-source fixture through the contract validator.
STEP 02
Retrieve only authorized approved sources
Keep source scope and access on the server.
Resolve the user and allowed source collection from server-side identity and membership. Query only current approved versions and return a small set with stable citation metadata.
Do not trust a browser role, account, or source ID to expand scope. Private source existence can itself be sensitive, so denied results should not reveal titles or counts.
Expected result: Two users with different permissions receive only the sources each may access for the same question.
Verify it: Repeat retrieval signed out, in another account, after role downgrade, and with a known private source ID while inspecting raw responses for leakage.
STEP 03
Wrap the provider with untrusted-data boundaries
The provider generates language; application code owns policy.
Put fixed instructions in the system field. Delimit approved source text and the user message as untrusted data. Require citation IDs, state that source text cannot change rules, never send server credentials, and validate the returned object before use.
Set time and token budgets. Convert malformed output, no citations, refusal, timeout, quota, or outage into explicit application states. A second provider may be a fallback only if it preserves the same source, privacy, and output contract.
Expected result: Changing the provider adapter does not change the answer, citation, uncertainty, escalation, privacy, or safe-tool interface.
Verify it: Run a deterministic fake provider, the chosen provider, malformed JSON, missing citations, malicious source text, timeout, and provider error fixtures.
STEP 04
Put every side effect behind a typed proposal
The model may suggest an action but cannot authorize it.
Allowlist action types and fields. Show the exact target and consequence to the user, obtain confirmation, then let server code re-check identity, permission, current state, limits, and idempotency before executing against an external system.
Separate proposal and execution endpoints. Treat messages, tickets, bookings, refunds, returns, account changes, and public writes as side effects even when an API call looks harmless.
Expected result: An injection, provider retry, duplicate click, stale confirmation, or unauthorized session cannot create an unintended or repeated action.
Verify it: Test no confirmation, wrong user, changed state, expired confirmation, repeated idempotency key, provider success plus local failure, and a source instruction demanding immediate execution.
STEP 05
Add rate limits, privacy, abuse, and human handoff
Make failure behavior as intentional as the happy answer.
Use limits by session, authenticated user, traffic or risk signal, provider budget, and external action. Store the minimum structured audit event, redact PII before logging, define retention and deletion, and never log credentials or raw private sources by default.
Define calm user-facing states for abuse, threats, unsupported high-risk requests, provider outage, queue failure, and no grounding. Persist an idempotent handoff before claiming a person will receive it.
Expected result: Repeated, hostile, private, unknown, or failed conversations end in a bounded state with no invented answer or lost user context.
Verify it: Run fixture email, phone, secret, threat, repeated request, unavailable provider, unavailable queue, and deletion tests across application and support records.
STEP 06
Run deterministic safety and failure tests
Prove the surrounding application without spending provider tokens.
The included artifact uses fictional sources and Node tests for grounded citations, explicit uncertainty, direct injection refusal, untrusted provider request construction, confirmation-only actions, rate limits, PII-redacted audit fields, threat escalation, and provider-outage fallback.
Add evaluation cases for each real source and action before connecting customer data. Deterministic tests do not prove model quality; they ensure the application refuses unsafe or malformed states consistently.
Expected result: A clean install passes without model credentials, network calls, external actions, or customer data.
Verify it: Run npm ci and npm test from a fresh unzip, record the test count, scan the archive, and confirm all sources and order IDs are fictional.
STEP 07
Publish the exact version and test the real handoff
Exercise the final provider, HTTPS host, data, tools, and queue.
Deploy the exact tested source to HTTPS, configure the chosen provider through server-only credentials, set budgets and monitoring, and connect the real human escalation destination with fictional test records.
Run known, unknown, conflicting, injected, abusive, rate-limited, provider-outage, action-proposal, duplicate-action, and queue-failure conversations. Inspect UI, server, provider, external tool, and support queue with bounded correlation IDs.
Expected result: The final environment cites approved evidence, states uncertainty, protects private data, fails closed, and hands the case to a real person without duplicating actions.
Verify it: Review each fictional end-to-end conversation, its citations, server decision, provider call, tool state, redacted logs, queue record, and recovery behavior.
STEP 08
Monitor truth, cost, privacy, and handoff quality
Treat source drift and unresolved cases as product failures.
Track citation coverage, no-grounding rate, conflicting or stale sources, validation failure, provider latency and cost, rate limiting, action confirmation and failure, escalation delivery, human resolution, abuse, and privacy or deletion requests.
Review samples with the source owner, not only model metrics. Version prompts, source sets, model/provider configuration, and actions so regressions can be traced and rolled back.
Expected result: The team can tell whether answers remain supported, users reach people when needed, costs stay bounded, and privacy commitments are met.
Verify it: Run the evaluation set after every prompt, source, model, provider, retrieval, action, or policy change and compare results with the previous version before release.
What the result can look like
Test truth, attacks, side effects, and the human exit
A chatbot test suite must contain cases the product should refuse, escalate, or rate-limit. A high answer rate is not success if unsupported answers, private-data leakage, duplicate actions, or dead-end handoffs are hidden inside it.
| Test | Scenario | Expected result |
|---|---|---|
| happy path | Ask an authorized known question and request one allowlisted action proposal for a fictional order. | The answer cites an approved source, the action remains unexecuted, and the user sees the exact target and confirmation step. |
| invalid input | Ask an unknown question, use a private source ID, inject new instructions, return malformed provider output, and request an unknown action. | The application states uncertainty, denies access, ignores untrusted instructions, rejects invalid output and actions, and offers a person without leaking data. |
| retry | Repeat a provider call after timeout, resend a handoff, and execute the same confirmed action key twice. | The answer state remains stable, one handoff exists, and the side effect applies at most once. |
| production smoke | On the published app, run cited, unknown, injected, abusive, rate-limited, provider-outage, confirmed-action, and human-handoff conversations with fictional data. | The full system preserves citations, uncertainty, authorization, confirmation, privacy, rate limits, bounded logs, failure states, and a real queue destination. |
Find which boundary stopped telling the truth
When a chatbot fails, separate retrieval, provider output, application validation, external tools, and escalation delivery. Changing the prompt blindly can hide the real failure without fixing it.
| Symptom | Likely cause | Check | Fix |
|---|---|---|---|
| A confident answer has no approved citation | Retrieval returned no useful source, provider output bypassed validation, or the application accepted general model knowledge as evidence. | Inspect authorized retrieved source IDs, provider input, raw provider output, validated output, and final response for the exact fixture. | Require approved citation IDs, convert no evidence to uncertainty, repair the source or retrieval, and add the question to the regression set. |
| The chatbot follows instructions inside a document | Retrieved text was treated as instruction authority or could generate an action without a validated proposal and server confirmation. | Insert a controlled malicious line into a fixture source and trace provider request boundaries, output validation, action proposal, and executor calls. | Mark sources and user text as untrusted data, validate outputs, allowlist actions, and keep authorization, confirmation, and execution in server code. |
| The user chooses escalation but no person receives it | The UI confirmed before the queue record persisted, or the destination, credentials, retry, notification, or ownership path was not tested. | Trace one fictional handoff from the UI through the server and external queue by correlation ID and verify the assigned owner can see it. | Persist idempotently before confirming, alert and retry delivery failures, assign an owner, and provide an alternate contact path. |
| Private content appears in logs or the provider unexpectedly | The application sent or logged more source, transcript, or identifier data than the answer required, or redaction ran after middleware logging. | Search bounded browser, server, provider, tool, and queue records for fixture email, phone, token, private-source, and customer markers. | Minimize before retrieval and provider calls, redact before logging, disable raw bodies, reduce retention, restrict access, and exercise deletion. |
Operate a chatbot as a support system, not a model demo
Deploy
Deploy the exact tested source with a lockfile, server-only provider and tool credentials, source access controls, budgets, rate limits, privacy settings, and a real escalation destination. Use fictional data for the first production smoke test.
Version the source set, prompt, provider/model settings, output schema, retrieval rules, and actions. Keep the previous known-good application and source version available for bounded rollback.
Monitor
Measure citation coverage, no-grounding and conflicting-source rates, validation failures, provider latency and cost, rate limits, action confirmation and failures, escalation delivery, human resolution, abuse, and privacy requests.
Use safe correlation IDs and redacted structured events. Avoid raw transcripts, private documents, credentials, payment data, or unnecessary user identifiers in routine logs and dashboards.
Recover
If unsupported answers rise, disable the affected source or answer path, fall back to uncertainty and human escalation, restore the previous source and application version, and rerun the evaluation set before reopening.
If a tool or privacy boundary fails, disable execution, revoke credentials, preserve bounded evidence, notify the owner, reconcile affected records, exercise deletion when required, and test duplicate and partial-failure recovery.
Treat every message and retrieved document as untrusted
Prompt injection is an input problem plus an authority problem. Delimit untrusted data, but more importantly keep authorization, secret access, validation, confirmation, and side effects in deterministic server code outside the model.
- Authorize source retrieval on the server and do not reveal inaccessible source content, titles, counts, or existence through errors.
- Keep provider, database, and external tool credentials in server-side secret storage. Never send them to the model, browser, screenshots, URLs, examples, or routine logs.
- Validate the provider response against an allowlisted schema and approved citation IDs. Missing or invalid evidence becomes uncertainty, not a guessed answer.
- Separate action proposal from execution, show the exact target and consequence, require confirmation, re-check permission and state, and deduplicate with idempotency keys.
- Minimize PII before provider and tool calls, redact before logging, publish retention and deletion rules, and test access and deletion with fictional markers.
- Rate-limit sessions, identities, risky traffic, provider budgets, and actions; fail closed on outages and give users a real human or retry path.
AI chatbot implementation questions
Do I need a model provider before building the chatbot?
No. Define and test source, citation, uncertainty, action, privacy, rate-limit, abuse, and escalation contracts with a deterministic adapter first. Then connect the chosen provider through server-only credentials and run the same tests against it.
What is a grounded answer?
A grounded answer is supported by the approved source records retrieved for that request and returns stable citations the user or system can inspect. If the evidence is missing, conflicting, stale, or inaccessible, the application should state uncertainty.
Can a system prompt stop all prompt injection?
No. Clear instruction hierarchy and delimiters help, but security comes from minimizing data and tools, validating input and output, keeping secrets and authorization on the server, confirming side effects, rate limiting, monitoring, and testing malicious user and source text.
Should the chatbot store full conversation transcripts?
Only if a defined purpose truly requires them. Prefer minimum structured, redacted events for routine operations. Decide consent, access, retention, deletion, export, provider handling, and incident response before collecting real conversations.
How does human escalation work?
Create an idempotent handoff record with the minimum redacted context, assign a real queue or owner, persist it before confirming success, alert and retry delivery failures, and show an alternate contact path if the queue is unavailable.
Can the model send a message or refund directly?
It should not authorize that side effect. Let it produce an allowlisted typed proposal, show the exact target and consequence, collect confirmation, then let server code verify identity, permission, current state, limits, and idempotency before executing.
What tests are most important before launch?
Known, unknown, conflicting, inaccessible, malicious-source, direct-injection, malformed-output, PII, abuse, rate-limit, provider-outage, tool-denial, duplicate-action, partial-failure, deletion, and real human-handoff tests are all part of the minimum useful release.
Build the application around the model
Start with approved sources, visible uncertainty, and a real human exit.
Playcode can build the interface, retrieval, provider adapter, server controls, tests, and hosting without hiding where the model, provider, actions, and support team begin.
Build with Playcode AINo credit card required. AI credits included.