Build a retrieval-augmented generation workflow

Define the answer boundary#

Build a release-note assistant that answers one question precisely: “Can an administrator export reports as CSV in HarborDesk 3.2, and what is the row limit?” HarborDesk, its documents and every result below are hand-authored fictional fixtures. The deliverable is an answer whose claims can be checked against a specific release, plus an explicit branch for missing evidence.

Retrieval-augmented generation supplies external evidence before generation; Cohere’s RAG overview provides provider-specific background. Here, separate four responsibilities: identify the requested release, select readable evidence, construct an answer, and verify its support. A matching keyword alone does not establish the requested capability.

Assume the reader may view public documentation but cannot view internal drafts. Explaining an administrator feature does not authorize the reader to execute it. Our application answers documentation questions only; it performs no export. Require the answer to identify version 3.2, preserve the administrator condition and cite the row limit. It must not silently borrow a planned feature from another release.

Prepare three traceable sources#

Use this tiny corpus before introducing an embedding index. Each passage has a stable source ID, release scope, publication state and visibility. The text is deliberately short so a reviewer can inspect the entire evidence set.

Store the passage text with its revision, not merely a link to a page that may change. Keep D32:p4 together: splitting the administrator condition from the limit would make the answer harder to verify. When extracting longer documents, preserve the heading that identifies the release.

For this experiment, use one immutable corpus snapshot and record its identifier beside each answer. Keep access decisions in application code. The model receives only evidence the reader may see; a source’s own text cannot change its visibility. A source update should produce a new snapshot before another evaluation run.

Retrieve, filter and select context#

The following fixed rankings represent a deliberately naive local retrieval fixture, not measured search performance. They show why choosing the first matching passage is insufficient. Internal diagnostics may inspect this closed fixture; production permissions must restrict the searchable collection before content reaches external scoring or generation.

The resulting context contains exactly D32:p4. No higher rank can override an access or release constraint. Keep excluded identifiers in restricted diagnostic logs if needed, not in the public answer. If nothing eligible remains, report the evidence gap rather than relaxing the constraints.

When adding semantic retrieval, use the encoder’s documented query/document settings; see Cohere’s embedding documentation for one implementation. Keep this three-passage fixture unchanged while comparing retrievers. Inspect whether each change preserves the correct release and eligibility decision before testing a larger corpus.

Generate a supported answer and a missing-evidence branch#

Give generation the question, the selected passage and its source ID. Require a short answer restricted to what that passage establishes. A correct answer for the fixture is:

Yes. In HarborDesk 3.2, administrators can export reports
as CSV from Reports > Export. [D32:p4]
Each file is limited to 10,000 rows. [D32:p4]

Verify two claims: the administrator-only export procedure and the per-file row limit. Both point to D32:p4 and retain its conditions. Do not replace “per file” with “per day,” infer unlimited exports, or promise that the current reader has administrator permissions.

Now ask: “Does 3.2 support scheduled nightly exports?” D32:p4 establishes manual export but says nothing about scheduling. The correct branch is: “These 3.2 notes describe manual CSV export [D32:p4]. They do not establish whether nightly scheduling is available, so I cannot confirm that capability from this evidence.” Missing documentation does not prove a feature is absent.

Save a compact trace for each branch:

{
  "corpus": "harbordesk-fixture-v1",
  "release": "3.2",
  "selected_source_ids": ["D32:p4"],
  "answer_status": "supported_answer",
  "claim_source_ids": [["D32:p4"], ["D32:p4"]]
}

For the scheduling question, set the status to insufficient_evidence; do not turn the internal 3.3 draft into a supporting citation.

Diagnose failures before changing the system#

Review retrieval and generation independently. If the final answer says CSV is unavailable because it used D31:p2, the release filter failed. If context contains D32:p4 but the answer promises nightly scheduling, the answer exceeded its evidence. If D33:p1 reaches a public response, the authorization boundary failed even when the quoted sentence is accurate.

For this fixture, acceptance requires the correct release and administrator condition, the exact per-file limit, valid source IDs and appropriate uncertainty about scheduling. Inspect each requirement directly. A fluent answer, a retrieved-source count or a citation marker cannot substitute for those checks.

Retest the same questions after each change and retain the old trace for comparison. Add an empty-result question and a paraphrase only after the core paths behave as specified. This three-document exercise establishes harness behavior on known inputs, not retrieval quality over an enterprise corpus. Larger evaluation needs representative documents, independent questions and checks for extraction errors, conflicting revisions and distribution changes. Preserve the small fixture as a regression case while expanding that evidence.