Quality and Evaluation

1 min read

After content processing, additional quality steps ensure the most relevant content reaches the orchestrator LLM and that the final answer is checked for accuracy.

Chunk Relevancy Sorting

Uses an AI model to evaluate and re-rank content chunks by relevance to the user's original question. This ensures the most useful content is prioritized when the token budget forces some chunks to be dropped.

Setting	Type	Default	Description
`chunk_relevancy_sort_config.enabled`	boolean	`true`	Whether to use AI-based relevancy sorting. When disabled, chunks retain their original order.

When enabled, each chunk is scored against the user's query and the chunks are reordered from most to least relevant before the token budget reduction step.

Answer Quality Checks (Evaluation)

After the orchestrator LLM generates its answer using the web search content, automated evaluation checks can be run to detect quality issues.

Setting	Type	Default	Description
`evaluation_check_list`	list of evaluation metrics	`[HALLUCINATION]`	Which quality checks to run on the generated answer.

Available Metrics

Metric	Description
`HALLUCINATION`	Detects when the generated answer contains claims that are not supported by the retrieved web search content.

The evaluation only runs when the tool returns content chunks. If the search returned no results, evaluation is skipped.

Source Citation Instructions

Controls how the AI model cites web search sources in its answers.

Setting	Type	Default	Description
`tool_format_information_for_system_prompt`	string (textarea)	Built-in citation instructions	Instructions injected into the orchestrator LLM's system prompt that specify how to format source references.

These instructions tell the AI how to:

Reference specific web sources when making claims.
Format inline citations.
Attribute information to its original source.