Web Search Infrastructure
4 min read
User Guide | Administration |
|---|---|
n/a |
|
Architecture Overview
System Context
The diagram above illustrates how Unique AI interacts with an external web search engine. While this documentation uses Google Programmable Search Engine as the example, the same technical flow can apply to other compatible web search engines as well.
Here’s a more detailed breakdown of how the system works:
User Input
The user writes a message in Unique AI that has web search capabilities enabled.Triggering the Web Search
If the language model (LLM) determines that external information is needed, or if the user explicitly requests a web search, the web search flow is initiated.Generating the Search Query: The LLM analyzes the conversation history and surrounding context to generate an initial search query.
Refining the Query: A separate LLM call is made to refine the original query. This step improves search accuracy and uses only the generated query (not the full conversation).
Executing the Search: The refined_query is sent to the web search engine.
Receiving Search Results: The web search engine returns a list of relevant links based on the query.
Processing and Responding
Unique AI scrapes the content from the returned links, processes the information, and uses it to generate a response to the user’s original question.
While all communications are secure and protected, the risk with the web search feature is the potential leakage of surrounding context in queries sent for search engine. This risk should be carefully evaluated before enabling web search.
Container Overview
This container diagram for Unique AI Web Search outlines how user queries are handled to retrieve information from the web. The end user writes a query, which is securely transmitted via HTTPS or WSS to the Unique AI Chat interface — a React-based frontend with web search capabilities. This frontend communicates with the Unique Assistant-Core, a Python-based backend service that hosts the agentic LLM logic. The Assistant-Core securely retrieves a search engine API key from Azure Key Vault, which stores configuration values leveraged for end-to-end automation.
Once authorized, the Assistant-Core sends a web query to an external Web Search Engine (Google Programmable Search Engine) over HTTPS. The search engine responds with relevant web pages, which the Assistant-Core processes and streams back the LLM response to the Unique AI Chat frontend.
While all communications are secure and protected, the risk with the web search feature is the potential leakage of surrounding context in queries sent for search engine. This risk should be carefully evaluated before enabling web search.
Google Connectivity
The Web Search module requires access to the Google Custom Search API, meaning it relies on external resources to function.
It does not communicate with any other Unique backend services. All data is retrieved exclusively from the Google Custom Search API hosted at cse.google.com.
Planning
Budget
Google search JSON API Pricing
The search api is priced as follows:
100 search queries per day for free
5$ per 1000 queries, up to 10k queries per day
It is possible to increase the 10k quota through the Google Cloud API console > Quota section.
Provisioning
Configuration of the assistants-core service
The assistants-core service expects 3 environment variables.
Stored in a key vault
GOOGLE_SEARCH_API_KEY: vault (manual-google-search-api-key)
Stored in code
.yamlGOOGLE_SEARCH_API_ENDPOINT=https://customsearch.googleapis.com/customsearch/v1GOOGLE_SEARCH_ENGINE_ID=00c5f6e9ac4644f42
The GOOGLE_SEARCH_API_ENDPOINT is exposed so that customers could use any wrapper engine around Google. The GOOGLE_SEARCH_ENGINE_IDis responsible of identifying the default configuration that you want to use for all your spaces (e.g. safe search). It’s possible to override this value from the search engine configuration in your space/tool. The environment variable must exist and can be supplied from your compliant secret origin.
Administering keys
To obtain a search engine (ID) as well as an API key you can follow the official guide.
Ensure you store the API key in a secure spot (e.g. an Azure Key Vault) which is also available to the assistants-core service.
Operating
Monitoring
Google Search provides basic monitoring via the Cloud Platform Console's API Dashboard. For more advanced monitoring—such as alerts and custom metrics— the Google Cloud's Operations suite can be used.
Authentication
Unique AI Web Search uses the public Google Custom Search API, which supports API keys only—service accounts are not supported.
Two environment variables must be configured for authentication:
GOOGLE_SEARCH_API_KEY: This API key is included as a parameter in the GET request sent from Unique to the Google API.GOOGLE_SEARCH_ENGINE_ID: Provided to the assistants-core service, this value is passed as the cx query parameter to specify which custom search engine to use.
GET https://www.googleapis.com/customsearch/v1?key=GOOGLE_SEARCH_API_KEY&cx=GOOGLE_SEARCH_ENGINE_IDTroubleshooting
Is the search correctly instantiated?
The Google credentials are loaded during the startup of the assistant-core service. If the credentials are successfully loaded, the following message is logged at INFO level:
"Google Search API is properly configured"If initialization fails, a WARNING log will be generated with the following message:
Google Search API missing required settings: <missing env variables>General Troubleshooting
If the credentials are correctly loaded in the assistants-core service, the next step is to verify that the Google API is responding properly. You can do this by running the following command:
curl "https://customsearch.googleapis.com/customsearch/v1?key=GOOGLE_SEARCH_API_KEY&cx=GOOGLE_SEARCH_ENGINE_ID&q=uniqueai"If the API is working as expected, it will return a JSON response containing search results. If that succeeds, the next recommended step is to check the assistants-core logs for any error messages. These logs will help identify the root cause of the issue.