Extended Unique Crawling Capabilities to Third Party Crawlers

Unique has added support for the following third party crawlers on demand:

• Firecrawl

• Jina Reader

• Tavily

These crawlers have more capabilities than the default internal solution (BasicCrawler). For example, they are capable of reading content from links pointing to pdf content.

The crawlers API keys must be configured on assistant-core services. If you wish to test one of these crawlers, please approach your Customer Success representative to have the corresponding API key configured in your environment.

Selecting Crawlers in Spaces

1. Go to Spaces > Tools > Web Search

2. in Web Search settings Crawler Config select the crawler

Configuring Crawlers for the Unique Platform

Firecrawl

The configuration of the firecrawl crawler is simple, only the timeout parameter can be altered.

Jina Reader

Please keep the default headers and do not change them if you don’t have experience with Jina Reader. You can find full documentation of the reader under this link.

Tavily

Tavily Extract endpoint has two different extraction depth possible, basic and advanced. The depth of the extraction process. advanced extraction retrieves more data, including tables and embedded content, with higher success but may increase latency. More details can be found under this link.