The first time you wire a real agent to the open web, you discover the boring truth: the model is the easy part. Finding clean, recent, citation-ready text on whatever the user just asked about — that’s where projects die. Raw Google scraping is fragile and against terms. A vector DB full of yesterday’s PDFs doesn’t help when somebody asks about a feature that shipped this morning. And once your agent loops, your $5/Mtok model is paying for a 40-result SERP of junk every turn.
That gap is why the AI search API category exists, and why it splintered into four very different products. Pick wrong and you’ll either burn your budget or quietly poison your agent’s output with stale snippets. Pick right — usually two of them, stacked — and it’s one of the highest-leverage decisions in the stack.
Here’s how I’d think about the lineup in May 2026, what each of them is actually for, and where the cost math gets ugly.
The category map nobody draws clearly
Before any vendor comparison, you have to know which kind of tool you’re shopping for. The “AI search API” label has been stretched to cover four pretty different jobs.
SERP wrappers sit in front of Google or Bing and hand you their result page as JSON. Serper, SerpAPI, ScraperAPI, Bright Data SERP. You get titles, snippets, and links — the same thing a browser does, minus the scraping headache. These are cheap and fast, but the result is a 200-word snippet, not the actual article. Your model still has to fetch the page.
LLM-native search is the newer flavor. Tavily, Exa, Linkup, You.com’s research mode. These crawl the web themselves (or partner with publishers), rerank with their own embeddings, and return passages that look like they were designed for an LLM context window. You ask “what changed in MCP this week” and get five paragraphs with URLs, not a SERP page.
Crawl and extract is the page-level layer. Firecrawl, Jina Reader, Spider. Hand them a URL, get back clean markdown — no nav junk, no cookie banners, no JavaScript-rendered noise. Most production agents end up calling these after a search API to actually read the top results.
Independent search engines with APIs is the small fourth bucket: Brave Search API, Kagi Search API, Marginalia, Common Crawl-backed services. Different index, different ranking, often better for queries that Google buries under SEO sludge.
Most teams I talk to end up running two of these in production — typically an LLM-native search plus a crawl-and-extract — because no single tool nails both “find the right URLs” and “extract clean text from them.”
Tavily: the default I keep coming back to
Tavily has become the closest thing to a default in agent search, and the reason isn’t magic — it’s that the API does exactly what an agent loop needs and not much more. You send a query, you get back five to ten passages with URLs, scores, and optional full-page content in a single round trip. There’s a search_depth parameter that lets you trade latency for recall, and a deep research mode that does its own multi-query expansion under the hood.
What I like: the answer is genuinely LLM-shaped. No SERP parsing, no nav text, no popup overlays in the content blob. Latency is usually under two seconds for basic searches and well under ten for advanced ones with extraction. Citations come back as structured fields you can hand to a frontend without regex gymnastics.
What I don’t like: pricing on the advanced tier scales fast if you have a chatty agent. Basic search is roughly half a cent per call (as of mid-2026 — verify on their docs), but flip to advanced search with extraction and you’re closer to four cents. A research agent that calls search five times per user message turns into real money at 10k daily sessions.
If I were starting a new agent project today and could only call one search API, this is where I’d start, then optimize down.
Exa: when the question is fuzzy, not factual
Exa is the one I reach for when keyword search is the wrong frame. Their pitch is neural search — embedding-based retrieval over a crawled web index — and the find-similar endpoint where you give it a URL and it returns conceptually adjacent pages. For longform retrieval (“essays that argue X,” “papers that benchmark Y”), it consistently beats keyword SERPs by enough that I notice.
The catch is the same as the strength. Exa’s neural index isn’t trying to find the most recent news story. If you ask “what happened in the OpenAI keynote yesterday,” you’ll get a stale essay about OpenAI’s product strategy, not yesterday’s coverage. They’ve added a keyword fallback and date filters, but the tool genuinely shines on questions where “similar to” beats “matching tokens.”
I use Exa as the secondary search inside research agents — first pass with Tavily for fresh hits, then an Exa find-similar on the best result to surface deeper context. The two stacks complement each other better than either alone.
Serper: the boring answer that’s right more often than it should be
Serper is a thin, fast Google SERP wrapper. No reranking, no LLM-aware extraction. You get titles, snippets, and URLs at something like a tenth of a cent per query — cheap enough that you stop thinking about cost.
Why does this still win sometimes? Because Google is still very good at “what is the official documentation page for this product,” and an agent that needs a known canonical URL doesn’t need a neural model to find it. Tools like Serper are the right answer when:
- You’re navigating to a known site and want the canonical URL (“Stripe pricing page”)
- Your agent already has a strong reranker downstream
- You’re high volume on hobby-budget infra
It is not the right answer for deep research, freshness-sensitive workflows, or anything where you actually want a passage in the response — you’ll be doing a second fetch on every result, which adds latency and cost back in.
SerpAPI, the older incumbent, lives in the same lane but covers many more engines (Bing, DuckDuckGo, Baidu, Yandex, Amazon, eBay) with structured JSON for each. If you need Amazon results in a comparison agent, that’s where to look. For pure web search, Serper is usually cheaper.
Firecrawl: the layer everybody forgets to budget for
Here’s the unglamorous reality of running an agent loop: most of the calls aren’t search calls, they’re fetches. The model picks three URLs from a search result, and now you have to actually read those pages. Cookie banners. JS-rendered content. PDFs. Rate limits.
Firecrawl is the cleanest answer I’ve used for this. Their /scrape endpoint returns clean markdown for a URL. Their /search endpoint stitches that together so you get the search step and the extract step in one call. They added a structured-extract mode where you can pass a schema and get JSON back — handy for product comparison agents.
The thing nobody mentions in the marketing copy: extract pricing is per page, not per query. Search-then-extract on five results costs five extract credits plus the search. If your agent re-extracts the same URL three times in one session because nothing is cached, you’re paying three times. I’ve seen Firecrawl bills triple just from a missing redis cache in front of the fetch layer.
Worth pairing with: Tavily, when you want Tavily’s reranking but Firecrawl’s extraction quality.
Jina: the kit nobody packages
Jina’s the odd one. They ship three things that matter for agent search and they don’t always sell them as one product: Reader (turn any URL into clean markdown — there’s even a free tier you can hit with no auth), DeepSearch (their LLM-native search), and a reranker API that’s genuinely useful as a separate layer.
The DIY appeal is real. You can build a pretty capable agent search stack on Jina Reader plus a cheap LLM for query rewriting plus their reranker, and your cost per query goes way down. The trade-off is glue code — Tavily ships you a finished result, Jina ships you parts. For teams with one engineer on the agent and not much time, that math points to Tavily. For teams running tens of millions of queries a month, the per-unit savings start to matter.
The free Reader endpoint is a gift for prototypes. I’ve shipped working demos on r.jina.ai/<url> alone.
Brave Search API and the independent-index case
Brave runs its own web index — not a Google or Bing rewrap. The API gives you news, web, and image endpoints with pretty generous free tiers and clean licensing terms.
Two reasons to care. First, the licensing: if you’re building anything where you’ll display search results in a product, the rights situation around Google SERP wrappers gets uncomfortable. Brave is upfront that you can use the data. Second, the index is different. For some queries — particularly ones Google has buried under SEO content farms — Brave’s results are noticeably less spammy.
That said, Brave’s index has more gaps than Google’s, especially for long-tail technical content. I treat it as a complementary source, not a replacement.
Linkup, You.com, Kagi: the long tail worth knowing
Linkup partners with premium publishers — paywalled or licensed content you can’t reach through a SERP wrapper. If your agent answers questions where citing the FT or Bloomberg matters, this is the only honest way to get there.
You.com’s API has two modes: web (fast, SERP-style) and research (multi-step with reasoning). The research mode is a hosted “agent that does search for you,” which is either exactly what you want or duplicative of your own agent loop. Kagi Search API rounds out the list for teams who want Kagi-quality results in their product — not cheap, not high-volume, but genuinely good.
The cost math nobody puts in the marketing page
Pricing pages all show different units. Per query, per 1k queries, per credit, per page extract, per token. Here’s the rough shape as of mid-2026 — check official docs for current numbers, because this category reprices every quarter:
- Cheapest SERP wrapper (Serper basic plan): roughly $0.30–$1 per 1,000 queries
- LLM-native search basic (Tavily, Exa): roughly $5–$8 per 1,000 queries
- LLM-native search advanced with extraction (Tavily advanced): closer to $30–$40 per 1,000
- Page extraction (Firecrawl, Jina Reader): roughly $0.5–$2 per 1,000 pages on volume tiers
Run that against a real agent. If your average user session triggers five search calls and ten page fetches, you’re looking at something like $0.30 per session on the advanced path and $0.05 on the budget path. For a B2C agent at 10k sessions a day, the difference is $90k a year — bigger than most teams’ inference bill.
The trick most production teams settle on: cache aggressively (URL-keyed, TTL tuned to acceptable staleness), use the cheap path for navigation queries, and save the advanced path for actual research turns. I’ve seen this cut search bills 70% with no impact on output quality.
Stack patterns that actually work
A few combinations I’ve seen ship and survive contact with users:
Search-then-extract: Tavily basic for the discovery step, Firecrawl for the extract step, with a redis cache keyed on the URL between them. Works well when you want clean separation of concerns and you need to swap one piece later.
Single-vendor research: Tavily advanced or You.com research mode. One API call, one bill, one place to debug. Right when you’re prototyping or when the team is small enough that ops cost matters more than per-query cost.
Neural-first: Exa for the discovery, Jina Reader for the fetch, your own reranker on top. Right when you’re doing concept-heavy retrieval (academic papers, longform essays, idea similarity) instead of fact lookup.
Cheap-and-cheerful: Serper plus Firecrawl plus a tiny rerank step (Cohere Rerank or a Jina Rerank call). Maximum cost control. The setup work is real but the per-query bill is the lowest of any working pattern.
The one I’d avoid: trying to use a single SERP wrapper as your whole search layer. The 200-character snippets a SERP returns are almost never enough for an agent to reason on, and you’ll spend more debugging hallucinated citations than you would have on a real LLM-native search service.
How to pick, by what you’re building
If you’re building a deep research agent (think Perplexity-style answers with citations), start with Tavily advanced or Linkup. Pair with Firecrawl for the extract layer. The cost stings but the output quality justifies it.
For a real-time news or finance agent where freshness matters in minutes, Tavily with its news endpoints or Brave Search News are your best bets. Exa is the wrong choice here — its index isn’t optimized for minute-level recency.
For RAG over the open web where the question is conceptual (“find me essays that argue against X”), Exa wins more often than not. Its find-similar endpoint is the differentiator nobody else has matched yet.
For an e-commerce or comparison agent, SerpAPI for the structured engine coverage (Amazon, eBay, Walmart) plus Firecrawl for retailer pages.
For an academic or longform retrieval workflow, Exa first, then Semantic Scholar’s API for the citation graph step. Tavily is fine but doesn’t reach into the academic corpus the same way.
For a low-budget hobby agent, Jina Reader free tier plus Brave Search free tier gets you surprisingly far. Add Serper if you need more than 2k queries a month.
And for the team that’s allergic to anything that touches Google ToS — Brave plus Tavily, both with clean licensing positions, gets you most of the way to a Google-class result quality with none of the lawyer conversations.
One thing to try this week
Pick whatever agent you’re building, instrument the search calls with timing and a content-length log, and read fifty of them by hand. Not the queries — the responses. You’ll find that maybe two-thirds of them returned something the model could actually use, and the rest were SEO sludge, paywalls, or empty extractions. That ratio is the real signal of whether your search layer is working, and no benchmark page on a vendor site will tell you the answer for your traffic.
Once you’ve seen the numbers from your own queries, the choice between Tavily and Exa and the rest mostly makes itself.