SearchInstance

Source: src/Cloudflare/AI/SearchInstance.ts

A Cloudflare.AI. Search (formerly AutoRAG) instance — a fully managed retrieval-augmented generation pipeline over your own data.

An instance continuously indexes a data source (an R2 bucket or a web crawl), embeds it into a managed Vectorize index, and answers search and chat queries against it. Creation returns immediately; the initial indexing run happens asynchronously.

The instance instanceId, namespace, type, source, and embeddingModel are fixed at creation — changing any of them triggers a replacement. Everything else (models, chunking, caching, reranking, public endpoint, sync interval) is mutable in place.

For the common R2 case, prefer the Search construct, which also mints the service token the indexer needs to read your bucket. Use this low-level resource directly when you manage the token yourself, share one token across instances, or group instances under a SearchNamespace.

Creating a SearchInstance

R2-backed instance

An R2 source needs a service token to read the bucket. Either pass a tokenId (see SearchToken) or let the Search construct provision one for you.

const bucket = yield* Cloudflare.R2.Bucket("docs", {});
const instance = yield* Cloudflare.AI.SearchInstance("docs-search", {
  source: bucket.bucketName,
  tokenId: serviceToken.id,
});

Tuned retrieval settings

const instance = yield* Cloudflare.AI.SearchInstance("docs-search", {
  source: bucket.bucketName,
  aiSearchModel: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
  chunkSize: 512,
  chunkOverlap: 64,
  maxNumResults: 20,
  cache: true,
  cacheThreshold: "close_enough",
});

R2 source options

For an r2 source, sourceParams filters which objects are indexed (all fields optional):

• prefix — only index keys under this prefix.
• includeItems / excludeItems — micromatch glob patterns (* within a path segment, ** across segments; max 10 each). Only objects matching an includeItems pattern are indexed; excludeItems takes precedence.
• r2Jurisdiction — R2 data-residency jurisdiction of the source bucket.

const instance = yield* Cloudflare.AI.SearchInstance("docs-search", {
  source: bucket.bucketName,
  tokenId: serviceToken.id,
  sourceParams: {
    prefix: "docs/",
    includeItems: ["/docs/**"],
    excludeItems: ["/docs/drafts/**"],
  },
});

Web-crawler source options

sourceParams.webCrawler tunes how a web-crawler source is fetched, parsed, and stored. All fields are optional.

parseType selects how pages are discovered:

• "sitemap" (Cloudflare default) — read <seed>/sitemap.xml (discovered via robots.txt) and index the URLs it lists.
• "crawl" — start at source and follow links.
• "feed-rss" — treat the seed as an RSS / Atom feed.

crawlOptions controls link discovery (mainly for parseType: "crawl"):

• depth — how many links deep to follow from the seed.
• includeSubdomains — also crawl subdomains of the seed host.
• includeExternalLinks — follow links off the seed host.
• maxAge — skip re-fetching pages younger than this (seconds).
• source — where links come from: "all", "sitemaps", or "links".

parseOptions controls how each page is parsed:

• useBrowserRendering — render JS in a headless browser before parsing.
• includeImages — index image content.
• specificSitemaps — explicit sitemap URLs to read (for "sitemap").
• contentSelector — { path, selector }[] CSS selectors scoping which part of a page is indexed per URL path.
• includeHeaders — extra request headers sent while crawling.

storeOptions overrides where crawled content is stored — Cloudflare provisions managed storage by default:

• storageId — R2 bucket name to store crawl output in.
• storageType — "r2".
• r2Jurisdiction — R2 data-residency jurisdiction for the store bucket.

Basic web-crawler instance

const instance = yield* Cloudflare.AI.SearchInstance("site-search", {
  type: "web-crawler",
  source: "https://example.com",
  sourceParams: { webCrawler: { parseType: "crawl" } },
});

Fully-configured crawl

const instance = yield* Cloudflare.AI.SearchInstance("site-search", {
  type: "web-crawler",
  source: "https://example.com",
  sourceParams: {
    webCrawler: {
      parseType: "crawl",
      crawlOptions: {
        depth: 3,
        includeSubdomains: true,
        includeExternalLinks: false,
        maxAge: 86_400,
        source: "all",
      },
      parseOptions: {
        useBrowserRendering: true,
        includeImages: false,
        contentSelector: [{ path: "/docs", selector: "main" }],
      },
    },
  },
});

Sitemap and RSS sources

// Index the URLs listed in one or more sitemaps (the default parse mode).
const fromSitemap = yield* Cloudflare.AI.SearchInstance("sitemap-search", {
  type: "web-crawler",
  source: "https://example.com",
  sourceParams: {
    webCrawler: {
      parseType: "sitemap",
      parseOptions: { specificSitemaps: ["https://example.com/sitemap.xml"] },
    },
  },
});

// Treat the seed as an RSS / Atom feed.
const fromFeed = yield* Cloudflare.AI.SearchInstance("feed-search", {
  type: "web-crawler",
  source: "https://example.com/feed.xml",
  sourceParams: { webCrawler: { parseType: "feed-rss" } },
});

Store crawl output in a specific R2 bucket

const instance = yield* Cloudflare.AI.SearchInstance("site-search", {
  type: "web-crawler",
  source: "https://example.com",
  sourceParams: {
    webCrawler: {
      parseType: "crawl",
      storeOptions: { storageId: "my-crawl-bucket", storageType: "r2" },
    },
  },
});

Grouping under a namespace

SearchInstances live in a namespace (the account-provided default when unspecified). Pass a SearchNamespace’s name to group related instances — the engine then orders this instance after the namespace on deploy. The namespace is immutable; changing it replaces the instance.

const ns = yield* Cloudflare.AI.SearchNamespace("docs-ns", {});
const instance = yield* Cloudflare.AI.SearchInstance("docs-search", {
  source: bucket.bucketName,
  namespace: ns.name,
});

Binding to an Effect Worker

Bind the instance during the Worker’s init phase with Cloudflare.AI.QuerySearch(instance), which attaches the single-instance ai_search binding and returns an Effect-native client whose search / chatCompletions methods return Effects. Provide QuerySearchBinding in the Worker’s runtime layer.

import * as Cloudflare from "alchemy/Cloudflare";
import * as Effect from "effect/Effect";
import { HttpServerRequest } from "effect/unstable/http/HttpServerRequest";
import * as HttpServerResponse from "effect/unstable/http/HttpServerResponse";

export default class Api extends Cloudflare.Worker<Api>()(
  "api",
  { main: import.meta.url },
  Effect.gen(function* () {
    const bucket = yield* Cloudflare.R2.Bucket("docs", {});
    const instance = yield* Cloudflare.AI.SearchInstance("docs-search", {
      source: bucket.bucketName,
    });
    const search = yield* Cloudflare.AI.QuerySearch(instance);

    return {
      fetch: Effect.gen(function* () {
        const request = yield* HttpServerRequest;
        const query = new URL(request.url).searchParams.get("q") ?? "";
        const answer = yield* search.chatCompletions({
          messages: [{ role: "user", content: query }],
        });
        return yield* HttpServerResponse.json(answer);
      }),
    };
  }).pipe(Effect.provide(Cloudflare.AI.QuerySearchBinding)),
) {}

Binding to an Async Worker

For a vanilla async fetch Worker, pass the instance under Worker.env. The engine attaches the same ai_search binding and InferEnv types env.SEARCH as the runtime SearchInstance handle.

export const Api = Cloudflare.Worker("api", {
  main: "./worker.ts",
  env: { SEARCH: search },
});
export type ApiEnv = Cloudflare.InferEnv<typeof Api>;

// worker.ts
export default {
  async fetch(request: Request, env: ApiEnv): Promise<Response> {
    const query = new URL(request.url).searchParams.get("q") ?? "";
    return Response.json(
      await env.SEARCH.chatCompletions({
        messages: [{ role: "user", content: query }],
      }),
    );
  },
};