API overview

One API for Scrape, Search, Extract, Batch, Crawl and Watch.

Base URL and authentication

Send JSON requests to https://api.crawlkit.sh with your key directly in Authorization.

Request headers

1
2Authorization: ck_your_api_key
Content-Type: application/json

Core operations

Choose the operation that matches your request.

Endpoint	Use it to
POST /scrape	Read one page in your chosen formats.
POST /search	Discover public web pages as structured search results.
POST /extract	Extract fields with a JSON Schema or instruction.
POST /crawl	Start a website crawl.
POST /batch	Submit up to 25 ordered Scrape or Extract items.
POST /watch	Create a scheduled page check.

Queued work

A 202 response means the request is queued. Poll GET /jobs/:id for Scrape or Extract, GET /crawl/:id for crawl results, or GET /batch/:id for ordered batch results. Reuse an Idempotency-Key only for the same request input.

Stream a batch

GET /batch/:id/events returns ordered server-sent events. Resume with Last-Event-ID or ?after=. Optional signed webhooks deliver the same durable events. Client disconnect never cancels batch work.

Resume progress

1
2curl -N 'https://api.crawlkit.sh/batch/BATCH_ID/events?after=12' \
  -H 'Authorization: ck_your_api_key'

Credits

1 credit per successful page or Watch check, plus 3 when AI is used. Screenshots included. Unused reservations are refunded. Search: 1 credit per 10 returned results, rounded up. Empty results cost 0 credits. Unused reservations are refunded. Credits do not expire.

Limits

Crawls default to 100 pages and depth 2, with a maximum of 1,000 pages. Watch intervals are at least 15 minutes. Only public HTTP(S) URLs are accepted. Text PDFs are supported; scanned PDFs require the bounded, opt-in OCR setting.

Watch changes

The first successful check creates a baseline. Failed checks do not replace it. Select structured fields with compare.paths (for example /price); array order matters unless compare.keyedSets is explicit. If the old extraction rule is unavailable, a rebaseline event has changeDetected=false. Events are kept for 30 days; the current baseline is preserved.

Verify Watch webhooks

Save webhookSecret when you create the watch. Calculate HMAC-SHA256 with that secret and the message below; compare its hex digest with x-crawlkit-signature (v1=hex). Use the raw request body, validate the timestamp and deduplicate by event ID.

Signed message

1x-crawlkit-event-id + "." + x-crawlkit-timestamp + "." + rawBody

Read or cancel a crawl

Read pages with GET /crawl/:id?offset=0&limit=20. Send POST /crawl/:id/cancel to stop the crawl; unused reserved credits are returned. Include and exclude filters use path globs. Add planner.mode=bestFirst with bounded query keywords to prioritize relevant URLs. Optional content filters and coverage or no-progress stopping return score provenance and a stop reason.