Fetch
Fetch one URL or a batch of URLs on demand.
The fetch endpoints retrieve content immediately without creating a crawl job, useful for testing how a URL responds.
These endpoints share the error conventions.
Reusable shapes
These shapes appear in the request and response bodies below.
Cookie
- name stringrequired
The cookie name.
- value stringrequired
The cookie value.
- domain string
The domain the cookie applies to.
- path string
The path the cookie applies to.
- expires string
When the cookie expires.
format: date-time- http_only boolean
Whether the cookie is restricted to HTTP requests.
- secure boolean
Whether the cookie is sent only over secure connections.
- same_site string
The cookie's same-site policy.
LaxStrictNone
Redirect
- url string
The URL that issued the redirect.
- location string
The URL the redirect points to.
- side string
Whether the redirect was issued by the server or the client.
serverclient- type string
Whether the redirect is permanent or temporary.
permanenttemporary
Fetch a URL
/v1/fetchFetches a single URL and returns its response.
- url stringrequired
The URL to fetch.
- method stringdefault: GET
The HTTP method to use.
GETPOSTPUTDELETEHEADOPTIONSPATCHTRACECONNECT- body string
The base64-encoded request body.
- client objectdefault: standard
Selects the fetch client (
standardby default; other kinds enable browser-based fetching). See fetch clients.- middleware object[]default: []
An array of request and response middleware objects. See middleware.
- user_agent stringdefault: inndx
The user agent to send. Accepts the keywords
inndx,chrome,firefox,safari,random, or a custom user-agent string.- referrer string
The referrer to send with the request.
- headers string[][]default: []
Header name and value pairs to send.
- cookies Cookie[]default: []
Cookies to send. See Cookie.
- proxies string[]default: []
Proxy identifiers configured for the fetcher.
- timeout stringdefault: 30s
Request timeout as a duration, for example
30sor5m.- max_redirects numberdefault: 0
The maximum number of redirects to follow.
- max_retries numberdefault: 0
The maximum number of retries on failure.
curl -X POST 'http://localhost:8022/v1/fetch' \
-H 'X-Tenant-Id: acme' \
-H 'Content-Type: application/json' \
-d '{ "url": "https://example.com" }'Fetch a batch of URLs
/v1/fetch/batchFetches multiple URLs in one request.
Per-URL failures are reported inside results as error items rather than failing the whole request.
- items object[]required
The requests to fetch, between 1 and 64 items. Each item uses the same shape as the single fetch request body.
curl -X POST 'http://localhost:8022/v1/fetch/batch' \
-H 'X-Tenant-Id: acme' \
-H 'Content-Type: application/json' \
-d '{ "items": [{ "url": "https://example.com" }, { "url": "https://example.org" }] }'