crawlctl

crawlctl is the command-line client for inndx. It is to inndx what kubectl is to Kubernetes: one consistent tool for inspecting and operating every resource the platform exposes, including crawl jobs and their runs, triggers, data schemas, host data maps, URL patterns, hosts, links, and artifacts, plus one-off tools for fetching a URL, parsing content, and driving interactive browser sessions.

Every command follows the same shape: a plural noun naming a resource (jobs, runs, triggers, schemas, data-maps, url-patterns, hosts, links, artifacts, audit-logs), a verb (list, get, create, update, delete, apply), and a shared set of global flags. Once you learn one resource you know them all.

This page covers the model you need to use crawlctl day to day. Run crawlctl --help for the command tree, crawlctl <command> --help for any command's flags, and crawlctl --help-detail to print the full command reference as Markdown. For the field-level shape of each resource (what goes in a job manifest, a trigger condition, and so on), see the matching HTTP API reference page; crawlctl sends the same documents the API accepts.

Connecting to the services

inndx is five services: orchestrator, fetcher, parser, sink, and analytics. crawlctl talks to each over its own URL, set per command with a flag or through the environment:

crawlctl --orchestrator-url http://orchestrator:8022 jobs list
export CRAWLCTL_FETCHER_URL=http://fetcher:8023

crawlctl ping reports each service's health and version. It is the recommended first check that a deployment is reachable:

crawlctl ping

Authentication is passed with --token <token> (sent as a bearer token) or one or more repeatable -H key=value headers. A tenant is selected with --tenant <tenant>, sent as the X-Tenant-Id header described in the HTTP API overview.

Output formats

The -o/--output flag is the same across every command:

name and cel= make crawlctl easy to compose in shell pipelines:

crawlctl jobs list -o name

crawlctl runs links <run-id> -o cel='items.map(l, l.url)'

Working with resources

The verbs are consistent across resources:

crawlctl jobs list                 # list, with filters as flags
crawlctl jobs get <id>             # a single resource, by id or name where unique
crawlctl jobs create -f job.yaml   # create from a document
crawlctl jobs update <id> --set schedule="0 * * * *"
crawlctl jobs delete <id>

Mutations echo a kind/id <action> line, such as job/<id> created or trigger/<id> disabled, so the effect of a command is obvious; -o json returns the resulting object instead. References can usually be given by a human-friendly name instead of a UUID where one is unique, for example a job or schema name, or a hostname instead of its hash; crawlctl resolves it.

The resources and where their full field shape is documented:

Listing and pagination

list commands take their filters as flags and return one page at a time. On a terminal the results page interactively (Enter for the next page, q to stop), redrawing in place. When the output is piped or --no-pager is set, a single page is printed followed by a cursor to resume with --page, keeping stdout clean for scripts. --all follows every page and returns the whole set:

crawlctl artifacts list --kind raw      # browse interactively
crawlctl jobs list --no-pager           # one page, no prompt
crawlctl links list --all -o json       # the entire set as JSON

Time filters such as --created-after, --before, and --occurred-after accept either an absolute RFC 3339 timestamp or a relative duration meaning "ago":

crawlctl runs list --created-after 20m
crawlctl audit-logs list --after 3d

Declarative apply

For jobs, schemas, data maps, URL patterns, and triggers, apply reconciles a document against the platform: creating the resource if it does not exist, updating it if it does. This keeps configuration in version control instead of one-off create calls:

crawlctl jobs apply -f job.yaml
crawlctl schemas apply -f schema.yaml

Identity comes from an explicit id in the document, or otherwise a natural key (a job's name, a data map's host and schema, and so on). apply reports created, updated, or replaced.

Building request bodies

Commands that take a document accept -f <file> (JSON or YAML, or - for stdin) plus any number of --set key=value overrides using dotted and bracketed paths:

crawlctl jobs create -f base.yaml --set schedule="@hourly" --set targets[0].url=https://example.com

echo '{"name":"demo"}' | crawlctl schemas create -f -

Logs and analytics

The analytics service records a per-run crawl log plus a global audit trail. Crawl logs hang off a run, so they live under runs:

crawlctl runs logs <run-id> --status error --step fetch
crawlctl runs metrics <run-id> --window hour --group-by status
crawlctl runs latency <run-id> --window five-minute
crawlctl audit-logs list --actor alice --action delete

runs metrics and runs latency return time-bucketed aggregates: counts by dimension, and latency percentiles. Crawl logs run at several times the volume of page visits, so lean on the filters and the pager to navigate them, or --all to export. See Inspect runs and logs for a walkthrough and the Crawl logs reference for every available dimension.

One-off tools

Fetch

fetch retrieves a single URL through the fetcher and writes the raw response body to stdout, or to a file with --out:

crawlctl fetch https://example.com > page.html
crawlctl fetch https://example.com --request-header 'Accept: application/json' -o json

With -o json you get the whole response (status, headers, cookies, redirects) instead of just the body. See Fetch for the underlying request and response shape.

Parse

parse is the parser's one-off counterpart to fetch: you provide page content and a pipeline configuration, and it returns what each pipeline extracted. Content comes from stdin or --content <file>, and the pipeline template comes from -f:

cat page.html | crawlctl parse https://example.com --content-type text/html -f pipelines.yaml

crawlctl fetch https://example.com | crawlctl parse https://example.com --content-type text/html -f pipelines.yaml

The result is a structured list of per-pipeline outputs, and the whole response is addressable with CEL, so you can pull out a single pipeline's data and pipe it onward:

crawlctl parse https://example.com --content-type text/html -f pipelines.yaml \
  -o cel='results.filter(r, r.identifier == "prices")[0].data.data'

See Parse for the request and response shape.

Sessions

A session is a live, interactive browser the fetcher keeps open. crawlctl manages these through a small background daemon, so each command is an ordinary, stateless invocation while the connection stays alive between them:

crawlctl sessions open https://example.com     # opens a session, prints its id
crawlctl sessions list                          # the live sessions
crawlctl sessions content > page.html           # the current page
crawlctl sessions evaluate 'document.title' -o json
crawlctl sessions fill '#search' 'widgets'
crawlctl sessions click '#submit'
crawlctl sessions screenshot --out shot.png
crawlctl sessions close                          # close it (stops the daemon if it was the last)

The daemon starts automatically on open and shuts down when the last session closes. Commands act on the oldest open session by default; target a specific one with --session-id <id>. A session closes itself after a period of inactivity, configurable with --keepalive (for example --keepalive 30m). Pre-start or tear down the daemon explicitly with crawlctl sessions daemon start and crawlctl sessions daemon stop. See Sessions for the underlying API.

Shell completions

Generate a completion script for your shell and load it:

crawlctl completions bash > /etc/bash_completion.d/crawlctl
crawlctl completions zsh  > "${fpath[1]}/_crawlctl"
crawlctl completions fish > ~/.config/fish/completions/crawlctl.fish

Full reference

This page covers the model, not every flag on every command. Run crawlctl --help-detail against your installed version to print the complete command reference, including every subcommand and flag, as Markdown.