Configuration

inndx is configured hierarchically: built-in defaults, then a config file, then environment variables (highest priority). This page documents what each setting does. Production wiring of these settings with secrets is part of enterprise onboarding; here we describe the knobs themselves.

The settings are grouped into the shared sections every service reads (General, Data stores, Secrets, Messaging and coordination, Runtime, Retention, and Observability) and the per-service settings that each service adds on top.

How configuration is loaded

Configuration is resolved in three layers, each overriding the one before:

1. Built-in defaults.
2. A configuration file, if one is given with the --config-file flag or the INNDX_CONFIG_FILE environment variable. The file may be YAML, TOML, or JSON; the format is detected from the file extension.
3. Environment variables prefixed with INNDX, which override the file and defaults.

Run inndx config to print the fully resolved configuration.

Several sections are tagged unions that select a backend with a kind field (for example the cache, broker, blob storage, and columnar store). For those, set kind and then the fields for that kind.

Environment variables

Environment variables map onto the nested configuration with a double-underscore separator: INNDX__SECTION__FIELD. For example, INNDX__SERVER__PORT=8022 sets the server.port value.

INNDX__SERVER__PORT=8022
INNDX__DATABASE__PRIMARY=postgresql://user:pass@db:5432/inndx
INNDX__COLUMNAR_STORE__KIND=duckdb
INNDX__CACHE__KIND=redis
INNDX__CACHE__URL=redis://localhost:6379

Nested values follow the same rule, with each level joined by double underscores:

• Lists index their entries by position: append __0__, __1__, and so on. A field within an entry follows as another segment. For example, the first configured proxy's identifier is INNDX__FETCHER__PROXIES__0__IDENTIFIER, and the first address of that proxy is INNDX__FETCHER__PROXIES__0__ADDRESSES__0.

• Maps use each entry's key as a path segment. A secondary blob store named archive, for example, is configured under INNDX__BLOB_STORAGE__SECONDARY__ARCHIVE__*:

  copy

  INNDX__BLOB_STORAGE__SECONDARY__ARCHIVE__KIND=file
  INNDX__BLOB_STORAGE__SECONDARY__ARCHIVE__DATA_DIR=./archive

A list-valued field may alternatively be set from a single variable holding a JSON array.

Duration values

Fields typed as a duration accept a human-readable string such as 30s, 5m, 1h, or a combination like 1h 30m. The same format is used everywhere a duration appears in this page.

General

General identity settings, under the meta section.

Data stores

Where inndx persists crawl state, analytics, and content.

Database

The relational database holding crawl state. The driver is selected by the URL scheme (sqlite://... or postgresql://...).

Database options

Columnar store

The columnar store holding analytics data (audit logs, crawl logs, metrics). Selected by kind.

columnar_store:
  kind: duckdb
  path: analytics.db

Blob storage

Blob storage holds raw content and results. A deployment has one primary store and an optional map of named secondary stores; the storage.target on a to_blob action's identifier source selects a secondary store by name. A to_blob action can also carry its own object storage connection inline, instead of naming one configured here; see Secrets for how that connection's credentials can be either typed in literally or resolved from a configured secret backend.

Blob store

A blob store is selected by kind.

blob_storage:
  primary:
    kind: s3
    endpoint_url: https://s3.example.com
    bucket: inndx
    access_key: <your-access-key>
    secret_key: <your-secret-key>

Secrets

Some component parameters, rather than naming something this deployment already configured ahead of time, can carry connection details directly: an access key, a token, a password. Any such field accepts either a literal value, or a reference to a secret resolved from a configured secret backend at the moment that field's value is actually needed, never any sooner and never written back to this deployment's own database. A reference is an object with a ref field naming the secret, and an optional backend field naming which configured backend to resolve it from:

access_key: { ref: s3-access-key }
secret_key: { ref: s3-secret-key, backend: vault_primary }

When backend is omitted, resolution uses the backend named env, which always exists, with no configuration required: it reads the secret's name as an environment variable on the machine running the service. Additional backends are configured under secrets.backends, a map of operator-chosen names to a backend definition selected by kind; a reference's backend field names one of these map keys, not a kind, so a deployment can run more than one backend of the same kind (for example, two separate HashiCorp Vault servers) simply by giving each one a different key.

Secret backend

A secret backend is selected by kind.

secrets:
  backends:
    vault_primary:
      kind: vault
      address: https://vault.example.com
      token: <your-vault-token>
      mount: secret
  cache_ttl_seconds: 30

A secret named under the vault backend is a path into that mount, optionally followed by #field_name when the secret stored at that path has more than one field: database/creds#password reads the password field of the secret at database/creds, while a path with no #field_name reads that secret's only field.

Messaging and coordination

How services share state and discover one another. In single-process dev mode the in-process defaults suffice; a distributed deployment points these at shared backends.

Cache

The cache shared across services. Selected by kind.

cache:
  kind: redis
  url: redis://localhost:6379

Broker

The message broker carrying work between services. Selected by kind.

broker:
  kind: kafka
  bootstrap_servers: localhost:9092

Service registry

The registry services use to discover one another. Selected by kind, with the same options as the cache: memory (default), redis (url), and redis_cluster (urls).

service_registry:
  kind: redis
  url: redis://localhost:6379

Runtime

How each service process binds and schedules its own work.

Server

The HTTP server each service binds.

Task queue

The in-process worker queue each service uses to run its work.

Retention

Background, age based deletion of data that has grown stale, so storage doesn't grow without bound.

Retention resources

Each entry below is independently configured under retention.resources.

The configurable resources are:

A small number of global, deduplicated resources (URL hosts, URL links, and their labels) are not configurable here and are always kept, since they may be shared across many crawl job runs.

retention:
  enabled: true
  sweep_interval_seconds: 300
  resources:
    crawl_job_runs:
      ttl_seconds: 2592000 # 30 days
      batch_size: 100
    trigger_executions:
      ttl_seconds: 7776000 # 90 days

Observability

Tracing, metrics, and error reporting.

Telemetry

OpenTelemetry export of traces and metrics.

Sentry

Error reporting to Sentry.

Per-service settings

Each service has its own configuration section (orchestrator, fetcher, parser, sink, analytics) holding its component groups and a few operational fields alongside them. Every component is an object with an enabled flag in addition to its own settings, so any component can be turned on or off. What each component does and its per-job parameters live in the component reference; this page covers which components each service has and the operational fields next to them.

orchestrator

orchestrator:
  filters:
    robots_txt:
      enabled: false
  stopping_criteria:
    max_urls:
      enabled: true
  start_stop_interval: 10s
  run_evaluate_concurrency: 8

The component groups under orchestrator. Each holds the components shown, every one an object with an enabled flag and enabled by default. See the linked reference page for what each does and its per-job parameters.

The scheduling fields directly under orchestrator:

fetcher

The fetcher section configures the browser clients and any proxies, in addition to enabling clients and middleware. The browser-client settings are operational counterparts to the per-job client params on Clients.

fetcher:
  clients:
    cdp:
      enabled: true
      headless: true
  middleware:
    set_language:
      enabled: false
  timeout: 30s
  concurrency: 10

Proxies

The proxies list configures the connection to the proxy servers:

The proxy addresses can be a list of explicit hosts and ports:

fetcher:
  proxies:
    - identifier: myproxy
      addresses:
        - proxy1.example.com:8080
        - proxy2.example.com:8080
      scheme: http
      username: user
      password: pass

Or it can be an expanded template:

fetcher:
  proxies:
    - identifier: myproxy
      addresses:
        host:
          - proxy1.example.com
          - proxy2.example.com
        port:
          start: 8080
          end: 8085
      scheme: http
      username: user
      password: pass

Where the host under addresses is a single host, or a list of hosts, and the port is either a single port, a list of ports, or a range defined with start and end.

Clients

The clients section configures the available fetcher clients. Each client has its own settings, but all have the enabled flag like other components to turn them on or off. All clients besides standard are disabled by default.

The standard (HTTP) client has no startup settings of its own. The browser clients are below.

playwright client

cdp client

stealth_cdp client

The stealth_cdp client has the same startup settings as the cdp client.

remote_cdp client

remote_stealth_cdp client

The remote_stealth_cdp client has the same startup settings as the remote_cdp client.

Remote browsers

The browsers list on the remote_cdp and remote_stealth_cdp clients configures the connection to each remote browser:

Middleware

The middleware section enables or disables request and response middleware. Each is an object with an enabled flag, and all are enabled by default. See Middleware for what each does and its per-job parameters.

parser

parser:
  conditions:
    fingerprint:
      enabled: false
  concurrency: 10

The component groups under parser. Each holds the components shown, every one an object with an enabled flag and enabled by default. See the linked reference pages for what each does and its per-job parameters.

The heuristics group lists the heuristic kinds available to the heuristics condition, and feature_sets defines the feature sets used by fingerprint conditions and the data-map extractors.

The field directly under parser:

sink

sink:
  actions:
    to_file:
      enabled: true
      directory: ./output
  concurrency: 10

The actions group under sink enables or disables the delivery actions. Each is an object with an enabled flag, and all are enabled by default. See Actions for what each does and its per-job parameters.

The field directly under sink:

analytics

The analytics service currently has no service-specific components or fields beyond the shared sections.