Mutators
The mutator components that transform URLs before they enter the frontier.
Mutators normalize or rewrite URLs before they are queued, which helps avoid duplicates and unwanted variants. A job lists mutators under config.mutators. This page catalogs the available kinds and their parameters.
How mutators are configured
Each entry in config.mutators is an object with a kind and an optional params object. Mutators are applied to a URL before it enters the frontier, so two URLs that normalize to the same value collapse into a single queue entry.
config:
mutators:
- kind: sanitize
params:
strip_fragment: true
strip_query: truesanitize
Normalizes a URL by removing parts that do not identify a distinct page.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
strip_fragment | boolean | no | true | Remove the #fragment portion of the URL. |
strip_query | boolean | no | false | Remove the ?query portion of the URL. Strip the query only when it does not change which page is served. |
mutators:
- kind: sanitize
params:
strip_fragment: true
strip_query: true