Extractors
Pipeline steps that pull structured data out of fetched content.
Extractors produce the output of a crawl: markdown, raw content, or structured records. They appear as extractor steps in a pipeline. This page catalogs the available kinds and their parameters.
How extractors are configured
An extractor is a pipeline step whose params carry a kind selecting the extractor and its options:
steps:
- kind: extractor
params:
kind: markdownraw
Emits the page content unchanged. It takes no parameters.
steps:
- kind: extractor
params:
kind: rawmarkdown
Converts the page to clean markdown.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
content_xpath | string | no | //body | The XPath of the region to convert. |
skip_tags | list of strings | no | built-in list | Tag names to drop before converting. Defaults to a built-in list of non-content tags: script, style, noscript, iframe, template, head, svg, canvas, object, embed, form, button, input, textarea, select, label, option, and dialog. |
steps:
- kind: extractor
params:
kind: markdown
params:
content_xpath: //main
skip_tags:
- nav
- footerdata_map
Extracts named fields using a data map supplied inline.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
map | data map | yes | none | The data map describing the fields to extract. |
schema_id | UUID | no | none | A data schema to validate the extracted record against. |
steps:
- kind: extractor
params:
kind: data_map
params:
map:
fields:
- name: title
source:
type: extractor
kind: xpath
params:
expressions:
- //h1/text()host_data_map
Extracts fields using a data map stored for the page's host, looked up automatically at extraction time.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
validate | boolean | no | true | Whether to validate the extracted record against the stored map's data schema. |
steps:
- kind: extractor
params:
kind: host_data_map
params:
validate: trueFor the data map shape itself (fields, sources, repeated structures), see Data maps.