Data maps

The shape of a data map, field by field.

A data map is a list of fields to extract from a page. It is used inline by the data_map extractor, referenced by id from a data schema for validation, and stored per host for the host_data_map extractor (see Field extraction with data maps for a walkthrough, and Data maps for the HTTP API that manages stored host data maps). This page documents the map shape itself.

Field	Type	Required	Default	Description
`fields`	list of fields	yes	none	The fields to extract. At least one is required.

Field

A field is one of three distinct kinds, distinguished by which keys it carries: a scalar field, an array field, or an object field. A map's fields, and an object field's nested fields, can mix all three kinds freely.

Scalar field

A single value read from one field source.

Field	Type	Required	Default	Description
`name`	string	yes	none	The output name of the field.
`source`	field source	yes	none	Where the value comes from.

Array field

A repeated structure, such as a list of records, produced by selecting a set of elements and extracting an item field from each.

Field	Type	Required	Default	Description
`name`	string	yes	none	The output name of the field.
`root`	root selector	no	none	The selector for the repeated elements. When omitted, the array has exactly one element, with `item` evaluated against the current node rather than a re-selected set.
`item`	field	yes	none	The field describing each repeated element. Itself a scalar, array, or object field.

Object field

A group of named sub-fields, nested under the parent field's name.

Field	Type	Required	Default	Description
`name`	string	yes	none	The output name of the field.
`fields`	list of fields	yes	none	The nested sub-fields. At least one is required.

Field source

A field source is an object whose type selects where a value comes from.

Type	Fields	Description
`extractor` (xpath)	`kind: xpath`, `params.expressions` (list of strings)	Extract the value with XPath expressions.
`extractor` (selector)	`kind: selector`, `params.selectors` (list of selector expressions)	Extract the value with CSS selectors.
`extractor` (json)	`kind: json`, `params.selectors` (list of selector expressions), `params.expression` (string)	Locate an element, parse its content as JSON, and evaluate a CEL expression against it. See JSON expression.
`context`	`key` (string)	Take the value from the crawl context by key.
`static`	`value` (any)	Use a constant value.

Selector expression

Field	Type	Required	Default	Description
`expression`	string	yes	none	The CSS selector.
`accessor`	accessor	no	none	What part of the matched element to read. Defaults to its text.

Accessor

An accessor is an object with a type field selecting one of three forms.

Type	Field	Description
`attribute`	`name` (string)	Read the named attribute.
`text`	`recursive` (boolean)	Read the text, optionally including descendant text.
`html`	`outer` (boolean)	Read the element's HTML, inner or outer.

JSON expression

The json extractor source is for pages that embed pre-structured data directly, most commonly as JSON inside a <script type="application/ld+json"> tag (a convention named "JSON-LD," used by many product and content pages to describe what's on the page in a machine-readable way). It locates an element with params.selectors (the same selector expressions the selector source uses, including the same accessor options), reads a string out of the matched element, parses that string as JSON, and evaluates params.expression against the parsed value to produce the field's final value.

Field	Type	Required	Default	Description
`selectors`	list of selector expressions	yes	none	Selectors locating the element whose content is the JSON.
`expression`	string	yes	none	A CEL (Common Expression Language) expression evaluated against the parsed JSON, bound to a variable named `data`.

When a selector's accessor is omitted, the json source reads the matched element's text content (unlike the selector source, it does not special-case meta, a, or img tags, since JSON payloads are read as text, not attributes).

The result of params.expression becomes the field's value as-is: a string, number, boolean, or null stays that type, and an object or array is kept as a nested structure rather than being flattened to a string. For example, given:

<script type="application/ld+json">
{
  "@type": "Product",
  "offers": { "priceCurrency": "USD", "price": "29.99" }
}
</script>

the expression data.offers.price produces the string "29.99", while data.offers produces the whole nested object { "priceCurrency": "USD", "price": "29.99" }. When a page wraps its JSON-LD in a @graph array of mixed types, CEL's filter lets you pick out the one you want without knowing its position: data["@graph"].filter(g, g["@type"] == "Product")[0].offers.price.

Root selector

A root selector locates the repeated elements of a repeated field.

Field	Type	Required	Default	Description
`type`	`selector`	yes	none	Always `selector`.
`kind`	`xpath` or `selector`	yes	none	Whether the expressions are XPath or CSS selectors.
`params.expressions`	list of strings	yes	none	The expressions selecting the repeated elements.

Example

fields:
  - name: title
    source:
      type: extractor
      kind: xpath
      params:
        expressions:
          - //h1/text()
  - name: products
    root:
      type: selector
      kind: selector
      params:
        expressions:
          - .product-card
    item:
      name: product
      fields:
        - name: title
          source:
            type: extractor
            kind: xpath
            params:
              expressions:
                - .//h2/text()
        - name: price
          source:
            type: extractor
            kind: selector
            params:
              selectors:
                - expression: .price
  - name: sku
    source:
      type: extractor
      kind: json
      params:
        selectors:
          - expression: script[type="application/ld+json"]
        expression: data.sku