Skip to main content

Modality Reference

What is a modality?​

A modality is a property attached to an annotation in the image recognition (IR) response. It carries annotation-level metadata produced by a model or pipeline step (price, realogram position, facing count, fixture type, etc.).

Modalities are not strongly typed. Every value is a string; composite values are underscore-delimited (e.g. {price}_{quantity}_{text}). Consumers must know the format per key.

Response shape (API v3)​

Each annotation has a modalities list. Each modality has a key and one or more predictions:

{
  "annotation_id": 34,
  "annotation_type": "promo",
  "bbox": { "x": 2454, "y": 5001, "width": 1174, "height": 482 },
  "modalities": [
    {
      "key": "promo:price",
      "predictions": [
        { "value": "5_1",   "score": 1.0, "prediction_info": null },
        { "value": "7.5_1", "score": 1.0, "prediction_info": null }
      ]
    }
  ]
}

General behaviour​

  • Multiple predictions per key are allowed (e.g. two prices on one label: base + promo). We do not determine which is RRP vs promo — we just provide the values.
  • Empty modalities are omitted (since April 2026): if nothing is extracted, the key is not present at all. Previously, price modalities were returned with an empty value.

Modalities​

catalog-item​

What: Provides several options for which in-scope catalogue item the annotation belongs to. All predictions have a relative confidence score. The is_main field indicates that our models have determined this to be the correct label. If none have is_main=true, the models were unable to match to an existing catalogue item and the item is "unknown".

Value: catalogue-item UUID.

Example:

{
  "key": "catalog-item",
  "predictions": [
    { "value": "725d1bc8-9138-4748-8cbb-5e94574f9404", "score": 0.874146446, "is_main": true },
    { "value": "350a75d6-5500-4299-8f04-2be93ba6f48a", "score": 0.758885698, "is_main": false },
    { "value": "f9eedd9f-2157-4231-993a-6c8468a696c7", "score": 0.733337572, "is_main": false }
  ]
}

fixture​

What: The fixture/row type of a shelf-row annotation. Attached to shelf-row annotations (returned as bounding boxes) when the realogram return_fixture parameter is enabled, powered by the fixture-level model.

SHELF_ROW​

These are shelving units that are ambient temperature.

  • Ambient conventional primary shelf
  • Side Stacks
  • gondola end/end-cap

COLD_ROW​

These are shelving units that are chilled.

  • Coolers
  • Chilled row

GRIDWALL_ROW​

  • Gridwall/pegboard

SECONDARY_ROW​

These are free-standing promotional fixtures with their own floor footprint. In this case because rows are not universally applicable to the wide format of secondary display types, the annotation covers the entire display unit.

  • Free Standing Display Units (FSDU)
  • Freestanding Bin
  • Floor Stack

Values: COLD_ROW, SHELF_ROW, GRIDWALL_ROW, SECONDARY_ROW.

Example:

{
  "key": "fixture",
  "predictions": [
    { "value": "SHELF_ROW", "score": 1, "is_main": true }
  ]
}
note

A fuller fixture-type taxonomy (Main Fixture, Cooler, Chilled Shelf, Gondola End, Gridwall, T-bar, Hot Spot, Floor Stack, Side Stack, Clip Strip, FSDU, Freestanding Bin, Generic, Gutter Tray) is defined but not yet shipped as modality values.

realogram:shelf​

What: Which shelf (row) an annotation sits on within the realogram. Attached to product/POSM annotations. Additionally, shelf-row concept annotations (annotation_type: shelf_row) carry their own ID as a realogram:shelf value.

Value: shelf index as a string. Index starts at 1 (Top shelf is row 1, second from top is row 2 etc)

Example:

{
  "key": "realogram:shelf",
  "predictions": [
    { "value": "1", "score": 1, "is_main": true }
  ]
}

realogram:slot​

What: The horizontal slot (position along the shelf) of an annotation.

Value: slot index as a string. Index starts at 1 (left most is slot 1, second left slot 2 etc)

Example:

{
  "key": "realogram:slot",
  "predictions": [
    { "value": "1", "score": 1, "is_main": true }
  ]
}

realogram:stack​

What: The vertical stack position of an annotation within its slot (products stacked on top of each other).

Value: stack index as a string. Index starts at 0 (bottom in stack is stack position 0 etc.)

Example:

{
  "key": "realogram:stack",
  "predictions": [
    { "value": "0", "score": 1, "is_main": true }
  ]
}
Realogram notes (all three)
  • Realogram is calculated within the shelf rows of interest of an image.
  • Gap detection is deprecated: gaps are to be replaced by negative space (shelf-row area minus annotations for that shelf).

promo:price​

What: A price detected on a price/promo label, or the price in a SKU from an associated price-tag/promo. Multiple predictions indicate multiple valid prices — all are returned with is_main: true when the models consider them correct. We do not determine which is RRP vs promo; we just provide the values.

Format: {price}_{quantity}. Extended 3-part format {price}_{quantity}_{associated text} is in development — contact Neurolabs support (e.g. 1.2_1_Clubcard price).

Example: Consider a price annotation that has 2 instances of pricing information: "£5" and "2 for £8.50". The promo:price modality would provide the following output:

{
  "key": "promo:price",
  "predictions": [
    { "value": "5_1",   "score": 1, "is_main": true },
    { "value": "8.5_2", "score": 1, "is_main": true }
  ]
}

promo:quantity​

What: A quantity-based deal on the label. Either "buy X get Y free" or "buy A for the price of B". Multiple predictions apply.

Format: {buy-quantity}_{get-quantity}_{text}.

Example:

{
  "key": "promo:quantity",
  "predictions": [
    { "value": "1_2_buy 1 get 1 free",           "score": 1.0, "is_main": true },
    { "value": "3_4_buy 4 for the price of 3",   "score": 1.0, "is_main": true }
  ]
}

promo:price-discount​

What: A discount expressed on the label. Multiple predictions apply.

Format: {quantity}_{discount-as-fraction}_{text} — the discount is a fraction normalised to 0–1 (0.4 = 40% off).

Example:

{
  "key": "promo:price-discount",
  "predictions": [
    { "value": "1_0.4_40%", "score": 1.0, "is_main": true }
  ]
}

promo:text​

What: Free-text promotional wording detected on the label. Other observed values: save 1/3, buy 1 get 1 40%.

Example:

{
  "key": "promo:text",
  "predictions": [
    { "value": "buy 1 get 1 free clubcard super value rollback", "score": 1.0, "is_main": true }
  ]
}
Promo notes (all four)
  • The four promo:* keys are produced by the price/promo recogniser and populate for promo-flagged tags.
  • If no price/promo info is extracted, the modality is omitted.

face-count​

What: The number of facings represented by an annotation — primarily for multipack items, where one annotation (e.g. a 2-can multipack) represents more than one facing.

Value: integer as a string.

Example:

{
  "key": "face-count",
  "predictions": [
    { "value": "2", "score": 1.0, "is_main": true }
  ]
}
note
  • For standard SKUs the facing count is the annotation itself (value 1).
  • We can either count facings within an image (default) or extract facings from the packaging size in the catalogue item details ({total in pack}P/{facings long side}) and provide long/short side facings based on the aspect ratio of the annotation. Contact Neurolabs support for this.