Location coordinates

Overview

When extracting information from documents, knowing where the information is located can be as important. This is useful for verification, highlighting the source in the UI, or further processing.

Location granularity

The API provides a location_granularity parameter to control the precision of the coordinate extraction.

Mode	Description
`element`(default)	Returns the coordinates of the entire HTML element (e.g., a table cell, a paragraph) containing the extracted value. This is faster and sufficient for general localization.
`word`	Returns the word-level coordinates of the specific words referenced by the extracted value. This is more granular but may take slightly longer.
`all`	Returns both the element-level and word-level coordinates.

Usage

To get location coordinates, you need to specify location as true and location_granularity in the extra_body of your request. When location is set to true, the response includes tool_calls with the function name additional_values. The arguments of this function call contain the extracted values along with their location coordinates.

Request

For client setup, check simple extraction for more details.

extraction_response = client.chat.completions.create(
    model="information-extract",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:application/octet-stream;base64,{base64_data}"}
                }
            ]
        }
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "document_schema",
            "schema": {
                "type": "object",
                "properties": {
                    "bank_name": {
                        "type": "string",
                        "description": "The name of bank in bank statement"
                    }
                }
            }
        }
    },
    extra_body={ 
        "location": True, 
        "location_granularity": "element"
    } 
)
print(extraction_response)

Response

{
  "id": "iex-AQZoWf2p5j6TO-AE",
  "choices": [
    {
      "finish_reason": "stop",
      "index": null,
      "logprobs": null,
      "message": {
        "content": "{\"bank_name\":\"Bank of Dream\"}",
        "role": "assistant",
        "function_call": null,
        "tool_calls": [
          {
            "type": "function",
            "function": {
              "name": "additional_values",
              "arguments": "{\"bank_name\": {\"_value\": \"Bank of Dream\", \"page\": 1, \"coordinates\": [{\"x\": 0.07, \"y\": 0.148}, {\"x\": 0.2074, \"y\": 0.148}, {\"x\": 0.2074, \"y\": 0.1606}, {\"x\": 0.07, \"y\": 0.1606}]}}"
            }
          }
        ]
      }
    }
  ],
  "created": 1742838017,
  "model": "information-extract-260904",
  "object": null,
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 9,
    "prompt_tokens": 951,
    "total_tokens": 960,
    "completion_tokens_details": null,
    "prompt_tokens_details": null
  }
}

Parsing location coordinates in responses

Users can parse the response to get the location coordinates as follows:

import json

# ... (API request code) ...

if response.choices[0].message.tool_calls:
    parsed_result = json.loads(response.choices[0].message.tool_calls[0].function.arguments)
    print(parsed_result)

The parsed result will be as below according to the location_granularity set in the request.

Element mode (location_granularity="element")

In this mode, the response includes coordinates which represent the bounding box of the element. The coordinates are relative values (0.0 to 1.0) based on the page size.

{
  "key_name": {
    "_value": "extracted value",
    "page": 1,
    "coordinates": [
        {'x': 0.0745, 'y': 0.1005},
        {'x': 0.2096, 'y': 0.1005},
        {'x': 0.2096, 'y': 0.1591},
        {'x': 0.0745, 'y': 0.1591}
    ]
  }
}

Word mode (location_granularity="word")

In this mode, the response includes both coordinates (for the element) and word_coordinates (for individual words).

{
  "key_name": {
    "_value": "extracted value",
    "page": 1,
    "word_coordinates": [
    [{'x': 0.0708, 'y': 0.1475},
        {'x': 0.1168, 'y': 0.1475},
        {'x': 0.1168, 'y': 0.1588},
        {'x': 0.0708, 'y': 0.1588}],
    [{'x': 0.1221, 'y': 0.1475},
        {'x': 0.1469, 'y': 0.1475},
        {'x': 0.1469, 'y': 0.16},
        {'x': 0.1221, 'y': 0.16}],
    [{'x': 0.1522, 'y': 0.1488},
        {'x': 0.2071, 'y': 0.1488},
        {'x': 0.2071, 'y': 0.16},
        {'x': 0.1522, 'y': 0.16}]]
  }
}

All mode (location_granularity="all")

In this mode, the response includes both coordinates (for the element) and word_coordinates (for individual words).

{
  "key_name": {
    "_value": "extracted value",
    "page": 1,
    "coordinates": [
        {'x': 0.0745, 'y': 0.1005},
        {'x': 0.2096, 'y': 0.1005},
        {'x': 0.2096, 'y': 0.1591},
        {'x': 0.0745, 'y': 0.1591}
    ],
    "word_coordinates": [
        [{'x': 0.0708, 'y': 0.1475},
        {'x': 0.1168, 'y': 0.1475},
        {'x': 0.1168, 'y': 0.1588},
        {'x': 0.0708, 'y': 0.1588}],
        [{'x': 0.1221, 'y': 0.1475},
        {'x': 0.1469, 'y': 0.1475},
        {'x': 0.1469, 'y': 0.16},
        {'x': 0.1221, 'y': 0.16}],
        [{'x': 0.1522, 'y': 0.1488},
        {'x': 0.2071, 'y': 0.1488},
        {'x': 0.2071, 'y': 0.16},
        {'x': 0.1522, 'y': 0.16}]]
  }
}

Visualization example

You can visualize the extracted coordinates on the original document image using Python and PIL.

from PIL import Image, ImageDraw
import json

def plot_coordinates(image_path, response):
    # Load image
    img = Image.open(image_path)
    draw = ImageDraw.Draw(img)
    width, height = img.size
    
    # Parse response
    if not response.choices[0].message.tool_calls:
        print("No extraction result found.")
        return
        
    tool_call = response.choices[0].message.tool_calls[0]
    parsed_result = json.loads(tool_call.function.arguments)

    def draw_polygon(coords, color):
        if not coords:
            return
        # Scale coordinates to image size
        scaled_coords = [(x * width, y * height) for x, y in coords]
        # Draw polygon with outline
        draw.polygon(scaled_coords, outline=color, width=3)

    def traverse(data):
        if isinstance(data, dict):
            # Draw if coordinates exist
            if 'coordinates' in data:
                draw_polygon(data['coordinates'], 'red')
            
            if 'word_coordinates' in data:
                for word_poly in data['word_coordinates']:
                    draw_polygon(word_poly, 'blue')
            
            # Recursively traverse all values
            for value in data.values():
                traverse(value)
                
        elif isinstance(data, list):
            for item in data:
                traverse(item)

    traverse(parsed_result)
    
    # Display image
    img.show()

# Example Usage
# image_path = 'document.jpg'
# plot_coordinates(image_path, response)

Location coordinates

Overview

Location granularity

The API provides a location_granularity parameter to control the precision of the coordinate extraction.

Mode	Description
`element`(default)	Returns the coordinates of the entire HTML element (e.g., a table cell, a paragraph) containing the extracted value. This is faster and sufficient for general localization.
`word`	Returns the word-level coordinates of the specific words referenced by the extracted value. This is more granular but may take slightly longer.
`all`	Returns both the element-level and word-level coordinates.

Usage

Request

For client setup, check simple extraction for more details.

extraction_response = client.chat.completions.create(
    model="information-extract",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:application/octet-stream;base64,{base64_data}"}
                }
            ]
        }
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "document_schema",
            "schema": {
                "type": "object",
                "properties": {
                    "bank_name": {
                        "type": "string",
                        "description": "The name of bank in bank statement"
                    }
                }
            }
        }
    },
    extra_body={ 
        "location": True, 
        "location_granularity": "element"
    } 
)
print(extraction_response)

Response

{
  "id": "iex-AQZoWf2p5j6TO-AE",
  "choices": [
    {
      "finish_reason": "stop",
      "index": null,
      "logprobs": null,
      "message": {
        "content": "{\"bank_name\":\"Bank of Dream\"}",
        "role": "assistant",
        "function_call": null,
        "tool_calls": [
          {
            "type": "function",
            "function": {
              "name": "additional_values",
              "arguments": "{\"bank_name\": {\"_value\": \"Bank of Dream\", \"page\": 1, \"coordinates\": [{\"x\": 0.07, \"y\": 0.148}, {\"x\": 0.2074, \"y\": 0.148}, {\"x\": 0.2074, \"y\": 0.1606}, {\"x\": 0.07, \"y\": 0.1606}]}}"
            }
          }
        ]
      }
    }
  ],
  "created": 1742838017,
  "model": "information-extract-260904",
  "object": null,
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 9,
    "prompt_tokens": 951,
    "total_tokens": 960,
    "completion_tokens_details": null,
    "prompt_tokens_details": null
  }
}

Parsing location coordinates in responses

Users can parse the response to get the location coordinates as follows:

import json

# ... (API request code) ...

if response.choices[0].message.tool_calls:
    parsed_result = json.loads(response.choices[0].message.tool_calls[0].function.arguments)
    print(parsed_result)

The parsed result will be as below according to the location_granularity set in the request.

Element mode (location_granularity="element")

In this mode, the response includes coordinates which represent the bounding box of the element. The coordinates are relative values (0.0 to 1.0) based on the page size.

{
  "key_name": {
    "_value": "extracted value",
    "page": 1,
    "coordinates": [
        {'x': 0.0745, 'y': 0.1005},
        {'x': 0.2096, 'y': 0.1005},
        {'x': 0.2096, 'y': 0.1591},
        {'x': 0.0745, 'y': 0.1591}
    ]
  }
}

Word mode (location_granularity="word")

In this mode, the response includes both coordinates (for the element) and word_coordinates (for individual words).

{
  "key_name": {
    "_value": "extracted value",
    "page": 1,
    "word_coordinates": [
    [{'x': 0.0708, 'y': 0.1475},
        {'x': 0.1168, 'y': 0.1475},
        {'x': 0.1168, 'y': 0.1588},
        {'x': 0.0708, 'y': 0.1588}],
    [{'x': 0.1221, 'y': 0.1475},
        {'x': 0.1469, 'y': 0.1475},
        {'x': 0.1469, 'y': 0.16},
        {'x': 0.1221, 'y': 0.16}],
    [{'x': 0.1522, 'y': 0.1488},
        {'x': 0.2071, 'y': 0.1488},
        {'x': 0.2071, 'y': 0.16},
        {'x': 0.1522, 'y': 0.16}]]
  }
}

All mode (location_granularity="all")

In this mode, the response includes both coordinates (for the element) and word_coordinates (for individual words).

{
  "key_name": {
    "_value": "extracted value",
    "page": 1,
    "coordinates": [
        {'x': 0.0745, 'y': 0.1005},
        {'x': 0.2096, 'y': 0.1005},
        {'x': 0.2096, 'y': 0.1591},
        {'x': 0.0745, 'y': 0.1591}
    ],
    "word_coordinates": [
        [{'x': 0.0708, 'y': 0.1475},
        {'x': 0.1168, 'y': 0.1475},
        {'x': 0.1168, 'y': 0.1588},
        {'x': 0.0708, 'y': 0.1588}],
        [{'x': 0.1221, 'y': 0.1475},
        {'x': 0.1469, 'y': 0.1475},
        {'x': 0.1469, 'y': 0.16},
        {'x': 0.1221, 'y': 0.16}],
        [{'x': 0.1522, 'y': 0.1488},
        {'x': 0.2071, 'y': 0.1488},
        {'x': 0.2071, 'y': 0.16},
        {'x': 0.1522, 'y': 0.16}]]
  }
}

Visualization example

You can visualize the extracted coordinates on the original document image using Python and PIL.

from PIL import Image, ImageDraw
import json

def plot_coordinates(image_path, response):
    # Load image
    img = Image.open(image_path)
    draw = ImageDraw.Draw(img)
    width, height = img.size
    
    # Parse response
    if not response.choices[0].message.tool_calls:
        print("No extraction result found.")
        return
        
    tool_call = response.choices[0].message.tool_calls[0]
    parsed_result = json.loads(tool_call.function.arguments)

    def draw_polygon(coords, color):
        if not coords:
            return
        # Scale coordinates to image size
        scaled_coords = [(x * width, y * height) for x, y in coords]
        # Draw polygon with outline
        draw.polygon(scaled_coords, outline=color, width=3)

    def traverse(data):
        if isinstance(data, dict):
            # Draw if coordinates exist
            if 'coordinates' in data:
                draw_polygon(data['coordinates'], 'red')
            
            if 'word_coordinates' in data:
                for word_poly in data['word_coordinates']:
                    draw_polygon(word_poly, 'blue')
            
            # Recursively traverse all values
            for value in data.values():
                traverse(value)
                
        elif isinstance(data, list):
            for item in data:
                traverse(item)

    traverse(parsed_result)
    
    # Display image
    img.show()

# Example Usage
# image_path = 'document.jpg'
# plot_coordinates(image_path, response)

Location coordinates

Overview

Location granularity

Usage

Parsing location coordinates in responses

Visualization example

On this page

Location coordinates

Overview

Location granularity

Usage

Parsing location coordinates in responses

Visualization example

On this page