Confidence

Overview

Confidence in information extraction is a metric that indicates the reliability of the extracted information.

Confidence vs. Accuracy

It is important to distinguish between confidence and accuracy:

• Confidence is subjective to the model. It indicates how sure the model thinks it is.
• Accuracy is objective. It measures whether the extracted data matches the actual ground truth.

Usually, high confidence generally correlates with high accuracy. However, there can be cases where the result is "confident" but "incorrect". Therefore, confidence should be used just as a heuristic for reliability rather than a guarantee of correctness.

Human-in-the-Loop (HITL)

Confidence scores are particularly useful for machine assistedHuman-in-the-Loop (HITL) workflows instead of manually reviewing every extracted document.

• High Confidence (> Threshold): Automatically process the data.
• Low Confidence (< Threshold): Flag the result for human review.

This approach significantly reduces manual effort while maintaining high data quality by focusing human attention where it is most needed.

Usage

To get confidence, you need to specify confidence as true in the extra_body of your request. The response will include tool_calls with the function name additional_values. The arguments of this function call contain the extracted values along with their confidence level (high or low).

Request For client setup, check simple extraction for more details.

extraction_response = client.chat.completions.create(
    model="information-extract",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:application/octet-stream;base64,{base64_data}"}
                }
            ]
        }
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "document_schema",
            "schema": {
                "type": "object",
                "properties": {
                    "bank_name": {
                        "type": "string",
                        "description": "The name of bank in bank statement"
                    }
                }
            }
        }
    },
    extra_body={ 
        "confidence": True
    } 
)
print(extraction_response)

Response

{
  "id": "iex-AQZoWf2p5j6TO-AE",
  "choices": [
    {
      "finish_reason": "stop",
      "index": null,
      "logprobs": null,
      "message": {
        "content": "{\"bank_name\":\"Bank of Dream\"}",
        "role": "assistant",
        "function_call": null,
        "tool_calls": [
          {
            "type": "function",
            "function": {
              "name": "additional_values",
              "arguments": "{\"bank_name\": {\"_value\": \"Bank of Dream\", \"confidence\": \"high\"}}"
            }
          }
        ]
      }
    }
  ],
  "created": 1742838017,
  "model": "information-extract-260904",
  "object": null,
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 9,
    "prompt_tokens": 951,
    "total_tokens": 960,
    "completion_tokens_details": null,
    "prompt_tokens_details": null
  }
}