Chart Recognition

Although charts within documents contain useful information, they are normally parsed as images, making it difficult to utilize them for tasks such as search, QA, or RAG. Extracting numerical data from charts into tables allows seamless integration into existing text-based pipelines, enabling the effective utilization of chart information. While one could use a multi-modal LLM to address this issue, it tends to be more expensive and leads to a more complex pipeline compared to text-based LLMs.

With the chart recognition feature in Upstage Document Parse, the following chart types are recognized and converted into a table format.

Bar charts
Line charts
Pie charts

The following is an example.

HTML output

<figure id='2' data-category='chart'>
  <img data-coord="top-left:(118,157); bottom-right:(900,393)" />
  <figcaption>
    <p>Chart Title: FMD VACCINATION COVERAGE IN SELECTED ZONE 2 EXTENTION AREAS DURING THE OCTOBER 2014 TUBU FMD OUTBREAK</p>
    <p>X-Axis: EXTENTION AREAS</p>
    <p>Y-Axis: Cumulative % Coverage</p>
    <p>Chart Type: bar</p>
  </figcaption>
  <table>
    <tr>
      <th></th>
      <th>TUBU</th>
      <th>NOKANENG</th>
      <th>HABU</th>
      <th>ETSHA 6</th>
      <th>GUMARE</th>
      <th>TSAU</th>
    </tr>
    <tr>
      <td>item_01</td>
      <td>79%</td>
      <td>90%</td>
      <td>71%</td>
      <td>69%</td>
      <td>77%</td>
      <td>37%</td>
    </tr>
  </table>
</figure>

<img>: Coordination information for the chart
<figcaption>: Chart metadata, with each item separated by a <p> tag
1. Chart title (Optional): The title of the chart
2. X-axis (Optional): The x-axis label
3. Y-axis (Optional): The y-axis label
4. Chart type: Identified as bar, line, or pie chart
<table>: The recognized chart data in table format

Under the hood, confidence scores for chart recognition are calculated for each chart. If chart recognition is successful according to the confidence score, the chart is categorized as a chart and the resulting table is displayed in HTML. If recognition fails, the chart is categorized as a figure and the response contains HTML, Markdown and text.

If chart recognition succeeds → "Chart"

{
  "category": "chart",
  "content": {
    "html": "<figure id='4' data-category='chart'><img data-coord=\"top-left:(175,1305); bottom-right:(785,1784)\" /><figcaption><p>Chart Type: bar</p></figcaption><table><tr><th></th><th>Nuclear</th><th>Renewables</th><th>Hydro</th><th>Natural gas</th><th>Coal</th><th>Oil</th></tr><tr><td>item_01</td><td>4%</td><td>4%</td><td>7%</td><td>24%</td><td>27%</td><td>34%</td></tr></table></figure>",
    "markdown": "- Chart Type: bar\n|  | Nuclear | Renewables | Hydro | Natural gas | Coal | Oil |\n| --- | --- | --- | --- | --- | --- | --- |\n| item_01 | 4% | 4% | 7% | 24% | 27% | 34% |\n",
    "text": "Chart Type: bar \nNuclear Renewables HydroNatural gas Coal Oil\n item_01 4% 4% 7% 24% 27% 34%"
  },
  "id": 4,
  "page": 1
}

html: <figcaption> and <table> elements are added inside the <figure data-category='chart'> tag
markdown: The result in Markdown format
text: The result in raw text

If chart recognition fails → "Figure"

{
  "category": "figure",
  "content": {
    "html": "<figure id='4' data-category='chart'><img style='font-size:20px' alt=\"Nuclear 4%\nRenewables 4%\nHydro 7%\nNatural gas 24%\nCoal 27%\nOil 34%\" data-coord=\"top-left:(175,1306); bottom-right:(785,1783)\" /></figure>",
    "markdown": "Nuclear 4%\nRenewables 4%\nHydro 7%\nNatural gas 24%\nCoal 27%\nOil 34%",
    "text": "Nuclear 4%\nRenewables 4%\nHydro 7%\nNatural gas 24%\nCoal 27%\nOil 34%"
  },
  "id": 4,
  "page": 1
}

html: The OCR result is added to the alt attribute of the <img> tag within <figure data-category='chart'>
markdown: The OCR result
text: The OCR result

Chart Recognition

With the chart recognition feature in Upstage Document Parse, the following chart types are recognized and converted into a table format.

Bar charts
Line charts
Pie charts

The following is an example.

HTML output

<figure id='2' data-category='chart'>
  <img data-coord="top-left:(118,157); bottom-right:(900,393)" />
  <figcaption>
    <p>Chart Title: FMD VACCINATION COVERAGE IN SELECTED ZONE 2 EXTENTION AREAS DURING THE OCTOBER 2014 TUBU FMD OUTBREAK</p>
    <p>X-Axis: EXTENTION AREAS</p>
    <p>Y-Axis: Cumulative % Coverage</p>
    <p>Chart Type: bar</p>
  </figcaption>
  <table>
    <tr>
      <th></th>
      <th>TUBU</th>
      <th>NOKANENG</th>
      <th>HABU</th>
      <th>ETSHA 6</th>
      <th>GUMARE</th>
      <th>TSAU</th>
    </tr>
    <tr>
      <td>item_01</td>
      <td>79%</td>
      <td>90%</td>
      <td>71%</td>
      <td>69%</td>
      <td>77%</td>
      <td>37%</td>
    </tr>
  </table>
</figure>

<img>: Coordination information for the chart
<figcaption>: Chart metadata, with each item separated by a <p> tag
1. Chart title (Optional): The title of the chart
2. X-axis (Optional): The x-axis label
3. Y-axis (Optional): The y-axis label
4. Chart type: Identified as bar, line, or pie chart
<table>: The recognized chart data in table format

If chart recognition succeeds → "Chart"

{
  "category": "chart",
  "content": {
    "html": "<figure id='4' data-category='chart'><img data-coord=\"top-left:(175,1305); bottom-right:(785,1784)\" /><figcaption><p>Chart Type: bar</p></figcaption><table><tr><th></th><th>Nuclear</th><th>Renewables</th><th>Hydro</th><th>Natural gas</th><th>Coal</th><th>Oil</th></tr><tr><td>item_01</td><td>4%</td><td>4%</td><td>7%</td><td>24%</td><td>27%</td><td>34%</td></tr></table></figure>",
    "markdown": "- Chart Type: bar\n|  | Nuclear | Renewables | Hydro | Natural gas | Coal | Oil |\n| --- | --- | --- | --- | --- | --- | --- |\n| item_01 | 4% | 4% | 7% | 24% | 27% | 34% |\n",
    "text": "Chart Type: bar \nNuclear Renewables HydroNatural gas Coal Oil\n item_01 4% 4% 7% 24% 27% 34%"
  },
  "id": 4,
  "page": 1
}

html: <figcaption> and <table> elements are added inside the <figure data-category='chart'> tag
markdown: The result in Markdown format
text: The result in raw text

If chart recognition fails → "Figure"

{
  "category": "figure",
  "content": {
    "html": "<figure id='4' data-category='chart'><img style='font-size:20px' alt=\"Nuclear 4%\nRenewables 4%\nHydro 7%\nNatural gas 24%\nCoal 27%\nOil 34%\" data-coord=\"top-left:(175,1306); bottom-right:(785,1783)\" /></figure>",
    "markdown": "Nuclear 4%\nRenewables 4%\nHydro 7%\nNatural gas 24%\nCoal 27%\nOil 34%",
    "text": "Nuclear 4%\nRenewables 4%\nHydro 7%\nNatural gas 24%\nCoal 27%\nOil 34%"
  },
  "id": 4,
  "page": 1
}

html: The OCR result is added to the alt attribute of the <img> tag within <figure data-category='chart'>
markdown: The OCR result
text: The OCR result

Chart Recognition