Find out how to Replicate Uber’s GenAI Bill Processing System?

Handbook knowledge entry from invoices is a sluggish, error-prone process that companies have battled for many years. Not too long ago, Uber Engineering revealed how they tackled this problem with their “TextSense” platform, a complicated system for GenAI bill processing. This method showcases the ability of clever doc processing, combining Optical Character Recognition (OCR) with Massive Language Fashions (LLMs) for extremely correct, automated knowledge extraction. This superior method might sound out of attain for smaller tasks. Nevertheless, the core rules at the moment are accessible to everybody. This information will present you the way to replicate the elemental workflow of Uber’s system. We are going to use easy, highly effective instruments to create a system that automates bill knowledge extraction.

Understanding Uber’s “TextSense” System

Earlier than we construct our model, it’s useful to grasp what impressed it. Uber’s purpose was to automate the processing of thousands and thousands of paperwork, from invoices to receipts. Their “TextSense” platform, detailed of their engineering weblog, is a sturdy, multi-stage pipeline designed for this objective.

The determine exhibits the total doc processing pipeline. For processing any doc, pre-processing is often frequent earlier than calling an LLM.

At its core, the system works in three primary levels:

1. Digitization (through OCR): First, the system takes a doc, like a PDF or a picture of an bill. It makes use of a sophisticated OCR engine to “learn” the doc and convert all of the visible textual content into machine-readable textual content. This uncooked textual content is the inspiration for the following step.
2. Clever Extraction (through LLM): The uncooked textual content from the OCR course of is usually messy and unstructured. That is the place the GenAI magic occurs. Uber feeds this textual content to a big language mannequin. The LLM acts like an knowledgeable who understands the context of an bill. It may determine and extract particular items of data, such because the “Bill Quantity,” “Whole Quantity,” and “Provider Identify,” and arrange them right into a structured format, like JSON.
3. Verification (Human-in-the-Loop): No AI is ideal. To make sure 100% accuracy, Uber carried out a human-in-the-loop AI system. This verification step presents the unique doc alongside the AI-extracted knowledge to a human operator. The operator can shortly verify that the info is appropriate or make minor changes if wanted. This suggestions loop additionally helps enhance the mannequin over time.

This mix of OCR with AI and human oversight makes their system each environment friendly and dependable. The next determine explains the workflow of TextSense in an in depth method, as defined within the above factors.

Our Sport Plan: Replicating the Core Workflow

Our purpose is to not rebuild Uber’s complete production-grade platform. As a substitute, we are going to replicate its core intelligence in a simplified, accessible approach. We are going to construct our GenAI bill processing POC in a single Google Colab pocket book.

Our plan follows the identical logical steps:

1. Ingest Doc: We are going to create a easy method to add a PDF bill on to our pocket book.
2. Carry out OCR: We are going to use Tesseract, a robust open-source OCR engine, to extract all of the textual content from the uploaded bill.
3. Extract Entities with AI: We are going to use the Google Gemini API to carry out the automated knowledge extraction. We’ll craft a particular immediate to instruct the mannequin to tug out the important thing fields we’d like.
4. Create a Verification UI: We are going to construct a easy interactive interface utilizing ipywidgets to function our human-in-the-loop AI system, permitting for fast validation of the extracted knowledge.

This method offers us a robust and cheap method to obtain clever doc processing with no need advanced infrastructure.

Arms-On Implementation: Constructing the POC Step-by-Step

Let’s start constructing our system. You may comply with these steps in a brand new Google Colab pocket book.

Step 1: Setting Up the Atmosphere

First, we have to set up the required Python libraries. This command installs packages for dealing with PDFs (PyMuPDF), working OCR (pytesseract), interacting with the Gemini API, and constructing the UI (ipywidgets). It additionally installs the Tesseract OCR engine itself.

!pip set up -q -U google-generativeai PyMuPDF pytesseract pandas ipywidgets

!apt-get -qq set up tesseract-ocr

Step 2: Configuring the Google Gemini API

Subsequent, you might want to configure your Gemini API key. To maintain your key secure, we’ll use Colab’s built-in secret supervisor.

1. Get your API key from Google AI Studio.
2. In your Colab pocket book, click on the important thing icon on the left sidebar.
3. Create a brand new secret named GEMINI_API_KEY and paste your key as the worth.

The next code will securely entry your key and configure the API.

import google.generativeai as genai

from google.colab import userdata

import fitz  # PyMuPDF

import pytesseract

from PIL import Picture

import pandas as pd

import ipywidgets as widgets

from ipywidgets import Structure

from IPython.show import show, clear_output

import json

import io

# Configure the Gemini API

attempt:

   api_key = userdata.get(“GEMINI_API_KEY”)

   genai.configure(api_key=api_key)

   print("Gemini API configured efficiently.")

besides userdata.SecretNotFoundError:

   print("ERROR: Secret 'GEMINI_API_KEY' not discovered. Please comply with the directions to set it up.")

Step 3: Importing and Pre-processing the PDF

This code uploads a PDF file that’s an bill PDF. While you add a PDF, it converts every web page right into a high-resolution picture, which is the perfect format for OCR.

import fitz  # PyMuPDF

from PIL import Picture

import io

import os

invoice_images = []

uploaded_file_name = "/content material/sample-invoice.pdf"  # Substitute with the precise path to your PDF file

# Make sure the file exists (non-obligatory however really useful)

if not os.path.exists(uploaded_file_name):

   print(f"ERROR: File not discovered at '{uploaded_file_name}'. Please replace the file path.")

else:

   print(f"Processing '{uploaded_file_name}'...")

   # Convert PDF to photographs

   doc = fitz.open(uploaded_file_name)

   for page_num in vary(len(doc)):

       web page = doc.load_page(page_num)

       pix = web page.get_pixmap(dpi=300) # Increased DPI for higher OCR

       img = Picture.open(io.BytesIO(pix.tobytes()))

       invoice_images.append(img)

   doc.shut()

   print(f"Efficiently transformed {len(invoice_images)} web page(s) to photographs.")

   # Show the primary web page as a preview

   if invoice_images:

       print("n--- Bill Preview (First Web page) ---")

       show(invoice_images[0].resize((600, 800)))

Output:

Now, we run the OCR course of on the pictures we simply created. The textual content from all pages is mixed right into a single string. That is the context we are going to ship to the Gemini mannequin. This step is a vital a part of the OCR with AI workflow.

full_invoice_text = ""

if not invoice_images:

   print("Please add a PDF bill within the step above first.")

else:

   print("Extracting textual content with OCR...")

   for i, img in enumerate(invoice_images):

       textual content = pytesseract.image_to_string(img)

       full_invoice_text += f"n--- Web page {i+1} ---n{textual content}"

   print("OCR extraction full.")

   print("n--- Extracted Textual content (first 500 characters) ---")

   print(full_invoice_text[:500] + "...")

Output:

That is the place the GenAI bill processing occurs. We create an in depth immediate that tells the Gemini mannequin its position. We instruct it to extract particular fields and return the lead to a clear JSON format. Asking for JSON is a robust approach that makes the mannequin’s output structured and straightforward to work with.

extracted_data = {}

if not full_invoice_text.strip():

   print("Can't proceed. The extracted textual content is empty. Please test the PDF high quality.")

else:

   # Instantiate the Gemini Professional mannequin

   mannequin = genai.GenerativeModel('gemini-2.5-pro')

   # Outline the fields you wish to extract

   fields_to_extract = "Bill Quantity, Bill Date, Due Date, Provider Identify, Provider Tackle, Buyer Identify, Buyer Tackle, Whole Quantity, Tax Quantity"

   # Create the detailed immediate

   immediate = f"""

   You might be an knowledgeable in bill knowledge extraction.

   Your process is to investigate the supplied OCR textual content from an bill and extract the next fields: {fields_to_extract}.

   Comply with these guidelines strictly:

   1.  Return the output as a single, clear JSON object.

   2.  The keys of the JSON object should be precisely the sphere names supplied.

   3.  If a area can't be discovered within the textual content, its worth within the JSON ought to be `null`.

   4.  Don't embody any explanatory textual content, feedback, or markdown formatting (like ```json) in your response. Solely the JSON object is allowed.

   Right here is the bill textual content:

   ---

   {full_invoice_text}

   ---

   """

   print("Sending request to Gemini API...")

   attempt:

       # Name the API

       response = mannequin.generate_content(immediate)

       # Robustly parse the JSON response

       response_text = response.textual content.strip()

       # Clear potential markdown formatting

       if response_text.startswith('```json'):

           response_text = response_text[7:-3].strip()

       extracted_data = json.hundreds(response_text)

       print("n--- AI Extracted Knowledge (JSON) ---")

       print(json.dumps(extracted_data, indent=2))

   besides json.JSONDecodeError:

       print("n--- ERROR ---")

       print("Didn't decode the mannequin's response into JSON.")

       print("Mannequin's Uncooked Response:", response.textual content)

   besides Exception as e:

       print(f"nAn surprising error occurred: {e}")

       print("Mannequin's Uncooked Response (if accessible):", getattr(response, 'textual content', 'N/A'))

Output:

Step 6: Constructing the Human-in-the-Loop (HITL) UI

Lastly, we constructed the verification interface. This code shows the bill picture on the left and creates an editable kind on the proper, pre-filled with the info from Gemini. The person can shortly assessment the data, make any obligatory edits, and make sure.

# UI Widgets

text_widgets = {}

if not extracted_data:

   print("No knowledge was extracted by the AI. Can't construct verification UI.")

else:

   form_items = []

   # Create a textual content widget for every extracted area

   for key, worth in extracted_data.objects():

       text_widgets[key] = widgets.Textual content(

           worth=str(worth) if worth is just not None else "",

           description=key.substitute('_', ' ').title() + ':',

           fashion={'description_width': 'preliminary'},

           structure=Structure(width="95%")

       )

       form_items.append(text_widgets[key])

   # The shape container

   kind = widgets.VBox(form_items, structure=Structure(padding='10px'))

   # Picture container

   if invoice_images:

       img_byte_arr = io.BytesIO()

       invoice_images[0].save(img_byte_arr, format="PNG")

       image_widget = widgets.Picture(

           worth=img_byte_arr.getvalue(),

           format="png",

           width=500

       )

       image_box = widgets.HBox([image_widget], structure=Structure(justify_content="middle"))

   else:

       image_box = widgets.HTML("No picture to show.")

   # Affirmation button

   confirm_button = widgets.Button(description="Verify and Save", button_style="success")

   output_area = widgets.Output()

   def on_confirm_button_clicked(b):

       with output_area:

           clear_output()

           final_data = {key: widget.worth for key, widget in text_widgets.objects()}

           # Create a pandas DataFrame

           df = pd.DataFrame([final_data])

           df['Source File'] = uploaded_file_name

           print("--- Verified and Finalized Knowledge ---")

           show(df)

           # Now you can save this DataFrame to CSV, and many others.

           df.to_csv('verified_invoice.csv', index=False)

           print("nData saved to 'verified_invoice.csv'")

   confirm_button.on_click(on_confirm_button_clicked)

   # Remaining UI Structure

   ui = widgets.HBox([

       widgets.VBox([widgets.HTML("Invoice Image"), image_box]),

       widgets.VBox([

           widgets.HTML("Verify Extracted Data"),

           form,

           widgets.HBox([confirm_button], structure=Structure(justify_content="flex-end")),

           output_area

       ], structure=Structure(flex='1'))

   ])

   print("--- Human-in-the-Loop (HITL) Verification ---")

   print("Evaluation the info on the proper. Make any corrections and click on 'Verify and Save'.")

   show(ui)

Output:

Modify some values after which save.

Output:

You may entry the total code right here: GitHub, Colab

Conclusion

This POC efficiently demonstrates that the core logic behind a complicated system like Uber’s “TextSense” is replicable. By combining open-source OCR with a robust LLM like Google’s Gemini, you may construct an efficient system for GenAI bill processing. This method to clever doc processing dramatically reduces guide effort and improves accuracy. The addition of a easy human-in-the-loop AI interface ensures that the ultimate knowledge is reliable.

Be at liberty to increase on this basis by including extra fields, bettering validation, and integrating it into bigger workflows.

Continuously Requested Questions