Lesson 06: Text, OCR & Key Information Extraction

Not everything you annotate is an object. Receipts, invoices, license plates, street signs, document scans — text annotation is its own discipline, and AnyLabeling supports it natively.

Three Levels of Text Annotation

Text annotation tasks fall into three levels of complexity:

Level 1: Text Detection

Task: Draw a bounding box or polygon around each text region. No transcription.

Use cases: Training text detection models (EAST, DBNet, CRAFT) that find where text is but do not read it.

In AnyLabeling: Use rectangle or polygon tools. Label each region as text or a more specific label like title, body, caption.

Level 2: Text Recognition (OCR)

Task: Draw a region around text AND transcribe the text content.

Use cases: Training end-to-end OCR systems (PaddleOCR, TrOCR, EasyOCR) that both detect and read text.

In AnyLabeling: Draw the text region, then set the label to the transcribed text content. For example, a bounding box around a street sign labeled "Main Street".

Alternatively, use the description field in the annotation properties to separate the class label (street_sign) from the transcription (Main Street).

Level 3: Key Information Extraction (KIE)

Task: Detect text regions, transcribe them, AND assign semantic roles (key-value pairs).

Use cases: Receipt parsing (total: $42.50), form understanding (name: John Doe), invoice processing, ID card reading.

In AnyLabeling: Use group IDs to link related annotations. A key label (Total) and a value label ($42.50) with the same group_id form a key-value pair.

Annotating Documents: A Practical Workflow

Here is how I approach document annotation in AnyLabeling:

Step 1: Define Your Label Schema

Before touching any images, decide on your labels. For a receipt parser:

Labels:
  - store_name
  - date
  - item_name
  - item_price
  - subtotal
  - tax
  - total
  - payment_method

Step 2: Text Detection Pass

Go through each image and draw tight bounding boxes or polygons around every text region. Use rectangles for horizontal text, rotated boxes for angled text, and polygons for curved text.

Speed tip: For documents with consistent layouts (e.g., receipts from the same store), annotate a few examples manually, then consider training a text detection model and using it as a custom auto-labeling model (Lesson 08).

Step 3: Classification and Transcription Pass

Go back through each image and assign the semantic label to each text region. Double-click the label in the Label List to change it from a generic text to the specific role (item_price, total, etc.).

If your downstream task requires transcription, add the text content to the annotation description or use a naming convention like total:$42.50.

Step 4: Link Key-Value Pairs

For KIE tasks, set group_id on related annotations. The label "Total" (key) and "$42.50" (value) should share the same group ID so the model learns to associate them.

Scene Text vs Document Text

The annotation strategy differs based on the text context:

Scene text (street signs, product labels, text in the wild) requires polygon annotations because text appears at arbitrary angles and orientations. Use the rotation tool for text at consistent angles, and polygons for curved or irregular text.

Document text (invoices, forms, scans) is usually horizontal, so rectangles work well. The challenge is not geometry but semantics — assigning the right role to each text region.

Handling Edge Cases

Partially occluded text: Annotate the visible portion. If you can read what the text says, include the full transcription in the label. If you cannot read it, mark it as illegible or skip it.

Multi-line text blocks: Annotate the entire block as one region if the text is semantically one unit (a paragraph, an address). Annotate line by line if each line is a separate entity (line items on a receipt).

Handwritten text: Expect higher annotator disagreement. Establish clear guidelines for letter-by-letter ambiguity (is that an a or an o?). Consider having two annotators per image with adjudication.

Rotated or perspective-distorted text: Use the polygon tool with four vertices matching the text corners. This captures both the region and the orientation, which is critical for rotation-aware OCR models.

Export Considerations for Text Tasks

When exporting text annotations (Lesson 07):

• COCO format preserves polygon coordinates and can include transcription in the attributes field.
• YOLO format only stores bounding box coordinates and class IDs — no transcription. Use COCO or a custom format for OCR tasks.
• PaddleOCR format requires a specific directory structure with label.txt files. You may need a conversion script.

For KIE tasks, the COCO format with custom attributes is usually the best export target, as it preserves both spatial and semantic information.

Key Takeaways

• Text annotation has three levels: detection (where), recognition (what), and KIE (what role).
• Define your label schema before annotating. Changing it mid-project is expensive.
• Use rectangles for document text, polygons/rotated boxes for scene text.
• Group IDs link key-value pairs for structured extraction tasks.
• COCO format is the best export target for text tasks because it preserves rich metadata.

In the next lesson, we cover all export formats in detail and build conversion scripts for training pipelines.