Lesson 09: Writing Annotation Guidelines

You can have the best annotation tool in the world and still produce a terrible dataset if your guidelines are vague. I have seen teams spend months annotating data, only to discover that different annotators interpreted the task differently. Half the dataset was internally inconsistent, and the model trained on it learned nothing useful.

Annotation guidelines are the specification for your dataset. They deserve the same rigor you give to API specifications or database schemas.

Why Guidelines Matter

Without clear guidelines, annotators face ambiguous decisions hundreds of times per session:

• Is a partially visible car still a "car"? What if only the bumper is visible?
• Should I draw the bounding box tight to the object, or include a small margin?
• Is a motorcycle with a sidecar one object or two?
• If two objects overlap, where does one annotation end and the other begin?

Every ambiguous decision that is resolved differently by different annotators introduces noise into your dataset. That noise directly degrades model performance.

Research from Paullada et al. (2021) and Northcutt et al. (2021) consistently shows that label noise is one of the top factors limiting model accuracy on real-world tasks.

Anatomy of Good Annotation Guidelines

A complete annotation guideline document has five sections:

1. Task Definition

What is the annotator doing, and why?

Task: Object detection for autonomous driving perception.

Goal: Detect all vehicles, pedestrians, and cyclists visible in
dashboard camera images. These annotations will train a YOLOv8
model deployed on an embedded system in the vehicle.

Annotation type: Axis-aligned bounding boxes.
Format: YOLO.

Two sentences of context about why the data exists changes how annotators approach edge cases. An annotator who knows this is for autonomous driving will draw tighter boxes than one who thinks it is for a generic image search engine.

2. Class Definitions

Define every class with:

• A one-line definition
• What IS this class (positive examples)
• What is NOT this class (negative examples / common confusions)
• Visual examples

## car
A four-wheeled motor vehicle designed for passenger transport.

**Includes:**
- Sedans, SUVs, hatchbacks, station wagons
- Taxis and ride-share vehicles
- Parked cars with no occupants
- Partially occluded cars (if more than 25% is visible)

**Excludes:**
- Trucks and vans (see: truck)
- Motorcycles (see: motorcycle)
- Toy cars, cars on billboards or screens
- Cars where less than 25% is visible

**Examples:**
[Include 4-6 annotated images showing typical cases and edge cases]

The "excludes" section is critical. It handles the boundary between classes that annotators will inevitably question.

3. Annotation Rules

Precise instructions for how to draw annotations:

## Bounding Box Rules

1. TIGHT BOXES: The box should touch the object on all four sides.
   Do not include padding. Do not leave space between the box edge
   and the object boundary.

2. OCCLUSION: If an object is partially occluded by another object:
   - If >25% of the object is visible: annotate the VISIBLE portion only.
     Do not extend the box to where you think the full object would be.
   - If less than 25% is visible: do not annotate.

3. TRUNCATION: If an object extends beyond the image boundary:
   - Annotate the visible portion. The box edge should touch the image
     edge where the object is truncated.

4. OVERLAP: If two bounding boxes overlap, that is fine. Each object
   gets its own box regardless of overlap.

5. MINIMUM SIZE: Do not annotate objects smaller than 20x20 pixels.
   These are too small for the model to learn from.

6. CROWD SCENES: In dense crowds where individual objects cannot be
   distinguished, annotate the clearly distinguishable individuals only.
   Do not attempt to separate every person in a dense cluster.

4. Edge Cases

The section that prevents 80% of annotation errors. Document every edge case you can think of, with a clear ruling:

## Edge Cases

| Case | Ruling | Reason |
|------|--------|--------|
| Car on a tow truck | Annotate the tow truck only | The car is cargo, not a road participant |
| Car reflected in a window | Do not annotate | Reflections are not real objects |
| Car in a photograph held by a person | Do not annotate | Images within images are not real objects |
| Emergency vehicle (ambulance, fire truck) | Annotate as "truck" | Matches our class taxonomy |
| Golf cart | Do not annotate | Not a road vehicle in our taxonomy |
| Person on a bicycle | Annotate as "cyclist" (one box around both) | Person and bike are one unit |
| Person walking next to a bicycle | Annotate person as "pedestrian", bike as "bicycle" | They are separate when not riding |

You will not think of every edge case in advance. That is fine. Keep the edge case table as a living document and add cases as annotators encounter them.

5. Quality Examples

Include annotated reference images that show correct annotations. Annotators should be able to compare their work against these references.

Include both "good" examples and "bad" examples with explanations of what is wrong:

## Good Example
[Image with correct, tight bounding boxes]
- All visible vehicles annotated
- Boxes are tight to object boundaries
- Partially occluded car correctly annotated (visible portion only)

## Bad Example — Common Mistakes
[Image with incorrect annotations]
- Box around the car has too much padding (10+ pixels on each side)
- Reflection in the shop window incorrectly annotated
- Small car in the distance (15x12 pixels) incorrectly annotated (below minimum size)

Measuring Annotation Quality: Inter-Annotator Agreement

If multiple people annotate the same data, how consistent are they? This is measured by Inter-Annotator Agreement (IAA).

IoU Agreement for Bounding Boxes

Have two annotators independently label the same set of 50-100 images. Then compute the mean IoU (Intersection over Union) between their annotations for matching objects:

def compute_iou(box_a, box_b):
    """Both boxes in [x1, y1, x2, y2] format."""
    x1 = max(box_a[0], box_b[0])
    y1 = max(box_a[1], box_b[1])
    x2 = min(box_a[2], box_b[2])
    y2 = min(box_a[3], box_b[3])

    intersection = max(0, x2 - x1) * max(0, y2 - y1)
    area_a = (box_a[2] - box_a[0]) * (box_a[3] - box_a[1])
    area_b = (box_b[2] - box_b[0]) * (box_b[3] - box_b[1])
    union = area_a + area_b - intersection

    return intersection / union if union > 0 else 0

Target: Mean IoU > 0.75 for bounding boxes. If your annotators consistently disagree (IoU < 0.6), your guidelines have ambiguity that needs to be resolved.

Cohen's Kappa for Classification

For class label agreement (did both annotators call it "car" vs "truck"?), use Cohen's Kappa:

from sklearn.metrics import cohen_kappa_score

# labels_a and labels_b are lists of class labels for the same objects
kappa = cohen_kappa_score(labels_a, labels_b)
# kappa > 0.8 = excellent, 0.6-0.8 = good, < 0.6 = guidelines need work

The Guidelines Iteration Cycle

1. Write initial guidelines based on your task understanding.
2. Pilot annotate 50 images with 2-3 annotators.
3. Measure IAA. If below threshold, identify the disagreements.
4. Update guidelines to resolve ambiguity. Add edge cases.
5. Re-pilot with the updated guidelines.
6. Repeat until IAA meets your target.

This cycle typically takes 2-3 iterations. It feels slow at the start, but it saves enormous time later because your dataset is consistent from the beginning.

Key Takeaways

• Annotation guidelines are the specification for your dataset. Vague guidelines produce noisy data.
• Define each class with positive examples, negative examples, and boundary cases.
• Document edge cases as a living table. Update it as annotators encounter new ambiguities.
• Measure inter-annotator agreement (IoU > 0.75, Kappa > 0.8) before scaling annotation.
• Invest 2-3 iterations in guidelines refinement. It pays for itself within the first 500 images.

In the next lesson, we close the loop: training a model on your annotations, using it to auto-label more data, and iterating.