Lesson 05: YOLO Auto-Labeling

SAM is interactive — you click, it segments. YOLO is automatic — it runs on your image and generates all detections at once. These are complementary workflows, and knowing when to use each one is the key to fast annotation.

SAM vs YOLO: When to Use Which

Supported YOLO Models

AnyLabeling includes pre-configured YOLO models:

Models auto-download on first use. The n (nano) variants are best for CPU; l and x variants benefit from GPU.

Activating YOLO Auto-Labeling

1. Click the brain icon or press Ctrl + A.
2. Select a YOLO model from the dropdown (e.g., YOLOv8m).
3. Wait for the model to download and load.
4. The model runs automatically on the current image.
5. All detected objects appear as labeled bounding boxes (or polygons for -seg models).

Unlike SAM, YOLO does not wait for you to click. It processes the image and dumps all detections at once. Your job shifts from "create annotations" to "review and correct."

The YOLO Review Workflow

YOLO will produce three kinds of output on any given image:

1. Correct detections — right class, tight box. Keep them.
2. Incorrect detections — wrong class or bad localization. Fix or delete.
3. Missing detections — objects the model missed. Add manually or with SAM.

Here is the efficient review pattern:

For each image:
  1. YOLO auto-generates all detections
  2. Scan the detections visually
  3. Delete false positives (select + Delete)
  4. Fix wrong labels (double-click label in Label List, retype)
  5. Adjust tight boxes by dragging handles
  6. Add missed objects manually (R for rectangle) or with SAM
  7. Press D for next image

Steps 2-6 take 10-30 seconds per image for typical scenes. Compare to 2-5 minutes for manual annotation from scratch.

Confidence Threshold

YOLO detections come with confidence scores. Lower confidence means more detections (including more false positives). Higher confidence means fewer but more reliable detections.

AnyLabeling uses the model's default confidence threshold. When you load a custom model (Lesson 08), you control this via the confidence_threshold field in your config.yaml:

confidence_threshold: 0.25 # Lower = more detections, more noise
nms_threshold: 0.45 # Non-max suppression: higher = more overlapping boxes

For auto-labeling, I recommend starting at 0.25. It is easier to delete false positives than to find missed objects.

YOLO-Seg for Instance Segmentation

The -seg variants (e.g., YOLOv8s-seg) produce both bounding boxes and segmentation masks:

1. Select YOLOv8s-seg from the model dropdown.
2. The model generates polygon annotations (not just rectangles).
3. Review and adjust the polygons as needed.

The segmentation quality is not as refined as SAM — edges are rougher, especially on small objects. But for bulk labeling where you need approximate masks across thousands of images, YOLO-Seg is dramatically faster than clicking SAM on every instance.

A practical hybrid: Run YOLO-Seg for the initial pass, then use SAM to refine the polygons on the 10-20% of images where the masks are not good enough.

Batch Annotation Strategy

For large datasets (1,000+ images), here is the workflow I recommend:

Phase 1: YOLO Draft (5-10 seconds per image)

Run YOLO on all images. Accept the detections as draft annotations. Do not spend time correcting yet.

Phase 2: Review Pass (15-30 seconds per image)

Go through every image sequentially:

• Delete obvious false positives.
• Fix wrong class labels.
• Flag images that need SAM refinement (add a needs_review flag or move to a separate folder).

Phase 3: SAM Refinement (only on flagged images)

Use SAM on the flagged subset — images where YOLO missed objects or produced poor masks.

Phase 4: Spot Check

Randomly sample 5-10% of images and verify annotation quality. If error rate is above your threshold, do another review pass.

This four-phase approach scales linearly. A single annotator can process 500-1,000 images per day for common object detection tasks.

Combining YOLO and SAM on the Same Image

You can switch between YOLO and SAM models during a session:

1. Start with YOLO to get automatic detections for COCO classes.
2. Switch to SAM (click brain icon, select a SAM model).
3. Use SAM to annotate custom objects that YOLO does not know about.
4. All annotations coexist on the same image.

The label file stores all shapes regardless of how they were created. YOLO-generated boxes and SAM-generated polygons live side by side.

When YOLO Auto-Labeling Falls Short

YOLO auto-labeling works best when your target classes overlap with COCO-80. It struggles in these cases:

• Domain-specific classes (medical lesions, manufacturing defects, custom parts). YOLO has never seen these. Use SAM or a custom model (Lesson 08).
• Dense, overlapping objects (a pile of screws, a crowded shelf). NMS aggressively suppresses overlapping detections.
• Very small objects (distant vehicles in aerial imagery, tiny components on a PCB). YOLO's detection resolution has limits.
• Fine-grained classification (bird species, car make/model). YOLO gives you "bird" or "car," not the subclass.

In all these cases, a custom YOLO model trained on your domain data will perform dramatically better. That is the active learning loop we cover in Lesson 10.

Key Takeaways

• YOLO auto-labeling generates all detections at once — no clicking required.
• Your role shifts from "create annotations" to "review and correct."
• Start with a low confidence threshold (0.25) to catch more objects, then delete false positives.
• YOLO-Seg produces approximate segmentation masks — use SAM to refine the ones that need it.
• For large datasets, use the four-phase workflow: YOLO draft, review pass, SAM refinement, spot check.

In the next lesson, we cover text annotation — OCR detection, recognition, and Key Information Extraction.