Lesson 10: Active Learning Pipelines

The single most powerful workflow in applied ML is not a fancy architecture or a larger dataset. It is a tight feedback loop between labeling and training. The concept is well-established in the ML literature — Settles (2009) provides the foundational survey.

Most teams treat annotation as a one-shot process: label everything, train once, deploy. Active learning flips this into an iterative cycle where each round of training tells you exactly which images to label next. The result is a better model with fewer total annotations.

The Active Learning Loop

The cycle has five stages that repeat:

After stage 5, loop back to stage 2 with the expanded dataset. Each cycle produces a stronger model that requires fewer corrections in the next cycle.

Each iteration:

1. Train a model on your current labeled data.
2. Run that model on your unlabeled data.
3. Select the images where the model is least confident.
4. Label those images (using the model's predictions as drafts in AnyLabeling).
5. Add them to the training set and retrain.

Why This Works

Labeling random images is wasteful. In a typical dataset, 60-70% of images are "easy" — the model would get them right after seeing just a few examples. Labeling more easy images adds redundant information.

The hard images — the ones where the model is uncertain — are where the decision boundary lives. Labeling those images gives the model the information it needs most. Research consistently shows that active learning reaches a given accuracy level with 2-5x fewer labeled images than random sampling.

Step-by-Step: Your First Active Learning Pipeline

Step 0: Collect Unlabeled Data

Gather all your images into a single directory. The more unlabeled data you have, the better active learning works, because you have more candidates to select from.

project/
├── unlabeled/          # All your images (1,000 - 100,000+)
├── labeled/            # Starts empty
│   ├── images/
│   └── labels/
├── models/             # Trained model checkpoints
└── scripts/            # Active learning scripts

Step 1: Label the Seed Dataset

Manually annotate 50-200 images in AnyLabeling. This is the only round where you start from scratch. Choose images that represent the diversity of your data — different lighting, angles, object sizes, backgrounds.

Export to YOLO format:

labeled/
├── images/
│   ├── img_001.jpg
│   └── ...
├── labels/
│   ├── img_001.txt
│   └── ...
└── classes.txt

Step 2: Train the First Model

# Create dataset.yaml
cat > dataset.yaml << 'EOF'
train: labeled/images
val: labeled/images    # Use train as val for seed round (small dataset)
names:
  0: class_a
  1: class_b
  2: class_c
EOF

# Train YOLOv8
yolo detect train data=dataset.yaml model=yolov8s.pt epochs=100 imgsz=640 patience=20

The first model will not be good. That is expected. It just needs to be better than random.

Step 3: Predict on Unlabeled Data

Run the model on all unlabeled images and save the predictions with confidence scores:

from ultralytics import YOLO
from pathlib import Path
import json

model = YOLO("runs/detect/train/weights/best.pt")
unlabeled_dir = Path("unlabeled")
results_file = Path("predictions.json")

predictions = {}
for img_path in sorted(unlabeled_dir.glob("*")):
    if img_path.suffix.lower() not in (".jpg", ".jpeg", ".png", ".bmp"):
        continue

    results = model(str(img_path), verbose=False)
    detections = []
    for box in results[0].boxes:
        detections.append({
            "class": int(box.cls),
            "confidence": float(box.conf),
            "bbox": box.xywhn.tolist()[0],  # normalized center format
        })
    predictions[img_path.name] = {
        "detections": detections,
        "max_confidence": max((d["confidence"] for d in detections), default=0),
        "min_confidence": min((d["confidence"] for d in detections), default=1),
        "num_detections": len(detections),
    }

results_file.write_text(json.dumps(predictions, indent=2))
print(f"Predictions saved for {len(predictions)} images")

Step 4: Select Hard Examples

This is the core of active learning. Select images where the model struggles:

import json
from pathlib import Path

with open("predictions.json") as f:
    predictions = json.load(f)

# Strategy 1: Lowest confidence — images where the model is least sure
by_min_conf = sorted(
    predictions.items(),
    key=lambda x: x[1]["min_confidence"]
)

# Strategy 2: Most uncertain — detections near the decision boundary (0.4-0.6)
def uncertainty_score(pred):
    scores = [d["confidence"] for d in pred["detections"]]
    if not scores:
        return 1.0  # No detections = high uncertainty
    return sum(abs(s - 0.5) for s in scores) / len(scores)

by_uncertainty = sorted(
    predictions.items(),
    key=lambda x: uncertainty_score(x[1])
)

# Strategy 3: Diversity — images with unusual detection patterns
# (e.g., images with 0 detections when most have 3+, or vice versa)
avg_detections = sum(p["num_detections"] for p in predictions.values()) / len(predictions)
by_diversity = sorted(
    predictions.items(),
    key=lambda x: abs(x[1]["num_detections"] - avg_detections),
    reverse=True
)

# Take top 50 from each strategy, deduplicate
selected = set()
for strategy in [by_min_conf, by_uncertainty, by_diversity]:
    for name, _ in strategy:
        if len(selected) >= 100:
            break
        selected.add(name)

print(f"Selected {len(selected)} images for labeling")

# Copy selected images to a labeling directory
import shutil
label_dir = Path("to_label_round_2")
label_dir.mkdir(exist_ok=True)
for name in selected:
    shutil.copy2(Path("unlabeled") / name, label_dir / name)

Step 5: Label in AnyLabeling with Model Assistance

Now the loop closes:

1. Export the current model to ONNX:

   yolo export model=runs/detect/train/weights/best.pt format=onnx simplify=True
   

2. Create a config.yaml for AnyLabeling (as in Lesson 08).

3. Open the to_label_round_2/ directory in AnyLabeling.

4. Load your custom model as the auto-labeling backend.

5. The model generates draft annotations. You review and correct.

Because these are the hard images, expect more corrections than on easy images. That is the point — you are teaching the model exactly where it is confused.

Step 6: Merge and Retrain

Add the newly labeled images to your training set and retrain:

# Move labeled images and labels to the main labeled directory
cp to_label_round_2/*.jpg labeled/images/
cp to_label_round_2/labels/*.txt labeled/labels/

# Retrain with the expanded dataset
yolo detect train data=dataset.yaml model=yolov8s.pt epochs=100 imgsz=640 patience=20

Step 7: Evaluate and Decide

After retraining, evaluate on a held-out test set:

from ultralytics import YOLO

model = YOLO("runs/detect/train2/weights/best.pt")
metrics = model.val(data="dataset.yaml")
print(f"mAP@50: {metrics.box.map50:.3f}")
print(f"mAP@50-95: {metrics.box.map:.3f}")

If the model meets your accuracy target, stop. If not, go back to Step 3.

When to Stop

Active learning has diminishing returns. Here are the signals that you are done:

1. Accuracy plateau — mAP improves by less than 1% between rounds.
2. High confidence everywhere — the model is confident on nearly all unlabeled images, leaving few hard examples to select.
3. Annotation corrections decrease — in the latest round, you barely changed the model's predictions. It is already labeling accurately.
4. Business threshold met — your model hits the accuracy target for deployment.

Most projects reach good performance in 3-5 active learning rounds with 500-2,000 total labeled images. Compare that to labeling 10,000+ images upfront with random selection.

Sampling Strategy Comparison

In practice, mixing strategies (as shown in Step 4) works best. Pure lowest-confidence sampling can over-focus on a single failure mode.

Common Mistakes

Labeling too many images per round. 50-200 images per round is optimal. Labeling 1,000 images before retraining wastes the feedback loop — you are labeling images that the model might already handle after seeing the first 100.

Not using the model for auto-labeling. Each round should be faster than the last, because the model's draft predictions improve. If you are still labeling from scratch in round 3, you are leaving speed on the table.

Ignoring the test set. Active learning selects biased training data (hard examples). Always evaluate on a random, held-out test set to check that the model generalizes, not just that it gets better on hard cases.

Stopping after one round. The first model is weak. The second is tolerable. The third is usually production-ready. Commit to at least 3 rounds before judging whether the approach works.

Key Takeaways

• Active learning reaches target accuracy with 2-5x fewer labeled images than random sampling.
• The cycle is: label seed data, train, predict on unlabeled, select hard examples, label with model assistance, retrain.
• Mix sampling strategies: lowest confidence + uncertainty + diversity.
• 50-200 images per round, 3-5 rounds total is typical for most projects.
• Each round should be faster than the last because the model improves as an auto-labeling backend.

In the final lesson, we scale this beyond a single annotator — team workflows, quality assurance, and dataset versioning.