Lesson 11: Scaling Annotation for Teams

A single annotator can label 500-1,000 images per day with AnyLabeling and AI assistance. That is enough for personal projects and small experiments. But production ML often requires 10,000-100,000+ labeled images, which means a team.

Team annotation introduces problems that solo annotation does not: inconsistency between annotators, quality variance, version conflicts, and the organizational overhead of coordinating who labels what. This lesson covers the patterns that make it work.

Splitting Work Across Annotators

Directory-Based Partitioning

The simplest approach: split your image directory into non-overlapping subsets and assign each subset to an annotator.

from pathlib import Path
import shutil

def partition_images(images_dir, num_annotators, output_base):
    images = sorted(Path(images_dir).glob("*"))
    images = [p for p in images if p.suffix.lower() in (".jpg", ".jpeg", ".png", ".bmp")]

    per_annotator = len(images) // num_annotators
    remainder = len(images) % num_annotators

    idx = 0
    for annotator_id in range(num_annotators):
        count = per_annotator + (1 if annotator_id < remainder else 0)
        annotator_dir = Path(output_base) / f"annotator_{annotator_id:02d}"
        annotator_dir.mkdir(parents=True, exist_ok=True)

        for img in images[idx:idx + count]:
            shutil.copy2(img, annotator_dir / img.name)
        idx += count

    print(f"Partitioned {len(images)} images across {num_annotators} annotators")

partition_images("unlabeled/", num_annotators=4, output_base="annotation_batches/")

Each annotator opens their directory in AnyLabeling, labels independently, and returns the JSON annotation files.

Overlap for Quality Measurement

Reserve 5-10% of images as overlap — images assigned to multiple annotators. This lets you compute inter-annotator agreement (Lesson 09) as an ongoing quality metric.

import random

def partition_with_overlap(images_dir, num_annotators, output_base, overlap_pct=0.1):
    images = sorted(Path(images_dir).glob("*"))
    images = [p for p in images if p.suffix.lower() in (".jpg", ".jpeg", ".png", ".bmp")]

    # Select overlap images
    overlap_count = int(len(images) * overlap_pct)
    overlap_images = set(random.sample(range(len(images)), overlap_count))

    # Partition the rest
    non_overlap = [img for i, img in enumerate(images) if i not in overlap_images]
    overlap = [images[i] for i in overlap_images]

    per_annotator = len(non_overlap) // num_annotators

    idx = 0
    for annotator_id in range(num_annotators):
        annotator_dir = Path(output_base) / f"annotator_{annotator_id:02d}"
        annotator_dir.mkdir(parents=True, exist_ok=True)

        # Unique images for this annotator
        end = idx + per_annotator + (1 if annotator_id < len(non_overlap) % num_annotators else 0)
        for img in non_overlap[idx:end]:
            shutil.copy2(img, annotator_dir / img.name)
        idx = end

        # Overlap images go to ALL annotators
        for img in overlap:
            shutil.copy2(img, annotator_dir / img.name)

Quality Assurance Workflow

Annotation without QA is annotation you cannot trust. Here is a three-tier QA system:

Tier 1: Automated Checks

Run automated validation on every annotation submission:

import json
from pathlib import Path

def validate_annotations(annotation_dir, min_box_size=20, valid_labels=None):
    issues = []
    for json_file in Path(annotation_dir).glob("*.json"):
        with open(json_file) as f:
            data = json.load(f)

        for shape in data.get("shapes", []):
            # Check label validity
            if valid_labels and shape["label"] not in valid_labels:
                issues.append(f"{json_file.name}: unknown label '{shape['label']}'")

            # Check minimum size
            if shape["shape_type"] == "rectangle":
                pts = shape["points"]
                w = abs(pts[1][0] - pts[0][0])
                h = abs(pts[1][1] - pts[0][1])
                if w < min_box_size or h < min_box_size:
                    issues.append(f"{json_file.name}: box too small ({w:.0f}x{h:.0f})")

            # Check empty labels
            if not shape["label"].strip():
                issues.append(f"{json_file.name}: empty label")

    return issues

issues = validate_annotations(
    "annotator_00/",
    valid_labels={"car", "truck", "pedestrian", "cyclist"}
)
for issue in issues:
    print(f"  {issue}")

Tier 2: Spot Check Review

A senior annotator or ML engineer reviews a random sample of each annotator's work. 10-20% is a reasonable sample rate.

Checklist for each reviewed image:

All visible objects of target classes are annotated (no misses)
Bounding boxes are tight (no excessive padding)
Labels are correct (no class confusion)
Edge cases follow the annotation guidelines
No duplicate annotations on the same object

Tier 3: Model-Assisted QA

Train a model on the annotations and examine its failure cases. If the model consistently fails on a specific annotator's images, that annotator may be introducing systematic errors.

from ultralytics import YOLO

model = YOLO("latest_model.pt")

# Run validation per annotator batch
for annotator_dir in Path("annotation_batches/").iterdir():
    metrics = model.val(data=f"{annotator_dir}/dataset.yaml")
    print(f"{annotator_dir.name}: mAP@50={metrics.box.map50:.3f}")

A significantly lower mAP on one annotator's batch is a red flag worth investigating.

Resolving Annotation Conflicts

When two annotators label the same image differently, you need a resolution strategy:

Strategy 1: Senior Adjudication

A senior annotator reviews all disagreements and makes the final call. This is the gold standard for quality but the most expensive in terms of time.

Strategy 2: Majority Vote

If three annotators label the same image, take the majority label for each object. Fast, but requires 3x annotation for overlap images.

Strategy 3: Confidence-Weighted Merge

Weight each annotator's labels by their historical agreement rate with the adjudicator:

def weighted_merge(annotations_by_annotator, annotator_weights):
    """
    annotations_by_annotator: dict of annotator_id -> list of (label, bbox) tuples
    annotator_weights: dict of annotator_id -> float (0-1, based on historical accuracy)
    """
    # For each detected region, accumulate weighted votes per label
    # This is a simplified version — production code needs IoU matching
    merged = []
    for ann_id, anns in annotations_by_annotator.items():
        weight = annotator_weights.get(ann_id, 1.0)
        for label, bbox in anns:
            merged.append({"label": label, "bbox": bbox, "weight": weight, "source": ann_id})
    return merged

Dataset Versioning

Your dataset is code. Version it.

Git LFS for Small-Medium Datasets (Under 10 GB)

# Initialize Git LFS
git lfs install
git lfs track "*.jpg" "*.png" "*.json"

# Structure
dataset/
├── .gitattributes
├── images/
├── labels/
├── annotations.json
└── CHANGELOG.md

Commit after each annotation round with a descriptive message:

git add images/ labels/ annotations.json
git commit -m "Round 3: +150 images, focused on nighttime driving scenes"

DVC for Large Datasets (Over 10 GB)

pip install dvc
dvc init
dvc add images/ labels/
git add images.dvc labels.dvc .gitignore
git commit -m "Round 3: +150 images"
dvc push  # pushes to remote storage (S3, GCS, etc.)

Changelog

Maintain a CHANGELOG.md that records what changed in each annotation round:

## Round 3 — 2026-03-15

- Added 150 images focused on nighttime driving
- Fixed 23 label errors found in Tier 2 QA review
- Updated guidelines: golf carts now classified as "other_vehicle"
- Training mAP improved from 0.72 to 0.78

## Round 2 — 2026-03-10

- Added 200 images from uncertainty sampling
- Annotator 02 retrained after low IAA score (0.58 -> 0.81)
- Total dataset: 450 images

The Complete Team Workflow

Putting it all together:

1. ML lead writes annotation guidelines (Lesson 09)
2. Pilot round: 2-3 annotators label 50 overlap images
3. Compute IAA. Refine guidelines if IAA < threshold
4. Partition images across annotators (with 10% overlap)
5. Annotators label in AnyLabeling with AI assistance
6. Automated validation catches format errors
7. Spot check review on 10-20% sample
8. Merge annotations, resolve conflicts on overlap images
9. Version the dataset (git commit / dvc push)
10. Train model, evaluate, select next round's images (Lesson 10)
11. Repeat from step 4

When to Use a Labeling Platform Instead

AnyLabeling is a desktop tool. For teams larger than 5-10 annotators, you may need a web-based platform with built-in task management:

Team Size	Recommended Approach
1-3	AnyLabeling + shared drive + git
3-10	AnyLabeling + partitioning scripts + DVC
10-50	Label Studio or CVAT (self-hosted)
50+	Commercial platform (Scale AI, Labelbox)

AnyLabeling remains valuable even in large teams as the tool individual annotators use for AI-assisted labeling. The web platform handles task management and QA; AnyLabeling handles the actual annotation.

Key Takeaways

Split images across annotators with 5-10% overlap for quality measurement.
Three-tier QA: automated checks, spot review, model-assisted validation.
Version your dataset like code. Use git LFS or DVC.
Maintain a changelog for every annotation round.
AnyLabeling scales to teams of 5-10. Beyond that, layer a web platform on top for task management.

Course Conclusion

You now have a complete annotation pipeline:

Understand why annotation quality is the highest-leverage ML activity.
Install AnyLabeling and navigate the interface.
Annotate with every tool — rectangles, polygons, points, text.
Accelerate with SAM and YOLO auto-labeling.
Export to any training framework format.
Extend with custom ONNX models.
Specify with annotation guidelines that eliminate ambiguity.
Iterate with active learning loops.
Scale across a team with QA and versioning.

The gap between a prototype model and a production model is almost always data. Close that gap deliberately, and the models follow.

Next steps:

Download AnyLabeling and annotate your first 50 images.
Write annotation guidelines for your specific task before touching any images.
Set up the active learning loop from Lesson 10 and commit to at least 3 rounds.
Join the community: open issues, contribute models, share your workflows.