Lesson 08: Custom Models for Auto-Labeling

The built-in YOLO and SAM models cover general-purpose annotation. But if you are detecting PCB defects, segmenting crop diseases, or finding tumors in medical scans, you need a model trained on your domain. AnyLabeling lets you load custom ONNX models with a simple YAML configuration file. See the AnyLabeling custom models documentation for the full reference.

Requires AnyLabeling version 0.2.22 or later.

The Three-Step Process

1. Convert your trained model to ONNX format.
2. Write a config.yaml that tells AnyLabeling how to use it.
3. Load the config in the AnyLabeling UI.

Step 1: Convert to ONNX

From Ultralytics (YOLOv5/v8/11)

This is the most common path. Ultralytics provides a one-command export:

# YOLOv8
yolo export model=best.pt format=onnx imgsz=640 simplify=True

# YOLOv5 (v6.2 recommended for compatibility)
python export.py --weights best.pt --include onnx --img-size 640 --simplify

Both produce a best.onnx file ready for AnyLabeling.

Important: Use simplify=True (or --simplify). ONNX simplification removes redundant operations and improves inference compatibility with ONNX Runtime.

From PyTorch (Generic)

import torch

model = MyDetector()
model.load_state_dict(torch.load("detector.pth"))
model.eval()

dummy = torch.randn(1, 3, 640, 640)
torch.onnx.export(
    model, dummy, "detector.onnx",
    input_names=["images"],
    output_names=["output"],
    opset_version=11,
    dynamic_axes={"images": {0: "batch"}, "output": {0: "batch"}},
)

Verify the ONNX Model

Always verify before loading into AnyLabeling:

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("best.onnx")
input_name = session.get_inputs()[0].name
input_shape = session.get_inputs()[0].shape
print(f"Input: {input_name}, shape: {input_shape}")

# Run a test inference
dummy = np.random.randn(1, 3, 640, 640).astype(np.float32)
outputs = session.run(None, {input_name: dummy})
for i, out in enumerate(outputs):
    print(f"Output {i}: shape={out.shape}")

If this runs without errors, the model is ready.

Step 2: Write config.yaml

The config file tells AnyLabeling how to load and interpret your model. The format depends on the model type.

YOLOv5 Config

type: yolov5
name: my-defect-detector-v5
display_name: PCB Defect Detector (YOLOv5)
model_path: best.onnx
input_width: 640
input_height: 640
score_threshold: 0.25
nms_threshold: 0.45
classes:
  - scratch
  - solder_bridge
  - missing_component
  - cold_joint
  - crack

YOLOv8 Config

type: yolov8
name: my-defect-detector-v8
display_name: PCB Defect Detector (YOLOv8)
model_path: best.onnx
input_width: 640
input_height: 640
score_threshold: 0.25
nms_threshold: 0.45
classes:
  - scratch
  - solder_bridge
  - missing_component
  - cold_joint
  - crack

Segment Anything Config

If you fine-tuned SAM on your domain:

type: segment_anything
name: my-sam-medical
display_name: Medical SAM (Fine-tuned)
encoder_model_path: sam_encoder.onnx
decoder_model_path: sam_decoder.onnx
input_size: 1024
max_width: 2048
max_height: 2048

SAM 3 Config (With Text Prompts)

type: segment_anything_3
name: my-sam3
display_name: SAM 3 (Custom)
encoder_model_path: sam3_encoder.onnx
decoder_model_path: sam3_decoder.onnx
language_encoder_path: sam3_language.onnx
input_size: 1008
max_width: 2048
max_height: 2048

Key Config Fields

Step 3: Load in AnyLabeling

1. Organize your files in a single folder:

my_model/
├── config.yaml
├── best.onnx

2. In AnyLabeling, click the brain icon to enter auto-labeling mode.
3. From the model dropdown, select Load Custom Model.
4. Browse to your config.yaml file and select it.
5. The model loads and appears in the dropdown with your display_name.

The model is now available for auto-labeling. YOLO models run automatically on each image. SAM models wait for point/rectangle/text prompts.

Practical Example: Training and Loading a Custom Detector

Here is the full cycle — the loop you will run repeatedly in production:

1. Initial Manual Annotation

Annotate 50-100 images manually in AnyLabeling. Export to YOLO format.

2. Train YOLOv8

yolo detect train data=dataset.yaml model=yolov8s.pt epochs=50 imgsz=640

3. Export to ONNX

yolo export model=runs/detect/train/weights/best.pt format=onnx simplify=True

4. Create Config

Write config.yaml with your class list.

5. Load in AnyLabeling

Load the custom model and use it to auto-label the next batch of images.

6. Correct and Retrain

Review the auto-labels, correct mistakes, add them to your training set, and retrain. Each iteration produces a better model and requires less correction.

This is the active learning loop we formalize in Lesson 10.

Troubleshooting

"Model failed to load" — Check that the ONNX file path in config.yaml is correct (relative to the config file location). Verify the ONNX file loads with onnxruntime directly.

Detections are garbage — Verify input_width and input_height match what the model was trained with. A model trained at 640x640 will produce nonsense if AnyLabeling feeds it 416x416.

Wrong class names — The classes list order must exactly match the class ID order from training. Class 0 in the model output maps to the first entry in the list.

Slow inference on CPU — Large models (YOLOv8x, SAM ViT-H) are slow on CPU. Use the GPU version (pip install anylabeling-gpu) or switch to a smaller model variant.

Key Takeaways

• Custom models turn AnyLabeling from a generic tool into a domain-specific annotation accelerator.
• The process is: ONNX export + config.yaml + load in UI.
• Ultralytics YOLO models export to ONNX with a single command.
• Always verify your ONNX model with onnxruntime before loading it into AnyLabeling.
• The train-export-load cycle is the foundation of the active learning pipeline in Lesson 10.

In the next lesson, we step back from the tool and focus on strategy: how to write annotation guidelines that produce consistent, high-quality datasets.