The problem
An image model can suggest a mask or a box. An annotator still needs to inspect it, correct the boundary, assign a label, and save useful training data. That gap between model output and a workable editing loop is what led me to build AnyLabeling.
I started from Labelme instead of rebuilding an annotation editor. My contribution was the redesigned interface and the integration of model-assisted labeling into that workflow. Other people contributed too: the original build account credits Henry with extending the YOLO integrations. This case study focuses on the early architecture described in that account, not a claim to have written every part of the current project.
The decisions that shaped the tool
The editor needed to remain useful while models changed. I kept the canvas and labeling interface separate from model management: an auto-labeling widget handles the controls, and a model manager loads and runs the selected model. Predictions become editable shapes in the existing annotation workflow.
Segment Anything introduced a useful separation of its own. The encoder computes an image embedding, while the decoder combines that embedding with the user's points or box prompts. Running both again for every click would repeat expensive work. The initial implementation ran the encoder once per image, reused embeddings through a bounded cache, and precomputed upcoming images in a worker thread.
The early SAM inference path used ONNX Runtime and NumPy instead of importing the full research package. That choice reduced the application's dependency surface, but it also made export compatibility and postprocessing part of the application's responsibility. Masks still needed to become polygons or rectangles a person could edit and save.
Alternatives and tradeoffs
A model-only demo would have been smaller, but would have left the actual labeling work elsewhere. A new editor would have offered more freedom while also requiring me to rebuild drawing, file handling, and other established behavior. Extending Labelme let the work concentrate on the interaction between the annotator and AI.
Caching and precomputation reduce repeated encoder work; they also consume memory and need sensible bounds. Model-assisted annotation still depends on the images, model, prompts, and human review. A plausible prediction is not automatically a correct training label.
What shipped
The result is an open-source desktop labeling tool with manual editing and AI assistance in the same application. The published project includes segmentation, object-detection, and text-labeling workflows. The project repository contains current capabilities, releases, contributors, and installation details.
This page does not assign a percentage speedup or annotation-accuracy improvement: those would require a specified dataset, hardware, and evaluation procedure. The concrete outcome is the working application and its published implementation.
What I would carry into another project
Separate the model from the user's editing workflow. Reuse expensive intermediate results when the interaction permits it. Keep predictions inspectable and correctable. The useful product is the complete loop from an image to saved labels, not just the moment a model produces a mask.
For implementation details and contributor attribution, read the original AnyLabeling build article.