Introduction

Image segmentation has long been a bottleneck in computer vision, requiring painstaking manual labeling and task-specific model training for every new object class. Segment Anything, developed by Meta AI (formerly Facebook Research), transforms this workflow by introducing a foundation model that can “cut out” any object in any image with minimal user input. With over 18k GitHub stars, this project provides the tools to implement promptable segmentation that generalizes to new domains without additional training.

What Is Segment Anything?

Segment Anything (SAM) is a promptable image segmentation tool that allows users to isolate objects in images using points, bounding boxes, or text prompts. It is built on a Vision Transformer (ViT) architecture and is released under the permissive Apache License 2.0, making it highly accessible for both research and commercial applications.

The project is primarily written in Python and PyTorch, providing a robust framework for running inference and generating high-quality masks. By leveraging a massive dataset of 1.1 billion masks from 11 million images (SA-1B), SAM achieves strong zero-shot performance, meaning it can segment objects it has never seen during training.

Why Segment Anything Matters

Before SAM, image segmentation was a fragmented process. Developers had to collect thousands of labeled images for a specific category—such as “medical tumors” or “satellite imagery of cars”—and fine-tune a model like Mask R-CNN. This process was slow, expensive, and didn’t scale. SAM fills this gap by acting as a foundation model for vision, similar to how GPT-4 acts for text.

The ability to perform zero-shot segmentation means that practitioners no longer need to collect their own segmentation data for every new use case. This drastically reduces the time from prototype to production. Furthermore, the project’s open-source nature and the release of the SA-1B dataset have democratized high-quality segmentation for developers worldwide.

As the industry moves toward more interactive AI, SAM’s promptable interface makes it an essential tool for building smart annotation software, real-time image editors, and advanced diagnostic tools in fields like medicine and agriculture.

Key Features

• Zero-Shot Generalization: SAM can segment objects in images it has never encountered before without requiring any additional fine-tuning or training.
• Promptable Interface: The model accepts multiple types of prompts, including single clicks (points), bounding boxes, and previous masks, to refine the segmentation output.
• Automatic Mask Generation: Beyond interactive prompts, SAM can be used to automatically generate masks for every object in an image in a single pass.
• High-Quality Masks: Trained on the SA-1B dataset (1.1 billion masks), the model produces pixel-perfect boundaries that are far more accurate than traditional bounding box detections.
• Multiple Model Sizes: The project offers different checkpoints (ViT-B, ViT-L, and ViT-H), allowing developers to balance the trade-off between inference speed and mask accuracy.
• ONNX Export Support: The model can be exported to ONNX format, enabling deployment on edge devices and directly within web browsers using WebAssembly.

How Segment Anything Compares

While YOLO and Detectron2 are superior for real-time object detection with predefined classes (e.g., “person,” “car”), SAM is designed for interactive segmentation. The primary differentiator is that SAM does not need to know what the object is to segment it; it only needs to know where it is based on the prompt. This makes SAM a powerful tool for data annotation and zero-shot tasks where class labels are not available.

Compared to Mask2Former, SAM’s flexibility is unmatched. While Mask2Former provides high-quality semantic segmentation, it is bound by its training set. SAM can segment any arbitrary object in any image, regardless of whether that object was part of the original training distribution. This makes it the ideal choice for developers building tools that must handle unpredictable user input.

Getting Started: Installation

Prerequisites

The code requires python >= 3.8, pytorch >= 1.7, and torchvision >= 0.8. For optimal performance, installing PyTorch and TorchVision with CUDA support is strongly recommended.

Installation via Pip

pip install git+https://github.com/facebookresearch/segment-anything.git

Installation via Local Clone

git clone git@github.com:facebookresearch/segment-anything.git
cd segment-anything
pip install -e .

Optional Dependencies

For mask post-processing, COCO format saving, and ONNX export, install the following:

pip install opencv-python pycocotools matplotlib onnxruntime onnx

How to Use Segment Anything

The basic workflow for using SAM involves three steps: loading a model checkpoint, setting an image, and providing a prompt. First, you must download a model checkpoint (e.g., sam_vit_h_4b8939.pth) from the official repository.

Once the checkpoint is loaded, the SamPredictor class is used to set the image. The image is processed by the image encoder to create an embedding. This embedding is then reused for all subsequent prompts provided to the model, making the interactive part of the process extremely fast.

Finally, you provide a prompt—such as a point coordinate or a bounding box—and the predict method returns a segmentation mask. You can refine the mask by adding more points (positive or negative) to include or exclude specific areas of the image.

Code Examples

Interactive Segmentation with Points

This example shows how to use a single point prompt to segment an object.

from segment_anything import SamPredictor, sam_model_registry

# Load model
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

# Set image
predictor.set_image(image)

# Predict mask using a point prompt
masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375]]),
    point_labels=np.array([1]),
)
print(f"Masks found: {len(masks)}")

Automatic Mask Generation

This example demonstrates how to generate masks for all objects in an image without any user prompts.

from segment_anything import SamAutomaticMaskGenerator, sam_model_registry

# Load model
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
mask_generator = SamAutomaticMaskGenerator(sam)

# Generate masks
masks = mask_generator.generate(image)
print(f"Total objects segmented: {len(masks)}")

Real-World Use Cases

Segment Anything is particularly powerful in scenarios where the object classes are unpredictable or where high-precision masks are needed quickly.

• AI-Assisted Image Labeling: Data scientists can use SAM to pre-label objects in thousands of images, reducing the time spent on manual polygon annotation by up to 90% by simply clicking on objects.
• Medical Imaging Analysis: Radiologists can isolate specific anomalies or organs in MRI or CT scans without needing a model trained specifically for that rare pathology, thanks to zero-shot generalization.
• E-commerce Background Removal: Developers can build tools that allow users to one-click remove backgrounds from product photos, providing a professional cutout of the product with pixel-perfect accuracy.
• Geospatial Analysis: Environmental researchers can segment buildings, roads, or specific vegetation types from satellite imagery to track urban sprawl or deforestation in real-time.

Contributing to Segment Anything

Meta AI encourages contributions to the project. While the project is primarily a reference implementation for the research paper, developers can contribute by reporting bugs via GitHub Issues or submitting pull requests for optimization and utility functions.

The project follows standard GitHub flow: fork the repository, create a feature branch, and submit a pull request to the main branch. Contributors should ensure that any new code is linted and that existing tests pass before submission.

Community and Support

The primary hub for support and support is the official GitHub repository. Developers can use GitHub Discussions to ask questions about implementation and share their use cases. The project is highly active, with thousands of contributors and a wide range of community-driven extensions, such as SAM 2 and SAM 3.

For deeper technical understanding, the project provides example notebooks in the /notebooks directory, which serve as the best starting point for most developers.

Conclusion

Segment Anything is a foundational shift in computer vision. By decoupling the mask generation from the class label, SAM allows developers to segment any object in any image without the need for task-specific training data. This makes it an essential tool for anyone building interactive vision applications, from medical diagnostics to creative tools.

While SAM is computationally expensive during the initial image encoding phase, its promptable interface is extremely efficient. For those needing real-time video segmentation, the project’s successors, SAM 2 and SAM 3, offer further improvements in speed and accuracy.

Star the repo, try the quickstart, and join the community to start building the future of vision AI.

Resources

• Official Segment Anything GitHub Repository
• Segment Anything Project Page
• Model Checkpoints Download