This tutorial walks you through the process of finetuning a Segment Anything Model (SAM) using custom data.

Code from this video is available here: https://github.com/bnsreenu/python_fo...

What is SAM?
SAM is an image segmentation model developed by Meta AI. It was trained over 11 billion segmentation masks from millions of images. It is designed to take human prompts, in the form of points, bounding boxes or even a text prompt describing what should be segmented.

What are the key features of SAM?
Zeroshot generalization: SAM can be used to segment objects that it has never seen before, without the need for additional training.

Flexible prompting: SAM can be prompted with a variety of input, including points, boxes, and text descriptions.

Realtime mask computation: SAM can generate masks for objects in real time. This makes SAM ideal for applications where it is necessary to segment objects quickly, such as autonomous driving and robotics.

Ambiguity awareness: SAM is aware of the ambiguity of objects in images. This means that SAM can generate masks for objects even when they are partially occluded or overlapping with other objects.

How does SAM work?
SAM works by first encoding the image into a highdimensional vector representation. The prompt is encoded into a separate vector representation. The two vector representations are then combined and passed to a mask decoder, which outputs a mask for the object specified by the prompt.

The image encoder is a vision transformer (ViTH) model, which is a large language model that has been pretrained on a massive dataset of images. The prompt encoder is a simple text encoder that converts the input prompt into a vector representation. The mask decoder is a lightweight transformer model that predicts the object mask from the image and prompt embeddings.

SAM paper: https://arxiv.org/pdf/2304.02643.pdf​

Link to the dataset used in this demonstration: https://www.epfl.ch/labs/cvlab/data/d...
Courtesy: EPFL

This code has been heavily adapted from this notebook but modified to work with a truly custom dataset where we have a bunch of images and binary masks. https://github.com/NielsRogge/Transfo...