Skip to main content
Prakriti Bista

AI/ML Systems

Special-SAM: Foundation Model Fine-Tuning

Trained SAM ViT-H's mask decoder with the rest of the 636M-parameter model frozen, raising camouflaged-object mIoU from 47.52% to 66.35% on 200 sampled COD10K test images.

Special-SAM: Foundation Model Fine-Tuning

Project thesis

A three-part fine-tuning study showing when foundation-model specialization helps, when it fails, and why the baseline matters.

Question

Could SAM specialize without retraining the whole foundation model?

Decision

Memory limits and class imbalance shaped the loss function, prompts, and training plan.

Can a generalist model learn to specialize?

That's what my classmate Anurag and I wanted to find out when we chose our final project for Computer Vision at GWU. SAM (Meta's Segment Anything Model) is remarkable at what it does. Give it any image, click anywhere, it'll segment something. But "good at everything" often means "not great at anything specific." We wanted to know if we could take that foundation model and push it toward tasks where the vanilla model genuinely struggles.

We picked three challenges, each harder than the last. What followed was one failure, one modest success, one breakthrough, and a trip to the Smithsonian Zoo.

Challenge 1: Wood textures, and how we lost 36 points

We started with wood texture segmentation: teaching SAM to distinguish between different wood grains (oak, walnut, pine) in 2x2 mosaic images. It felt like a reasonable warm-up.

We trained on 500 synthetic mosaics, used Dice loss only, ran 5 epochs. The training loss dropped to 0.0001. We were thrilled.

Then we tested it. On Hard Mode, our "specialized" model scored 59.52% mIoU. The vanilla MobileSAM scored 95.80%.

We knocked 36 points off it.

Looking back, it was obvious. A training loss of 0.0001 isn't learning, it's memorizing. We'd overfitted catastrophically on 500 samples, used only Dice loss which doesn't penalize noisy boundaries, had no validation set to catch it early, and most critically: we'd tried to improve a model that was already nearly perfect. When your baseline is 95.8%, there's almost no room to go up and a huge amount of room to fall.

That failure taught us more than a success would have. Don't fix what isn't broken. Low loss is a warning, not a reward.

Challenge 2: Carpet defects, synthetic data, and a modest win

The second challenge had a real industrial motivation: automated quality control in textile manufacturing. A carpet production line generates thousands of meters of fabric daily. Humans visually scan for defects and they miss things, especially after hours of inspection. Could a specialized vision model catch what human eyes miss?

The catch: we had only 89 labeled defect images from the MVTec AD dataset. That's nowhere near enough to train a neural network.

So we built a synthetic defect generator. We took 280 defect-free carpet images and programmatically injected three types of anomalies: darkened patches, thin dark lines, and hue-shifted circles. From those 280 clean images, we generated 500 synthetic training samples.

Applying lessons from the wood failure: no more wild loss drops, no more training without watching the curves. The loss decreased steadily from 0.25 to 0.15 across 5 epochs, a stable and gradual curve this time.

On real defects: 36.18% mIoU, up from the baseline's 29.43%. A 6.75-point improvement on industrial data our model had never seen during training.

The gain was not spectacular, but the direction was right, and we now had a framework that worked.

Challenge 3: Camouflaged animals, where it finally clicked

Evolution is a ruthless optimizer for defeating visual detection. It's one of computer vision's hardest problems for good reason.

Many animals have evolved over millions of years to be visually hard to spot, to predators, prey, and humans. COD10K holds 10,000 images, and 5,066 of them are camouflaged scenes with ground truth masks, while the rest cover non-camouflaged objects and backgrounds.

We applied every lesson from the previous two experiments. More data: our class-project copy of COD10K had 6,000 training files, which we doubled with horizontal flips, though the later project log corrects the camouflage-only count to 3,040 images, or 6,080 after flipping. Bigger model: we moved from MobileSAM (about 10M parameters) to SAM ViT-H, the largest SAM variant at about 632M parameters, although we kept its image encoder frozen and trained only the roughly 4M-parameter mask decoder. Combined loss: 50% BCE + 50% Dice loss. BCE forces pixel-level boundaries; Dice keeps the mask shape coherent. Multi-prompt training randomly alternated between point prompts and bounding box prompts, so the model couldn't rely on memorizing a prompt pattern. It had to actually learn where the animal was.

The training curves were everything we wanted: steady, gradual improvement over 7 epochs, loss dropping from 0.142 to 0.106 with no spikes, no collapses.

Results:

  • Base SAM ViT-H: 47.52% mIoU
  • Our specialized model: 66.35% mIoU
  • +18.83 percentage points, a 39.6% relative improvement.

Those figures come from our class-project summary on 200 sampled test images. The committed notebook records 65.73% for the specialized model with the same center-point prompt as the baseline, a gain of about 18.2 points, and I couldn't trace the 66.35% figure to that run. Either way, that's the difference between masks that overlap the animal by less than half and masks that cover about two-thirds of it.

The zoo visit

We didn't stop at the benchmark. We took the model to the Smithsonian National Zoo and photographed real animals in their enclosures, completely out-of-distribution from anything in our training set.

It held up, not perfectly but meaningfully. Seeing your model correctly segment a camouflaged animal in a real environment you photographed yourself is a different kind of validation than a test set number, although the zoo photos had no aggregate score, so they were an encouraging sign rather than proof.

What all three experiments showed, side by side

Baseline → specialized · mIoU

Wood textures — MobileSAM, Dice loss, 5 epochs

95.80%
59.52%
▼ 36.28 pts

Carpet defects — MobileSAM, 500 synthetic samples

29.43%
36.18%
▲ 6.75 pts

Camouflaged animals — SAM ViT-H, BCE+Dice, COD10K

47.52%
66.35%
▲ 18.83 pts
Wood and carpet fine-tune MobileSAM; camouflage uses the much bigger SAM ViT-H, so they aren't the same model. The thing that decided whether specializing helped was the baseline. At 95.8% there was nowhere to go but down. The 66.35% camouflage figure comes from our class-project summary; the committed notebook records 65.73% for the same center-point setup.

The single best predictor of whether specialization works isn't the model, the loss function, or the data volume. It's the baseline. How much is the model already struggling? Wood at 95.8% had nowhere to go but down. Camouflage at 47.5% had real room to improve.

Fine-tuning a model that's already excellent on your task is mostly just risk.

Fail, understand, apply, succeed

You learn more from a well-documented failure than from an undocumented success. The wood experiment failed, but because we understood why, every later experiment got better. The carpet experiment worked a little, but its limits were obvious. By the time we reached camouflage, we weren't guessing as much.

What worked

  • Combining BCE and Dice loss targeted the foreground/background imbalance, although we never ran either loss alone on the camouflage task for comparison
  • Alternating between prompt types (points and boxes) kept the model from relying on a single prompt pattern
  • The 18.83-point mIoU gain on 200 sampled test images suggests domain-specific fine-tuning was worth it here
  • Training loss fell steadily from 0.142 to 0.106 over seven epochs with no spikes, though without a validation split we can't rule out overfitting

What didn't

  • Still relies on quality prompts; SAM's architecture means prompt engineering matters as much as the model weights
  • Generalization beyond the training distribution is uncertain without more diverse domain data

Built with

Python · PyTorch · Computer Vision · SAM · Deep Learning · ViT