Can a generalist model learn to specialize?
That's what my classmate Anurag and I wanted to find out when we chose our final project for Computer Vision at GWU. SAM (Meta's Segment Anything Model) is remarkable at what it does. Give it any image, click anywhere, it'll segment something. But "good at everything" often means "not great at anything specific." We wanted to know if we could take that foundation model and push it toward tasks where the vanilla model genuinely struggles.
We picked three challenges, each harder than the last. What followed was one failure, one modest success, one breakthrough, and a trip to the Smithsonian Zoo.
Challenge 1: Wood textures, and how we lost 36 points
We started with wood texture segmentation: teaching SAM to distinguish between different wood grains (oak, walnut, pine) in 2x2 mosaic images. It felt like a reasonable warm-up.
We trained on 500 synthetic mosaics, used Dice loss only, ran 5 epochs. The training loss dropped to 0.0001. We were thrilled.
Then we tested it. On Hard Mode, our "specialized" model scored 59.52% mIoU. The vanilla MobileSAM scored 95.80%.
We knocked 36 points off it.
Looking back, it was obvious. A training loss of 0.0001 isn't learning, it's memorizing. We'd overfitted catastrophically on 500 samples, used only Dice loss which doesn't penalize noisy boundaries, had no validation set to catch it early, and most critically: we'd tried to improve a model that was already nearly perfect. When your baseline is 95.8%, there's almost no room to go up and a huge amount of room to fall.
That failure taught us more than a success would have. Don't fix what isn't broken. Low loss is a warning, not a reward.
Challenge 2: Carpet defects, synthetic data, and a modest win
The second challenge had a real industrial motivation: automated quality control in textile manufacturing. A carpet production line generates thousands of meters of fabric daily. Humans visually scan for defects and they miss things, especially after hours of inspection. Could a specialized vision model catch what human eyes miss?
The catch: we had only 89 labeled defect images from the MVTec AD dataset. That's nowhere near enough to train a neural network.
So we built a synthetic defect generator. We took 280 defect-free carpet images and programmatically injected three types of anomalies: darkened patches, thin dark lines, and hue-shifted circles. From those 280 clean images, we generated 500 synthetic training samples.
Applying lessons from the wood failure: no more wild loss drops, no more training without watching the curves. The loss decreased steadily from 0.25 to 0.15 across 5 epochs, a stable and gradual curve this time.
On real defects: 36.18% mIoU, up from the baseline's 29.43%. A 6.75-point improvement on industrial data our model had never seen during training.
The gain was not spectacular, but the direction was right, and we now had a framework that worked.
Challenge 3: Camouflaged animals, where it finally clicked
Evolution is a ruthless optimizer for defeating visual detection. It's one of computer vision's hardest problems for good reason.
Many animals have evolved over millions of years to be visually hard to spot, to predators, prey, and humans. COD10K holds 10,000 images, and 5,066 of them are camouflaged scenes with ground truth masks, while the rest cover non-camouflaged objects and backgrounds.
We applied every lesson from the previous two experiments. More data: our class-project copy of COD10K had 6,000 training files, which we doubled with horizontal flips, though the later project log corrects the camouflage-only count to 3,040 images, or 6,080 after flipping. Bigger model: we moved from MobileSAM (about 10M parameters) to SAM ViT-H, the largest SAM variant at about 632M parameters, although we kept its image encoder frozen and trained only the roughly 4M-parameter mask decoder. Combined loss: 50% BCE + 50% Dice loss. BCE forces pixel-level boundaries; Dice keeps the mask shape coherent. Multi-prompt training randomly alternated between point prompts and bounding box prompts, so the model couldn't rely on memorizing a prompt pattern. It had to actually learn where the animal was.
The training curves were everything we wanted: steady, gradual improvement over 7 epochs, loss dropping from 0.142 to 0.106 with no spikes, no collapses.
Results:
- Base SAM ViT-H: 47.52% mIoU
- Our specialized model: 66.35% mIoU
- +18.83 percentage points, a 39.6% relative improvement.
Those figures come from our class-project summary on 200 sampled test images. The committed notebook records 65.73% for the specialized model with the same center-point prompt as the baseline, a gain of about 18.2 points, and I couldn't trace the 66.35% figure to that run. Either way, that's the difference between masks that overlap the animal by less than half and masks that cover about two-thirds of it.
The zoo visit
We didn't stop at the benchmark. We took the model to the Smithsonian National Zoo and photographed real animals in their enclosures, completely out-of-distribution from anything in our training set.
It held up, not perfectly but meaningfully. Seeing your model correctly segment a camouflaged animal in a real environment you photographed yourself is a different kind of validation than a test set number, although the zoo photos had no aggregate score, so they were an encouraging sign rather than proof.
What all three experiments showed, side by side
Baseline → specialized · mIoU
Wood textures — MobileSAM, Dice loss, 5 epochs
Carpet defects — MobileSAM, 500 synthetic samples
Camouflaged animals — SAM ViT-H, BCE+Dice, COD10K
The single best predictor of whether specialization works isn't the model, the loss function, or the data volume. It's the baseline. How much is the model already struggling? Wood at 95.8% had nowhere to go but down. Camouflage at 47.5% had real room to improve.
Fine-tuning a model that's already excellent on your task is mostly just risk.
Fail, understand, apply, succeed
You learn more from a well-documented failure than from an undocumented success. The wood experiment failed, but because we understood why, every later experiment got better. The carpet experiment worked a little, but its limits were obvious. By the time we reached camouflage, we weren't guessing as much.



