A six-question checklist for trustworthy CV dataset provenance

Trustworthy AI: final paper by Anurag Dhungana and Prakriti Bista

Domain and failure mode

Camouflage object detection asks a model to find things people struggle to see. The data needed to train such models is exactly the data hardest to obtain. Annotators have to find what they can barely see in the first place, license constraints rule out many sources, and the camouflage subjects themselves are often photographed in research contexts where redistribution is restricted. Public camouflage datasets stay small for these reasons, and they tend to inherit unclear provenance, and they get reused well beyond their stated terms. We have been building two related projects this semester. Special-SAM is a fine-tune of Meta's Segment Anything Model [4], trained on COD10K, a public camouflage benchmark. CamoSet is a dataset we are building from scratch. Doing both at once made one thing obvious: when we were using somebody else's data we did not check very much. The moment we tried to produce data ourselves, the questions we had skipped over the first time became unavoidable. The failure mode we focus on is not when a model is wrong. It is when the dataset under the model has provenance the community cannot audit. Researchers end up training on corpora they cannot legally redistribute. Contributors whose footage was scraped lose attribution and any chance to refuse. Reviewers asked to evaluate dataset papers cannot really do their job, because the source records they would need to check the claims are not published. None of those individual harms are large. The point is that the field has been doing this for ten years.

Several existing frameworks try to document dataset provenance. Gebru et al. proposed Datasheets for Datasets [2], structured documents that describe a dataset's collection, composition, intended uses, and limitations. Bender and Friedman's Data Statements for NLP [5] cover similar ground for language data. Mitchell et al.'s Model Cards [6] document trained models rather than datasets but share the philosophy. The Hugging Face Dataset Card spec is essentially Datasheets adapted for that platform. The FAIR principles [7] require data to be Findable, Accessible, Interoperable, and Reusable, with explicit metadata. All of these frameworks are descriptive. They tell a reader what is in a dataset and how it was collected, assuming the authors fill the document in honestly. They do not give a reader anything to verify the description against, and they are not operationalizable as a yes/no checklist a third party could apply without trusting the dataset's authors. We see our contribution as something that sits alongside Datasheets, not as something that replaces them; the difference is that ours is meant to be checkable by someone who is not the dataset's author.

The criterion: six yes/no questions

We propose six questions that score a CV dataset's provenance. Each is a yes/no, and the total score is between 0 and 6. The questions are designed so that a third party with access to the dataset's documentation and a sample of the data can answer them without having to take the dataset authors' word for anything.

Per-sample license metadata. For each sample, can a reader find the license under which that sample was contributed, distinct from the dataset's overall license?
Source-side rights clearance. Did the dataset authors clear rights at the source (per-contributor agreements, per-sample CC verification, public-domain confirmation), or did they assert a blanket fair-use claim and aggregate scraped material?
Per-sample origin records. Is there a way to trace each sample back to its original source (URL, content hash, contributor identifier)?
Living documentation. Does the documentation get updated as license terms change, sources go offline, or contributors withdraw consent? Or is it written once at release and frozen?
Disclosed collection methodology. Does the documentation describe how the data was actually collected - specific platforms, specific date ranges, specific search queries, rate limits?
Stated and applied exclusion criteria. Are exclusions (humans, sensitive content, redistribution-restricted material) explicitly documented, and are they applied consistently across the dataset? A "yes" requires that the information be checkable, not just asserted. "We collected from public sources" is not a yes for question 5; "We collected from YouTube on Jan 4-11, 2020 using these queries" is. We made the criterion strict on this point on purpose, because the field's documentation often relies on phrasing that reads well at publication and turns out to be unverifiable in practice. We also allow a "partial" (worth 0.5) when the spirit of a question is met but the letter is not, as when exclusions are visible from inspecting the data but never stated in the documentation as exclusions.

Trying the checklist on public datasets

Our initial predictions were 2/6 for COD10K, 4/6 for MoCA, 5/6 for a hypothetical well-documented Wikimedia collection, 1/6 for an Internet Archive subset, and 3/6 for WCS Camera Traps through LILA-BC [8]. Those were working expectations, not audited ratings. We did not establish a formal preregistration.

Applying the checklist exposed a problem with our own scoring: some answers came from assumptions about a platform rather than records for individual samples. Before publishing a numerical ranking, we would need to fix those inconsistencies and retain the evidence for every answer.

COD10K: the documentation boundary

COD10K's official release contains 10,000 images in total, including 5,066 camouflaged images [1]. The camouflage split has 3,040 training and 2,026 test images. That distinction matters when describing what Special-SAM trained on.

The useful question for our checklist is whether a reader can follow an image back to its source and inspect its individual permissions. The public materials we reviewed did not provide enough per-image records for us to do that. This is a documentation limit; it does not establish that the authors lacked permission or that a particular use is unlawful.

MoCA: a license is not a rights-clearance record

MoCA's official page describes 141 sequences across 67 categories, drawn from YouTube videos [3]. It provides dataset terms, acknowledges the original owners' copyright, and offers a removal-request process.

That is useful evidence, but it does not establish our original claim that every uploader's rights had been individually cleared. We should mark source-side clearance as not established from the records we reviewed, rather than awarding a point by reputation. The annotations also include still motion states, so our earlier claim that static-pose footage was excluded was too broad.

For either dataset, a defensible assessment needs a record of what we checked and when. An unanswered question should remain unanswered, not become an accusation or a convenient zero.

A platform does not settle the answer

Wikimedia Commons exposes file-level license and source fields, which can help answer questions 1 and 3. We still need to retain those fields and check that they apply to the material we collect. Internet Archive and YouTube metadata need the same scrutiny; a label alone does not establish that an uploader owns all the underlying material.

For CamoSet, we chose a CC-BY-only sourcing policy. That is our collection constraint, not a claim that every noncommercial license prohibits every form of redistribution. Research collections such as those indexed by LILA-BC also need review at the individual collection level. A live zoo camera without explicit reuse terms remains a question to resolve before collecting footage.

Trust tax

Adopting the criterion costs real things. The most concrete cost is that it drops data. When we filtered Internet Archive's camera-trap content by CC-BY-only license, ten of fourteen sampled videos were rejected. That small sample helped us see the cost of our sourcing policy; it is not a platform-wide rejection rate. A dataset that maximizes scale will not score well on this checklist; a dataset that scores well will be smaller than the alternative. There is also the per-sample work. Maintaining per-sample license metadata, source URLs, and content hashes adds compute and storage at acquisition, plus review work whenever a new platform is incorporated. Living documentation (q4) costs ongoing maintenance, because somebody has to actually update records as sources change. Then there is the platform-politeness cost. Respecting rate limits is slower than aggressive scraping. We hit Wikimedia's rate limit on our second batch of file fetches; the polite alternative was to interleave scraping with analysis, which slowed our overall throughput by roughly an order of magnitude. The license said nothing about this. The platform enforced it anyway. The cost we are most worried about is harder to quantify. A criterion like ours can produce false reassurance when applied carelessly. A dataset can score 6/6 on these questions and still have problems we did not encode: synthetic-data contamination, biased contributor pools, downstream uses the contributors did not anticipate. A clean checklist score does not settle trustworthiness. The risk is that publishing a checklist makes it easier for downstream papers to point at the score and skip the parts of trustworthiness the score cannot see.

Limitations and who remains unprotected

The criterion is for CV image and video datasets. We have not tried to apply it to text corpora or multimodal datasets, both of which have provenance questions in different shapes. It does not address consent from non-author humans whose images appear in footage; q6 handles that through curation, but curation is fallible. It does not protect contributors whose data passes the checklist and is later used for purposes they did not anticipate, such as a public-domain wildlife video being repurposed to train a surveillance model. The live-zoo-camera question that motivated some of our project remains unresolved: without explicit reuse terms most checklist questions stay unanswered, and we have not decided whether that should block collection or simply be recorded as unknown. Synthetic data is the case the criterion does not handle at all. Recent work has produced AI-generated camouflage images at scale. A synthetic dataset can score 6/6 trivially because every "sample" has whatever provenance the generator's authors decide to assign. The criterion treats provenance as a property of how data was acquired. It does not say anything about what the data is, and that gap matters.

Closing

When we started CamoSet, we expected the modelling work to be the hard part. The model turned out to be tractable. The data work has been harder, and most of what makes it hard sits outside the legal layer. The six questions above are an attempt to put a small piece of trustworthy data sourcing into a form somebody else could actually check. None of this solves trustworthy AI. What it gives us is a record of the questions we need to answer before asking somebody else to use our data.

References

[1] Fan, Deng-Ping, et al. "Camouflaged Object Detection." CVPR 2020. (COD10K.) [2] Gebru, Timnit, et al. "Datasheets for Datasets." Communications of the ACM, 2021. [3] Lamdouar, Hala, et al. "Betrayed by Motion: Camouflaged Object Discovery via Motion Segmentation." ACCV 2020. (MoCA.) [4] Kirillov, Alexander, et al. "Segment Anything." ICCV 2023. [5] Bender, Emily M., and Batya Friedman. "Data Statements for Natural Language Processing." Transactions of the ACL, 2018. [6] Mitchell, Margaret, et al. "Model Cards for Model Reporting." FAT* 2019. [7] Wilkinson, Mark D., et al. "The FAIR Guiding Principles for scientific data management and stewardship." Scientific Data, 2016. [8] LILA-BC. Labeled Information Library of Alexandria: Biology and Conservation. lila.science/datasets/