Separate category from instance
A model that recognizes one familiar mug may face a different problem when asked to recognize mugs generally. State whether repeated use of the same object is desirable and which object properties matter. Include size, appearance, packaging, or functional differences only when relevant. Request identifiers that distinguish individual objects from category labels when your evaluation requires it.
Control what changes between attempts
Changing objects, layout, camera position, and instructions simultaneously makes it hard to explain differences in performance. Plan some comparisons in which the task remains stable while one meaningful condition changes. Naturalistic recordings can still be useful, but their metadata should reveal the variation that actually occurred rather than imply a controlled study that was never performed.
Include realistic distractors
A work area with only one obvious target can make object selection unusually simple. If deployment includes clutter or similar-looking items, describe those conditions in the brief. Determine whether distractor objects need labels and whether their placement must be recorded. Keep the task understandable enough that reviewers can tell a wrong-object interaction from an ambiguous instruction.
Review held-out combinations
Decide whether evaluation should introduce new instances, new layouts, or both. Group repeated recordings of the same scene appropriately to prevent related examples from crossing the intended boundary. Review coverage at the combination level and retain the information needed to reproduce the split. More object names alone do not establish that the collection tests meaningful generalization.
Review checklist
- Distinguish object categories and instances
- Record layout changes
- Specify relevant distractors
- Define held-out object-layout combinations