Training Set Selection 


Evaluation criteria

In this challenge, your task will be to design a data selection strategy that chooses the best training examples from a candidate pool of training images (a custom subset of the Open Images Dataset V6 train set) which maximizes the F1 score (previously: mean average precision) across a set of binary classification tasks for different visual concepts (e.g., “Cupcake”, “Hawk”, “Sushi”). 

Your submission will be a training set for each of the classification tasks in this challenge. 

Please refer to the Rules for additional information on valid submissions.

Please refer to our github repo for further details on the submission format.

Overview of supporting tools

In order to support this challenge, we've developed with a set of tools to facilitate development and submission of challenge solutions: MLCube and Dynabench

MLCube developed to help participants with offline evaluation. More specifically, MLCube helps participants:

Dynabench is a community-driven platform for benchmarks, and it is the platform where participants will upload their submissions for online evaluation. Valid submissions will then be recorded and ranked in the challenge's leaderboard.

Offline evaluation using MLCube

To start, see our MLCube tutorial. Note that you will need to first sign up for the Dynabench platform to access this tutorial.

You will also find further details in our github repo.

Online evaluation using Dynabench

Once you are ready to submit, please follow the instructions on the Dynabench page for this challenge.

A few important notes: