Training Set Selection 

Evaluation criteria

In this challenge, your task will be to design a data selection strategy that chooses the best training examples from a candidate pool of training images (a custom subset of the Open Images Dataset V6 train set) which maximizes the F1 score (previously: mean average precision) across a set of binary classification tasks for different visual concepts (e.g., “Cupcake”, “Hawk”, “Sushi”). 

Your submission will be a training set for each of the classification tasks in this challenge. 

Please refer to the Rules for additional information on valid submissions.

Please refer to our github repo for further details on the submission format.

Overview of supporting tools

In order to support this challenge, we've developed with a set of tools to facilitate development and submission of challenge solutions: MLCube and Dynabench. 

MLCube developed to help participants with offline evaluation. More specifically, MLCube helps participants:

• Download necessary resources for development

• Reliably and easily test developed solutions

• Prepare submissions for the online leaderboards

Dynabench is a community-driven platform for benchmarks, and it is the platform where participants will upload their submissions for online evaluation. Valid submissions will then be recorded and ranked in the challenge's leaderboard.

Offline evaluation using MLCube

To start, see our MLCube tutorial. Note that you will need to first sign up for the Dynabench platform to access this tutorial.

You will also find further details in our github repo.

Online evaluation using Dynabench

Once you are ready to submit, please follow the instructions on the Dynabench page for this challenge.

A few important notes:

• A valid submission on the online platform must follow the Rules for this challenge

• You must submit a training set for each of the classification tasks at the same time

• Submissions must follow the format is specified on the Submit Train Files page in Dynabench

• Submissions may take anywhere from 10 minutes to a few hours to validate

• We will email you once your submission has been validated

• To post your submission on the leaderboards:

  • Click on your user icon on the top right corner of the Dynabench site

  • Click on models (all of your model submissions will be listed here)

  • Select the model you want to publish, then click the "Publish" button. 

  • (Optional) You may include more information for your published model by selecting the "Edit" button and updating the information