#test-set

Question 1: Single choice

A hyperparameter search trains on FashionMNIST's 60,000 training images, picks the learning rate with the lowest error on the 10,000 test images, and reports that test error. Which protocol fixes the problem?

Tune on the test images, then rerun the search with another random seed to confirm the choice1
Hold out part of the training images for validation, tune on it, and use the test set once at the end2
Keep tuning on the test images, but report the training error, which the search cannot influence3
Merge the training and test images and pick the learning rate with the lowest error on the merged set4

Timed test