#test-set-reuse

Question 1: Single choice

Someone holds a test set of $n$ inputs with binary labels and shows you only the inputs. You may run any model on them, but all you learn is its error rate on the whole test set. If you change one prediction at a time, about how many evaluations do you need before you know every label and can score zero test error?

About $n$ , a baseline and then one model per point with its prediction flipped1
About $\log_2 n$ , by a binary search that halves the set of test points each time2
About $2^n$ , because every labelling must be tried until one of them scores zero3
None, because an error rate over all points says nothing about single labels4

Timed test