#model-instances

Question 1: Single choice

What is wrong with this 5-fold cross-validation of a PyTorch network? (train runs some epochs of SGD on the given data, starting from the model's current weights.)

model = make_model()
scores = []
for train_idx, val_idx in KFold(n_splits=5, shuffle=True).split(X):
    train(model, X[train_idx], y[train_idx])
    scores.append(evaluate(model, X[val_idx], y[val_idx]))
print(sum(scores) / len(scores))

One model keeps training across folds and has seen later validation folds1
The scores should not be averaged; report the score of the best fold instead2
Shuffling before splitting leaks validation examples into the training folds3
Five folds are too few for a valid cross-validation estimate of the error4

Timed test