Question 1: Short answer
mediumAI-checkedTraining Deep NetworksTo find a good learning rate, you train for 500 iterations while growing the learning rate exponentially, record the loss at each iteration, and later look for where the loss shoots up:
opt = torch.optim.SGD(model.parameters(), lr=1e-5)
sched = torch.optim.lr_scheduler.ExponentialLR(opt, gamma=g)
for step in range(500):
... # train on one batch, record the loss
opt.step()
sched.step()
Which value of g makes the learning rate reach 10 after the 500 scheduler steps? Give 4
decimals.