Learning-rate warmup is the gradual increase of the learning rate for a few steps or epochs at the start of the learning process. The increase is typically linear from 0 up to the initial learning rate sent to the learning-rate scheduler.