Shrink and Fine-Tune is a model distillation technique which consists of copying parameters from a subset of layers of the teacher model into the student model, then fine-tuning the student model.