| Téléchargement | - Voir la version de l’auteur : Over-training with Mixup may hurt generalization (PDF, 4.7 Mio)
|
|---|
| Auteur | Rechercher : Liu, Zixuan; Rechercher : Wang, Ziqiao; Rechercher : Guo, Hongyu1; Rechercher : Mao, Yongyi |
|---|
| Affiliation | - Conseil national de recherches Canada. Technologies numériques
|
|---|
| Format | Texte, Article |
|---|
| Conférence | Eleventh International Conference on Learning Representations (ICLR) 2023, May 1-5, 2023, Kigali, Rwanda [Hybrid] |
|---|
| Description physique | 20 p. |
|---|
| Sujet | Mixup; generalization; overfitting; regularization |
|---|
| Résumé | Mixup, which creates synthetic training instances by linearly interpolating random sample pairs, is a simple and yet effective regularization technique to boost the performance of deep models trained with SGD. In this work, we report a previously unobserved phenomenon in Mixup training: on a number of standard datasets, the performance of Mixup-trained models starts to decay after training for a large number of epochs, giving rise to a U-shaped generalization curve. This behavior is further aggravated when the size of original dataset is reduced. To help understand such a behavior of Mixup, we show theoretically that Mixup training may introduce undesired data-dependent label noises to the synthesized data. Via analyzing a least-square regression problem with a random feature model, we explain why noisy labels may cause the U-shaped curve to occur: Mixup improves generalization through fitting the clean patterns at the early training stage, but as training progresses, Mixup becomes over-fitting to the noise in the synthetic data. Extensive experiments are performed on a variety of benchmark datasets, validating this explanation. |
|---|
| Date de publication | 2023-05-05 |
|---|
| Maison d’édition | The Conference
OpenReview |
|---|
| Dans | |
|---|
| Autre version | |
|---|
| Langue | anglais |
|---|
| Publications évaluées par des pairs | Oui |
|---|
| Exporter la notice | Exporter en format RIS |
|---|
| Signaler une correction | Signaler une correction (s'ouvre dans un nouvel onglet) |
|---|
| Identificateur de l’enregistrement | 64a48c99-034a-4a0b-a1c7-b30e985ff93e |
|---|
| Enregistrement créé | 2023-08-11 |
|---|
| Enregistrement modifié | 2023-08-14 |
|---|
