Racing the Clock
Training a transformer from scratch for extractive summarization. Bachelor thesis, Vrije Universiteit Amsterdam, 2025.
Modern summarization models lean on large pre-trained encoders like BERT. I wanted to know how far you get without that: a transformer trained from random initialization, on one consumer GPU, in a fixed 12-hour budget. The approach follows the idea of "cramming" (Geiping and Goldstein, 2022).
Results
| Model | Pre-trained | R-1 | R-2 | R-L |
|---|---|---|---|---|
| Lead-3 baseline | No | 40.34 | 17.70 | 36.57 |
| BanditSum | No | 41.50 | 18.70 | 37.60 |
| NeuSum | No | 41.59 | 19.01 | 37.98 |
| BertSumExt | Yes | 43.25 | 20.24 | 39.63 |
| MatchSum | Yes | 44.41 | 20.86 | 40.55 |
| Mine, 12 h from scratch | No | 39.87 | 17.50 | 25.09 |
The model roughly matches the Lead-3 baseline on ROUGE-1 and ROUGE-2, but its ROUGE-L is much lower. It reliably picks the headline facts and tends to miss background details, which traces back to the sentence-level labels it was trained on. Pre-trained models stay about 4 points ahead on ROUGE-1.
How it works
- Model. A 12-layer, 768-dimensional transformer encoder written from scratch in PyTorch. Each sentence gets a [CLS] token, and a linear layer scores it for inclusion.
- Labels. CNN/DailyMail only has abstractive reference summaries, so I made "silver" labels: a sentence counts as positive when its ROUGE-L against the reference is at least 0.20.
- Training. AdamW, cosine learning-rate schedule with warm-up, gradient clipping and batches of 16, giving about 19,000 steps in 12 hours.
- Summaries. At inference the model keeps the three highest-scoring sentences in their original order.
What I'd try next
Better oracle labels and a simple redundancy filter at inference. Both stay cheap enough for a cramming budget and target the ROUGE-L gap directly.