LIMA shifts fine-tuning work from scale to selection

Yann LeCun discussed LIMA publicly on 22 May. The paper submitted on 18 May, by Chunting Zhou and colleagues, fine-tunes LLaMA 65B on 1,000 curated prompt-and-answer examples using supervised learning, without reinforcement learning or human preference modelling.

The earlier LLaMA paper reports 1.4 trillion training tokens for its 65B model. LIMA’s 1,000 examples are a later fine-tuning stage.

The less visible cost is selection. LIMA’s authors train for 15 epochs but manually choose checkpoints between epochs five and ten using 50 held-out examples. Validation perplexity, which measures how well a model predicts text, doesn’t track the answer quality they want.

In their appendix, validation perplexity rises while ChatGPT’s ratings of generated answers improve. That makes lowest validation loss a poor checkpoint rule in this experiment, rather than proof of better real-world performance.

The dataset also requires work: responses are curated for consistent style and prompts for variety. Adding 30 dialogue examples improves multi-turn conversations. The authors warn that constructing examples takes substantial effort and that unlucky decoding or adversarial prompts can produce weak answers.

The experiment suggests strong pretraining can reduce later demonstration needs without removing curation and evaluation work. That’s our reading of the papers, not a measured cost saving.

Research summary adapted from Chunting Zhou and colleagues’ LIMA paper, licensed CC BY 4.0.

Photo: LPS.1, “Meta Platforms Headquarters Menlo Park California” (2022). CC0 1.0. Resized; cropped; responsive crops.