Pretrain Alignment

Train LLM three steps

Pre-train -> Supervised Fine-tuning -> RLHF

• Alignment = Supervised Fine-tuning + RLHF = Fine-tuning

Alignment

Models Naming Tips

• Models with base mean only pre-train e.g. Llama-2-7b-base
• Models with chat , instruct mean with alignment e.g. Llama-2-7b-chat

Alignment using Datasets

• Paper:
  1. LIMA: Less Is More for Alignment
• Don't need to be large, but need to be high quality

Knowledge Distillation

• Paper (Choose answer from teacher model method)
  1. AlpaGasus: Training A Better Alpaca with Fewer Data
  2. Long Is More for Alignment

Alignment before and after

• Paper:

1. The Unlocking Spell on Base LLMs

Alignment different methods

1. Response Tuning
   • Paper: Revealing the Inherent Instructability of Pre-Trained Language Models
2. Rule-base adapter
   • Paper: Instruction Following without Instruction Tuning

Self-Alignment

• Paper: Self-Rewarding Language Models

Pretrain

Efficient Pretrain

• Physics of Language Models: Part 3.1

DataSet

• The FineWeb Datasets

DataSet Quality 重要性

:::[info] 說明 Textbook 是 chatGPT 生成的 may be effected the result :::

• Textbooks Are All You Need

Rephrasing the Web

• Paper: Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
• 在有限算力，固定模型下應該看更多不同的資料
• Paper: Scaling Data-Constrained Language Models
• Paper: Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget

Alignment 的極限

• Paper: The False Promise of Imitating Proprietary LLMs
• Paper: Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
• Paper: Learning or Self-aligning? Rethinking Instruction Fine-tuning