JaydenTeoh Releases Official Codebase for Multi-Token Prediction Done Right

The official codebase for the research paper "Multi-Token Prediction Done Right" has been published on GitHub by repository owner JaydenTeoh under the name Soft-MTP. The repository provides implementation resources for Next-Token Prediction (NTP), standard Multi-Token Prediction (hard_mtp), and Soft-MTP objectives, along with pre-trained model checkpoints made available on Hugging Face.

Repository Overview and System Requirements

The software package includes scripts for data acquisition, model configuration, training, and evaluation. According to the documentation, users must note specific dependency and hardware parameters before execution:

• Dependencies: PyTorch version 2.9 or later is required to utilize the Muon optimizer.
• Data Fetching: The setup script retrieves pre-tokenized FineWeb-Edu tokens from JaydenTeoh/fineweb-edu-100bt-gpt2. Using the `--shards N` argument fetches N times 10 billion tokens into the data directory.
• Storage: Configurations train on 31.5 billion tokens, making 4 shards the minimum requirement. The default setting of 5 shards requires approximately 120 GB of disk space during assembly.

Training and Objective Configuration

The repository implements loss functions through `combined_mtp_loss()` located in `src/models/loss.py`, which handles NTP, hard_mtp, and soft_mtp objectives. The distinction between standard and soft MTP relies on the `is_soft` condition, incorporating a KL gradient-recovery technique.

• Command-Line Toggles: Objectives can be adjusted via command-line arguments. Direct KL implementations materialize teacher and student softmaxes over the full vocabulary, resulting in higher memory consumption.
• Model Sizes: Four model size configurations used in the underlying paper are provided within the `configs/` directory.
• Metric Tracking: Training progress uses Weights & Biases, logging metrics such as per-head cross-entropy and KL-to-verifier at intervals defined by `eval_every=1000`.

Evaluation and Speculative Decoding

For assessing downstream benchmark performance, the repository integrates the `lm-evaluation-harness`. Users can execute data-parallel evaluations across multiple graphics processing units by prefixing commands with `accelerate launch --num_processes N`.

• Speculative Decoding Metrics: Post-training performance scripts measure metrics including tokens generated per trunk forward pass (`tok/fwd`), per-head acceptance conditional on earlier drafts (`accept_h*`), and wall-clock speedup against a KV-cached autoregressive baseline.
• Prompt Sets: Five standardized prompt sets—comprising fineweb, wiki, books, science, and math—are prepared as fixed 256 by 512 token arrays under `src/data/prompts/` to ensure consistency across evaluation runs.

Verification and Access

Developers and researchers can access the source files, check the exact KL gradient-recovery implementation code, and review citation guidelines directly on the official JaydenTeoh/Soft-MTP GitHub Repository. Performance comparisons should be conducted on identical hardware machines due to hardware-dependent wall-clock speedups.