An independent developer has reported training a 3.8-billion-parameter language model from random initialization for $998, presenting the project as evidence that meaningful pretraining experiments are becoming accessible outside major AI laboratories.
According to the developer’s technical account, the model processed 65 billion tokens and reached a score of 0.384 on the CORE evaluation suite. The complete run took 43 hours, after preliminary development and debugging on a GeForce RTX 5090 and final training on rented Nvidia B200 accelerators. The figures are self-reported and should be read as one project’s engineering results rather than an independently reproduced benchmark.
The model uses a decoder-only, Llama-style architecture with RMSNorm, rotary position embeddings, grouped-query attention and other contemporary components. Its training system, called little-lm, is configured through YAML files so datasets, optimizers, schedules and callbacks can be exchanged without rewriting the main training code. The author argues that this modularity shortened the cycle of diagnosing failed experiments.
A smaller baseline demonstrated why that iteration mattered. An earlier 858-million-parameter model was trained for 5.8 days on a single A100 using FineWeb-Edu, yet scored 60.45 percent on PIQA—below the roughly 63 percent result cited for the much smaller GPT-2 model. Its generated text was also described as repetitive and frequently incoherent.
The final recipe changed the learning-rate schedule, split optimization duties between Muon and AdamW, moved to the ClimbMix dataset, used FP8 matrix operations and padded the vocabulary for tensor-core efficiency. It also reduced training context from 2,048 to 1,024 tokens, allowing a larger batch within available memory. A repeat at 2,048 tokens produced the reported 0.384 score, with the author saying context-sensitive tasks explained much of the difference.
During steady-state training, the system reportedly processed about 480,000 tokens per second. CORE evaluations added substantial overhead, while the GPUs recorded high streaming-multiprocessor activity. The project used ordinary DistributedDataParallel rather than a sharded optimizer because gradient communication was not the bottleneck at this model size on a single node.
Several local optimizations also emerged from testing on the RTX 5090. FP8 computation, vocabulary padding and a fused linear cross-entropy implementation traded per-step behavior and memory consumption for higher overall throughput. The account emphasizes that optimizing total training progress matters more than choosing the fastest isolated operation.
The experiment does not approach frontier-model scale, and its results depend on the chosen data and evaluations. It nevertheless supplies a detailed cost and performance point for researchers exploring the widening middle ground between tutorial-scale models and training runs that require institutional budgets.
