ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage1

NCP-ArchPreview 8.9B - Stage 1

Model collection | Technical report | Training code (coming soon) | Evaluation code

NCP-ArchPreview is a latent-space autoregressive language model developed by The NCP Team at Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University. It learns to predict both the next token and the next concept: a representation spanning a short group of tokens in a learned latent space. Concept predictions guide the token decoder, while generation retains the standard next-token interface.

This is the Stage 1 base-model release, following large-scale pretraining on Dolma 3 Mix. The architecture follows the OLMo 3 7B token-level design and adds a Concept Module, a product-quantized concept vocabulary, and hierarchical residual connections, bringing the total parameter count to approximately 8.94B.

Highlights

• Joint token and concept learning. Next Concept Prediction (NCP) supplies explicit supervision over a latent sequence at one quarter of the token sequence length.
• Pretraining at scale. The report describes training on 5.73T Dolma-3 tokens. Stage 1 reaches the OLMo-3-7B final training loss using 51.3% of its training tokens, corresponding to 1.95x convergence in token budget.
• Stronger Stage 1 results. The report's main evaluation gives an Overall AVG of 49.04, compared with 46.59 for OLMo-3-7B, including +5.99 percentage points on GSM8K and +4.28 points on HumanEval.
• A learned interface for adaptation. Separate experiments adapt the existing concept codebooks and prediction heads, approximately 17M parameters, while keeping the token backbone fixed.

The convergence comparison measures tokens required to reach a reference loss; it does not measure wall-clock training speed or inference throughput.

Architecture

NCP-ArchPreview processes text through three modules:

1. A 16-layer Token Encoder produces contextual token states. Mean pooling over each group of four states forms a continuous concept representation.
2. An 8-layer Concept Module predicts the next concept. Product quantization defines the concept vocabulary using 32 codebooks. Predictions are differentiable weighted combinations of codewords.
3. A 16-layer Token Decoder receives token states and causally aligned concept predictions, then produces the next-token distribution.

Intra-module residual connections mix states across depths. Cross-module residual connections connect Encoder to Concept Module, Encoder to Decoder, and Concept Module to Decoder. Concept feedback is shifted and repeated at token resolution to preserve causality.

tokens -> Token Encoder -> mean pooling -> Concept Module -> concept prediction
                 |                                                |
                 +--------------> Token Decoder <-----------------+
                                       |
                                next-token logits

Property	Configuration
Hugging Face architecture	NCPOlmo3ForCausalLM
Total parameters	Approximately 8.94B
Encoder / Concept Module / Decoder	16 / 8 / 16 causal Transformer layers
Hidden size	4,096
FFN intermediate size	11,008
Attention heads / KV groups	32 / 32
Attention head dimension	128
Vocabulary size	100,278
Maximum training context	8,192 tokens
Token-level attention	4,096-token local window; full attention every fourth layer
Position encoding	RoPE, base 500,000
Activation / normalization	SwiGLU / RMSNorm; layer-wise QK RMSNorm
Concept compression	4 token states per concept
Product quantization	32 codebooks, each with 128 codewords of dimension 128
Parameter precision	BF16

Training

Stage 1 uses Dolma 3 Mix and the staged pretraining framework described in the report. The reported 5.73T-token budget describes the large-scale training run; intermediate checkpoints have consumed only the tokens preceding their saved step.

The model is optimized jointly with three objectives:

• NTP: the standard causal next-token cross-entropy loss.
• NCP: prediction of the next continuous concept through the learned codebooks, with a stop-gradient target.
• VQ: fitting codebook entries to the encoder's concept representations.

The report uses Moonlight Muon for matrix-valued parameters and AdamW for embeddings, biases, and other non-Muon parameters. Its default learning rate is 6e-5, with the OLMo-3-style cosine schedule.

Evaluation

Results below describe the report's final Stage 1 model. Scores are percentages and higher is better; deltas are absolute percentage points.

Metric	OLMo-3-7B Stage 1	NCP-ArchPreview Stage 1	Delta
Overall AVG	46.59	49.04	+2.45
MMLU	62.22	64.80	+2.58
GSM8K	39.27	45.26	+5.99
MATH-500	12.52	14.48	+1.96
HumanEval	27.10	31.38	+4.28
MBPP	34.53	35.91	+1.38
ARC-Challenge	77.99	81.57	+3.58
PIQA	72.25	80.85	+8.60

Results by domain and likelihood evaluation

Domain average	OLMo-3-7B Stage 1	NCP-ArchPreview Stage 1
MMLU family	54.50	56.73
Mathematics	20.79	24.54
Code	25.15	27.79
Multiple-choice STEM	84.47	86.93
Multiple-choice non-STEM	70.08	74.71
GenQA	54.29	54.76

Likelihood is reported separately in bits per UTF-8 byte (BPB), where lower is better.

Likelihood metric	OLMo-3-7B Stage 1	NCP-ArchPreview Stage 1
BPB AVG	0.824	0.811

The complete per-benchmark results are available in Table 1.

Overall AVG is the unweighted mean of the 26 constituent benchmark scores in Table 1, excluding the aggregate MMLU row, domain averages, and BPB results. BPB AVG is computed separately over ten likelihood benchmarks.

Detailed evaluation settings can be found in our technical report.

Checkpoint Trajectory

The final Stage 1 model is available as ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage1. The report also evaluates intermediate checkpoints from 100K to 1.3M training steps. Selected measurements from different training steps are shown below.

Training checkpoint	MMLU	GSM8K	HumanEval	Overall AVG
100K	53.08	23.43	20.05	39.64
300K	59.42	32.75	27.02	44.27
600K	61.27	40.49	26.14	46.06
900K	63.34	42.15	29.55	47.85
1.3M	64.77	45.34	29.40	48.87
Final	64.80	45.26	31.38	49.04

Use the final checkpoint for the main Stage 1 comparison and intermediate checkpoints to study training dynamics. Individual benchmark scores can fluctuate between checkpoints. When using this card for an intermediate release, its own training step identifies the applicable row; the final checkpoint's scores are reference results only.

Quick Start

The checkpoint includes custom Transformers model code. This example uses a single prompt on one CUDA GPU with sufficient memory for the BF16 weights, activations, and cache.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage1"
device = "cuda"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).eval().to(device)

prompt = "The role of hierarchical representations in language modeling is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        use_cache=True,
        pad_token_id=tokenizer.eos_token_id,
    )

continuation = output_ids[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(continuation, skip_special_tokens=True))

Use plain completion or few-shot prompts for this base model. The example is a loading and generation example; benchmark reproduction requires the prompts, sampling configuration, and scorers described in the report. For reproducible runs, pin the model and tokenizer to the same Hub commit with revision. See the inference guide for supported runtime versions and optimized serving.

Intended Use and Limitations

This release supports research on latent-space language modeling, evaluation of pretrained capabilities, analysis of training dynamics, continued pretraining, and concept-based adaptation.

• It is a pretrained base model and has not been aligned as a conversational assistant. It can produce inaccurate, biased, or harmful content.
• The learned concepts are codebook-based latent representations; individual codes are not guaranteed to correspond to human-interpretable concepts.
• The reported training context is 8,192 tokens. This card makes no claim of validated capability beyond that length.
• Results depend on the checkpoint, prompting, sampling, and evaluation implementation. The separate adaptation and recipe-screening experiments use their own reported evaluation results.

Citation

@techreport{ncpteam2026archpreview,
  title = {{NCP-ArchPreview} Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction},
  author = {{The NCP Team}},
  institution = {Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University},
  year = {2026},
  month = sep,
  note = {Technical report dated September 4, 2026}
}

License

The model weights are released under the Apache License 2.0.

Acknowledgements

We thank the OLMo and Dolma teams and the contributors to the open datasets, training libraries, and evaluation tools used in this work.