Skip to main content
Language model

SOLAR 10.7B v1.0

Upstage

Released
December 2023
Input
Text
Output
Text

SOLAR 10.7B v1.0 is Upstage’s compact open checkpoint, and its training method is the main reason to study it. Depth Upscaling changes the architecture of a 7B starting model, expands its layers, and continues pretraining with additional data. Upstage used the approach to target more capacity below the then-common 30B class without releasing an equally large dense checkpoint.

SOLAR 10.7B v1.0 is a base model with 10.7 billion parameters and a 4,096-token context. It has no instruction post-training for chat conversations.

Specifications and access

SpecificationValue and source
Weights
Open weightsSource
Parameters
10.7 billionSource
Context window
4,096 tokensSource
License
Apache 2.0Source
Inputs
TextSource
Outputs
TextSource