10-Expert Sovereign Matrix — MoE Upcycling

Approach

Rather than training a sparse MoE from scratch, we complete dense pretraining first, curate all 10 domain datasets simultaneously, then upcycle once — converting the dense base into a 10-expert MoE with Switch-Transformer-style load-balancing loss and weight perturbation for symmetry breaking.

Full training pipeline

A 9-file FSDP multi-GPU training system: config loading, GQA + QK-norm + SwiGLU MoE model definition, a swappable tokenizer adapter (kept modular given PU-Tok/Hyper-Token's evolving vocab size), async domain-routed data loading, an eval harness, and ZeRO-3 training with a WSD schedule, 8-bit AdamW, W&B logging, and profiling.

Status

Dense pretraining and identity-alignment fine-tuning tooling are built; upcycling begins once dense pretraining and domain dataset curation are complete.