Impala AI

Technical Blog

Production-grade notes on MoE serving, systems bottlenecks, and practical optimization in vLLM.

Series

Wide‑EP MoE Serving

3 parts

Part 1

Wide‑EP Mixture-of-Experts (MoE) Serving (Part 1/3): Why the Wire Becomes the Bottleneck

Part 1 of 3: build the Wide‑EP communication model from first principles—problem framing, notation, payload sizing, and the core communication-vs-compute time model.

Part 2

Wide‑EP Mixture-of-Experts (MoE) Serving (Part 2/3): Dual-Batch Overlap (DBO), Kernel Crossover, and the Hardware Cliff

Part 2 of 3: convert the model into tuning decisions—Dual-Batch Overlap (DBO), DeepEP low-latency (LL) vs high-throughput (HT) crossover, and where hardware locality boundaries create throughput cliffs.

Part 3

Wide‑EP Mixture-of-Experts (MoE) Serving (Part 3/3): Failure Modes, Load Balancing, and Portability

Part 3 of 3: production hardening for Wide‑EP—failure diagnostics, Expert Parallel Load Balancing (EPLB), Linear-Programming-Based Load Balancer (LPLB), software-stack portability, and final operator decision flow.
Read full reference edition →

Reference Editions

Impala Team

Wide‑EP Mixture-of-Experts (MoE) Serving: Dispatch/Combine, Dual-Batch Overlap (DBO), and Real Scaling Limits

A production guide to multi-node Mixture-of-Experts (MoE) inference in vLLM: model communication vs compute, choose DeepEP low-latency (LL) vs high-throughput (HT) by measurement, and tune Dual-Batch Overlap (DBO) and Expert Parallel Load Balancing (EPLB) for maximum throughput.