TA的每日心情 | 开心
2020-4-8 10:45 |
|---|
签到天数: 227 天 [LV.7]分神
|
- x9 Y' u- L9 }9 {在论文里，这是第3.2.2节的内容! t+ R0 u6 d2 G1 }1 ~3 P0 o
1 A/ ?. Q3 X0 K7 F2 Q3 E, E
3.2.2. Efficient Implementation of Cross-Node All-to-All Communication# u. `) ~: n$ j g* ~
In order to ensure sufficient computational performance for DualPipe, we customize efficient
, ]2 |" Z% o/ ]- \* L( m/ k8 ~! _% Mcross-node all-to-all communication kernels (including dispatching and combining) to conserve0 d- q) m, c% `0 p( A, [( s
the number of SMs dedicated to communication. The implementation of the kernels is codesigned with the MoE gating algorithm and the network topology of our cluster. To be specific,
- j& n( e% w0 @' \3 k8 b( I$ Xin our cluster, cross-node GPUs are fully interconnected with IB, and intra-node communications4 j- f) F$ ^2 t! f9 @
are handled via NVLink. NVLink offers a bandwidth of 160 GB/s, roughly 3.2 times that of IB6 k+ s0 @- _0 N9 l8 ?! g8 {
(50 GB/s). To effectively leverage the different bandwidths of IB and NVLink, we limit each! v* ]' }9 t+ R: ? B3 X( X. `
token to be dispatched to at most 4 nodes, thereby reducing IB traffic. For each token, when its
* Y( Y. ~4 `0 O# H) P# Rrouting decision is made, it will first be transmitted via IB to the GPUs with the same in-node
8 u' o( N& `: @# Nindex on its target nodes. Once it reaches the target nodes, we will endeavor to ensure that it is5 s Q! q* V1 i+ f% o. V
instantaneously forwarded via NVLink to specific GPUs that host their target experts, without+ g. [3 b0 j: }6 [' I* V5 a
being blocked by subsequently arriving tokens. In this way, communications via IB and NVLink
9 y( s6 y8 K' Z% J' c( care fully overlapped, and each token can efficiently select an average of 3.2 experts per node
7 R9 @ b8 i* i9 \ J' Ywithout incurring additional overhead from NVLink. This implies that, although DeepSeek-V32 D8 F& K8 s* E
13* |8 X2 a& T% O. N8 r5 i
selects only 8 routed experts in practice, it can scale up this number to a maximum of 13 experts
' k$ d5 E, G' M/ r y! K0 S5 ?(4 nodes × 3.2 experts/node) while preserving the same communication cost. Overall, under6 N0 I r' ]& X& h5 ]' l# m1 B
such a communication strategy, only 20 SMs are sufficient to fully utilize the bandwidths of IB s! a* u) L: z2 z* ^6 z
and NVLink.
% u4 ] M2 h, f4 EIn detail, we employ the warp specialization technique (Bauer et al., 2014) and partition$ W8 j) v4 e7 |( v
20 SMs into 10 communication channels. During the dispatching process, (1) IB sending, (2)
% O1 A; }3 ~+ KIB-to-NVLink forwarding, and (3) NVLink receiving are handled by respective warps. The
* n* F. L+ z8 _number of warps allocated to each communication task is dynamically adjusted according to the
. q6 D: u/ y0 ]$ ]" r- \actual workload across all SMs. Similarly, during the combining process, (1) NVLink sending,' q l# {' \+ g
(2) NVLink-to-IB forwarding and accumulation, and (3) IB receiving and accumulation are also
6 L9 i. m5 k! \- g: e6 e: I/ _handled by dynamically adjusted warps. In addition, both dispatching and combining kernels
# f# N9 ?" ]3 d- g/ O+ X6 P- O$ Moverlap with the computation stream, so we also consider their impact on other SM computation C" m, V% J/ i
kernels. Specifically, we employ customized PTX (Parallel Thread Execution) instructions and
6 O/ e; |3 Y) G5 ~1 t8 Kauto-tune the communication chunk size, which significantly reduces the use of the L2 cache
6 e/ n: G: u) G0 x* p! d/ N3 Pand the interference to other SMs.
' \$ r; V2 S4 h: }% ~5 w4 P' B/ ]' R: t) }; e( T
通俗一点说，就是为了实现高效的跨节点全面通信。解决的问题本质上和唐家山老师日志里说的双机对拷的场景差不多。一般来说单机多卡之间用nvlink，多机多卡之间依赖IB网络，但nvlink的速率是IB网络的速率的3.2倍，需要通过一些优化来实现更好的传输策略。这是一整套方案。& w: M2 m0 c+ Y* x
) K5 l+ V" ^8 U7 p我的理解，使用PTX在其中，是为了更精准的定制线程执行减少通信块分配传输之间的串扰。. V1 L; @% \4 T7 J
, Z9 o( V/ ^1 e8 k# ?' Z' ^
目的不是为了绕cuda，反而是为了让cuda的效率更高。9 D4 }1 s7 H8 s: r6 v: h
0 @$ g! q9 `' T2 [% K! h* j8 S, f
类比一下，就好比发现网卡驱动在对拷特定内存块的时候会和应用的线程执行出现串行导致效率降低，而绕开操作系统定义的与网卡驱动的接口，直接使用网卡支持的指令集进行了优化。 |
评分
-
查看全部评分
|
雷达卡
楼主
收藏
分享
淘帖
支持
反对
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
抢沙发
千斤顶
显身卡
发表于 2025-2-4 01:15:04