# Benchmark performance on NVIDIA A10 Here are some preview mooncake benchmark results on A10 with up to 2 RDMA NICs. We are currently having some trouble benchmarking `PyNcclConnector` now. For some unknown reasons, it crashes a lot for inter-node disaggregated scenarios. So the benchmark results haven't included the `PyNcclConnector` yet. In addition, we are also coordinating resources to integrate some machines with more RDMA NICs and more advanced GPUs. The official benchmark results will be released in due time. ## Varying tp (input length = 1024, qps = 2, output length =6) | Setting | num_rdma_nic | Successful Requests | Duration (s) | Total Input Tokens | Total Generated Tokens | Req Throughput (req/s) | Output Token Throughput (tok/s) | Total Token Throughput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) | Mean ITL (ms) | Median ITL (ms) | P99 ITL (ms) | |-----------------|--------------|---------------------|--------------|--------------------|------------------------|------------------------|---------------------------------|-------------------------------|----------------|-----------------|--------------|---------------|------------------|--------------|--------------|----------------|-------------| | tp = 1 | 2 | 200 | 99.47 | 201995 | 1200 | 2.01 | 12.06 | 2042.74 | 1056.76 | 635.00 | 4006.59 | 97.08 | 26.94 | 781.91 | 97.01 | 14.05 | 2205.51 | | tp = 2 | 2 | 200 | 98.98 | 201995 | 1200 | 2.02 | 12.12 | 2052.95 | 314.87 | 231.20 | 949.40 | 25.65 | 15.56 | 129.60 | 25.62 | 15.48 | 288.06 | | tp = 4 | 2 | 200 | 98.76 | 201995 | 1200 | 2.03 | 12.15 | 2057.44 | 198.10 | 160.03 | 461.61 | 23.52 | 18.93 | 94.38 | 23.50 | 18.01 | 187.79 | | tp = 1 | 1 | 200 | 99.44 | 201995 | 1200 | 2.01 | 12.07 | 2043.39 | 1071.12 | 631.56 | 4361.02 | 83.93 | 26.93 | 794.75 | 83.86 | 14.13 | 1932.66 | | tp = 2 | 1 | 200 | 98.96 | 201995 | 1200 | 2.02 | 12.13 | 2053.35 | 335.26 | 258.30 | 997.93 | 28.84 | 15.56 | 144.82 | 28.80 | 15.42 | 397.56 | | tp = 4 | 1 | 200 | 98.78 | 201995 | 1200 | 2.02 | 12.15 | 2057.03 | 201.68 | 162.85 | 456.33 | 22.31 | 16.74 | 94.76 | 22.29 | 16.73 | 189.13 | | tp = 1 | TCP | 200 | 99.55 | 201995 | 1200 | 2.01 | 12.05 | 2041.13 | 1414.05 | 766.23 | 6035.36 | 155.01 | 35.28 | 1191.24 | 154.91 | 14.32 | 3148.99 | | tp = 2 | TCP | 200 | 98.97 | 201995 | 1200 | 2.02 | 12.12 | 2053.03 | 333.74 | 251.32 | 954.63 | 28.74 | 15.49 | 161.24 | 28.70 | 15.35 | 393.52 | | tp = 4 | TCP | 200 | 98.78 | 201995 | 1200 | 2.02 | 12.15 | 2056.94 | 205.37 | 162.92 | 463.70 | 21.54 | 16.51 | 94.04 | 21.51 | 16.56 | 170.54 | ## Varying qps (length = 1024, tp = 4, output length =6) |Setting | num_rdma_nic | Successful Requests | Duration (s) | Total Input Tokens | Total Generated Tokens | Req Throughput (req/s) | Output Token Throughput (tok/s) | Total Token Throughput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) | Mean ITL (ms) | Median ITL (ms) | P99 ITL (ms) | |-----------------|--------------|---------------------|--------------|--------------------|------------------------|------------------------|---------------------------------|-------------------------------|----------------|-----------------|--------------|---------------|------------------|--------------|--------------|----------------|-------------| | qps = 2 | 2 | 200 | 98.77 | 201995 | 1200 | 2.02 | 12.15 | 2057.33 | 200.64 | 156.62 | 478.22 | 22.63 | 17.35 | 99.61 | 22.60 | 17.08 | 186.25 | | qps = 4 | 2 | 200 | 49.75 | 201995 | 1200 | 4.02 | 24.12 | 4084.03 | 341.88 | 240.68 | 1430.54 | 38.36 | 18.39 | 313.45 | 38.31 | 17.17 | 588.80 | | qps = 6 | 2 | 200 | 33.44 | 201995 | 1200 | 5.98 | 35.88 | 6075.54 | 851.15 | 501.59 | 3239.89 | 102.51 | 47.67 | 606.77 | 102.34 | 18.35 | 1704.79 | | qps = 8 | 2 | 200 | 27.16 | 201995 | 1200 | 7.36 | 44.19 | 7482.52 | 4835.08 | 5733.45 | 8846.27 | 1276.59 | 1150.11 | 4401.23 | 1274.43 | 48.34 | 20682.35 | | qps = 2 | 1 | 200 | 98.77 | 201995 | 1200 | 2.02 | 12.15 | 2057.31 | 201.77 | 161.53 | 473.44 | 22.13 | 16.52 | 96.18 | 22.11 | 16.51 | 190.40 | | qps = 4 | 1 | 200 | 49.76 | 201995 | 1200 | 4.02 | 24.12 | 4083.83 | 337.31 | 243.38 | 1395.85 | 39.95 | 17.61 | 325.39 | 39.88 | 17.06 | 838.68 | | qps = 6 | 1 | 200 | 33.44 | 201995 | 1200 | 5.98 | 35.88 | 6075.99 | 820.53 | 458.84 | 3169.52 | 83.92 | 30.50 | 663.07 | 83.78 | 17.85 | 1306.32 | | qps = 8 | 1 | 200 | 27.19 | 201995 | 1200 | 7.36 | 44.14 | 7473.44 | 5291.91 | 6160.55 | 9596.56 | 1190.36 | 1040.63 | 4418.66 | 1188.33 | 47.61 | 20815.23 | | qps = 2 | TCP | 200 | 98.76 | 201995 | 1200 | 2.03 | 12.15 | 2057.42 | 207.22 | 160.81 | 511.01 | 22.17 | 16.59 | 94.96 | 22.15 | 16.59 | 181.82 | | qps = 4 | TCP | 200 | 49.79 | 201995 | 1200 | 4.02 | 24.10 | 4081.06 | 355.43 | 252.63 | 1554.91 | 40.15 | 16.92 | 314.28 | 40.09 | 16.66 | 708.50 | | qps = 6 | TCP | 200 | 33.49 | 201995 | 1200 | 5.97 | 35.83 | 6067.71 | 907.74 | 514.85 | 3253.93 | 122.75 | 45.51 | 648.40 | 122.56 | 18.09 | 2282.92 | | qps = 8 | TCP | 200 | 28.39 | 201995 | 1200 | 7.04 | 42.26 | 7156.09 | 6714.57 | 7885.09 | 11787.51 | 1116.06 | 408.32 | 4645.25 | 1114.29 | 46.87 | 21898.03 | ## Varying input length (tp = 4, qps = 2, output length =6) | Setting | num_rdma_nic | Successful Requests | Duration (s) | Total Input Tokens | Total Generated Tokens | Req Throughput (req/s) | Output Token Throughput (tok/s) | Total Token Throughput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) | Mean ITL (ms) | Median ITL (ms) | P99 ITL (ms) | |-----------------|--------------|---------------------|--------------|--------------------|------------------------|------------------------|---------------------------------|-------------------------------|----------------|-----------------|--------------|---------------|------------------|--------------|--------------|----------------|-------------| | 1024 | 2 | 200 | 98.77 | 201995 | 1200 | 2.02 | 12.15 | 2057.32 | 195.47 | 151.55 | 482.84 | 22.83 | 19.27 | 96.55 | 22.81 | 18.12 | 158.16 | | 2048 | 2 | 200 | 99.22 | 406707 | 1200 | 2.02 | 12.09 | 4110.95 | 723.76 | 488.67 | 2941.96 | 67.25 | 18.93 | 632.73 | 67.20 | 17.49 | 1209.54 | | 4096 | 2 | 200 | 117.42 | 818415 | 1200 | 1.70 | 10.22 | 6979.90 | 14616.48 | 18323.82 | 23191.04 | 8042.84 | 7593.16 | 19851.11 | 8040.02 | 65.43 | 93511.26 | | 8192 | 2 | 200 | 247.77 | 1636065 | 1200 | 0.81 | 4.84 | 6608.10 | 75783.36 | 79331.60 | 147544.42 | 16961.27 | 15140.11 | 39278.98 | 16958.32 | 90.01 | 186151.61 | | 1024 | 1 | 200 | 98.77 | 201995 | 1200 | 2.02 | 12.15 | 2057.31 | 201.77 | 161.53 | 473.44 | 22.13 | 16.52 | 96.18 | 22.11 | 16.51 | 190.40 | | 2048 | 1 | 200 | 99.25 | 406707 | 1200 | 2.02 | 12.09 | 4109.96 | 719.43 | 482.02 | 3208.13 | 61.92 | 17.64 | 681.26 | 61.86 | 16.83 | 978.90 | | 4096 | 1 | 200 | 111.88 | 818415 | 1200 | 1.79 | 10.73 | 7326.16 | 20362.10 | 22807.05 | 31853.55 | 5915.16 | 4521.51 | 18739.12 | 5913.18 | 67.03 | 81600.29 | | 8192 | 1 | 200 | 270.01 | 1636065 | 1200 | 0.74 | 4.44 | 6063.79 | 103355.40 | 106546.65 | 172025.11 | 12894.35 | 11027.66 | 35110.13 | 12892.85 | 64.84 | 151774.68 | | 1024 | TCP | 200 | 98.81 | 201995 | 1200 | 2.02 | 12.14 | 2056.44 | 203.32 | 160.83 | 460.90 | 21.81 | 16.96 | 95.27 | 21.78 | 16.91 | 171.80 | | 2048 | TCP | 200 | 99.27 | 406707 | 1200 | 2.01 | 12.09 | 4108.98 | 731.60 | 484.78 | 3213.69 | 68.55 | 17.88 | 639.93 | 68.49 | 17.33 | 1257.45 | | 4096 | TCP | 200 | 118.37 | 818415 | 1200 | 1.69 | 10.14 | 6923.89 | 23735.69 | 27101.97 | 36573.47 | 6386.62 | 5102.00 | 20032.26 | 6384.71 | 69.57 | 92811.27 | | 8192 | TCP | 200 | 278.12 | 1636065 | 1200 | 0.72 | 4.31 | 5886.95 | 106873.23 | 109941.33 | 179781.64 | 13360.87 | 12155.24 | 36022.96 | 13359.20 | 68.01 | 156716.38 |