Multi-Node DGX Spark Clustering & vLLM Deployment Guide

 In Programming

Multi-Node DGX Spark Clustering & vLLM Deployment Guide

Target Architecture: 2x NVIDIA DGX Spark Nodes (Unified 256 GB Memory)
Primary Engine: vLLM with Ray Distributed Backend & Prometheus Monitoring
Target Workload: Qwen3.5 122B (NVFP4 Quantized / Uncompressed)


1. Architectural Architecture & “How It Works”

To successfully run a 122B parameter model across two separate physical machines without devastating performance degradation, the software and hardware must work in absolute synchronization.

The Network Layer: RoCEv2 and Kernel Bypass

In a typical networked environment, data sent from Machine A to Machine B passes through the operating system kernel, network stack, and standard TCP/IP processing. This introduces high latency—the absolute killer of Large Language Model generation speeds.

Your dual-Spark cluster bypasses this completely by utilizing RoCEv2 (RDMA over Converged Ethernet) via the onboard NVIDIA ConnectX-7 (200 Gbps) network interfaces.

  • RDMA (Remote Direct Memory Access) allows the memory of Node 1 to be read or written directly by Node 2 without involving either operating system’s kernel or CPU.
  • ConnectX-7 cards communicate directly over a Direct Attach Copper (DAC) cable, providing a dedicated 200 Gbps pipeline with sub-microsecond latency.

The Compute Layer: Tensor Parallelism (TP=2) via NCCL

Why did the 122B model fail on your single Spark? A single Spark has 128 GB of unified memory. An uncompressed or lightly quantized 122B model requires significantly more than 128 GB just to load weights, let alone account for the dynamic KV Cache (the context memory) during active generation.

By linking two Sparks, you establish a 256 GB unified memory pool. We optimize this using Tensor Parallelism (TP=2):

  • Instead of split-loading layers sequentially (Pipeline Parallelism), Tensor Parallelism slices individual weight matrices within every single layer across both nodes simultaneously.
  • During a forward pass, half of the matrix math occurs on Node 1, and half occurs on Node 2.
  • The NVIDIA Collective Communications Library (NCCL) handles the ultra-fast synchronization of these fractional tensor steps over the 200 Gbps RoCEv2 line, making the two machines behave as a single, giant, distributed GPU.

2. Hardware Configuration & Interconnect

Follow these exact steps to link the physical hardware before powering on the systems.

  1. The Physical Link: Take your QSFP56 / QSFP112 DAC cable and plug it into the left-most ConnectX-7 network port on Node 1. Plug the other end into the exact same left-most port on Node 2.
  2. Management Network: Ensure both systems are plugged into your local office network/switch via their standard RJ45 base management ports.
  3. Power On: Power on both systems. Ensure they are running identical versions of DGX OS.

3. Operating System & Interface Provisioning

Run these commands natively on the host OS of both nodes. We will designate one node as the Head Node and the other as the Worker Node.

Step 3.1: Configure RoCEv2 Static IPs

You must assign static IP addresses to the ConnectX-7 ports to ensure proper point-to-point routing. Replace mlx5_0 with your active CX7 network interface name if it differs.

On Node 1 (Head Node):

sudo ip addr add 192.168.10.1/24 dev mlx5_0
sudo ip link set dev mlx5_0 up

On Node 2 (Worker Node):

sudo ip addr add 192.168.10.2/24 dev mlx5_0
sudo ip link set dev mlx5_0 up

Step 3.2: Initialize Nodes via NVIDIA Sync

  1. Open NVIDIA Sync on your administrative workstation.
  2. Allow it to discover both nodes via mDNS, or manually input their management IPs.
  3. Ensure the exact same username is present on both machines. NVIDIA Sync will configure passwordless SSH between Node 1 and Node 2.
  4. Run the topology auto-mapping utility:
   /opt/nvidia/spark/scripts/discover-sparks.sh

4. Setting Up the Ray Distributed Cluster

vLLM utilizes Ray to coordinate multi-node orchestration.

Step 4.1: Start the Head Node (Node 1)

Run the following command on Node 1 to initialize the cluster manager. Note the inclusion of the Prometheus monitoring port flag.

ray start --head \
          --node-ip-address=192.168.10.1 \
          --port=6379 \
          --dashboard-host=0.0.0.0 \
          --metrics-export-port=8080

Take note of the token token outputted in your terminal after running this command.

Step 4.2: Join the Worker Node (Node 2)

Run this command on Node 2 to connect it to the head node.

ray start --address='192.168.10.1:6379' \
          --node-ip-address=192.168.10.2

5. Deploying Qwen3.5 122B via vLLM

With the Ray cluster active, you can now launch your multi-node model instance. This command leverages NCCL over your RoCEv2 network and enables Prometheus metrics extraction.

Run this from the Head Node (Node 1):

export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0
export NCCL_SOCKET_IFNAME=mlx5_0

vllm serve "Qwen/Qwen3.5-122B-Instruct-NVFP4" \
    --tensor-parallel-size 2 \
    --pipeline-parallel-size 1 \
    --worker-use-ray \
    --distributed-executor-backend ray \
    --host 0.0.0.0 \
    --port 8000 \
    --disable-log-requests \
    --max-model-len 32768

Key Parameter Breakdown:

  • --tensor-parallel-size 2: Shards the 122B model matrices perfectly in half across both DGX Sparks.
  • --worker-use-ray: Commands vLLM to utilize the active multi-node Ray architecture we provisioned in Step 4.
  • NCCL_IB_DISABLE=0: Explicitly forces NCCL to utilize InfiniBand/RoCEv2 verbs over your DAC link, maximizing speed.

6. Accessing Built-In Prometheus Tooling

vLLM natively exports metrics in Prometheus format.

  • Metrics Endpoint: The metrics will be served on port 8000 under the path /metrics (e.g., http://192.168.10).
  • Scraping Configuration: Add the following target stanza to your existing prometheus.yml file to begin collecting live performance charts:
scrape_configs:
  - job_name: 'vllm-dual-spark'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.10.1:8000']

Vital Metrics to Monitor for Your 122B Model:

  • vllm:num_requests_waiting: If this spikes high, your system is bottlenecked by user traffic queries.
  • vllm:gpu_cache_usage_factor: Monitors how much KV cache headroom you have left in your 256GB unified pool. Keep this below 0.90 for peak efficiency.
  • vllm:time_to_first_token_seconds: Monitors prompt processing latency.
  • vllm:time_per_output_token_seconds: Shows real-time token-per-second generation speeds.

About the author

Greg Phillips is the founder and principal engineer of Think IT Data Solutions LLC. He has been building and operating physical infrastructure since 1984 — industrial control systems, the U.S. Navy Advanced Electronics Program, satellite communications at Comsat World Systems and Intelsat, and terrestrial transport and video distribution networks since. He founded Think IT Data Solutions in 2013.

Think IT Data Solutions builds and operates the layers underneath AI workloads — power and facilities, network transport, and GPU compute — for organizations that can’t run their most sensitive work on shared infrastructure.

Building something like this and want a second set of eyes on it? greg@thinkitdata.com · 678-643-4065

Recent Posts

Start typing and press Enter to search