Upstream sync (#17)

* Revert "[Misc] Add S3 environment variables for better support of MinIO." (vllm-project#17021)

* [misc] tune some env vars for GB200 (vllm-project#16992)

Signed-off-by: youkaichao <youkaichao@gmail.com>

* [INTEL-HPU][v0] Port delayed sampling to upstream (vllm-project#16949)

Signed-off-by: Michal Adamczyk <michal.adamczyk@intel.com>
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Co-authored-by: Michal Adamczyk <madamczyk@habana.ai>

* [doc] add download path tips (vllm-project#17013)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix] Triton FA function takes no keyword arguments (vllm-project#16902)

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

* [V1] Avoid socket errors during shutdown when requests are in in-flight (vllm-project#16807)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [BugFix] llama4 fa3 fix - RuntimeError: scheduler_metadata must have shape (metadata_size) (vllm-project#16998)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* [Misc] Improve readability of get_open_port function. (vllm-project#17024)

Signed-off-by: gitover22 <qidizou88@gmail.com>

* [Bugfix] Fix AssertionError: skip_special_tokens=False is not supported for Mistral tokenizers (vllm-project#16964)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [CI] Run v1/test_serial_utils.py in CI (vllm-project#16996)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* Mistral-format support for compressed-tensors (vllm-project#16803)

Signed-off-by: mgoin <mgoin64@gmail.com>

* Categorize `tests/kernels/` based on kernel type (vllm-project#16799)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Doc] Add top anchor and a note to quantization/bitblas.md (vllm-project#17042)

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

* Ensure that `pid` passed to `kill_process_tree` is `int` for `mypy` (vllm-project#17051)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [CI] Update structured-output label automation (vllm-project#17055)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* Improve Transformers backend model loading QoL (vllm-project#17039)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* `CacheConfig.block_size` should always be `int` when used (vllm-project#17052)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Use `@property` and private field for `data_parallel_rank_local` (vllm-project#17053)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Frontend] Support guidance:no-additional-properties for compatibility with xgrammar (vllm-project#15949)

Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>

* [BugFix][V1] Fix int32 token index overflow when preparing input ids (vllm-project#16806)

* [V1][Spec Decode] Always use argmax for sampling draft tokens  (vllm-project#16899)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [CI/Build] workaround for CI build failure (vllm-project#17070)

Signed-off-by: csy1204 <josang1204@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>

* [Quantization]add prefix for commandA quantized model (vllm-project#17017)

* [Minor] Use larger batch sizes for A100/B100/B200/MI300x (vllm-project#17073)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [Bugfix] Enable V1 usage stats (vllm-project#16986)

Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>

* More informative error when using Transformers backend (vllm-project#16988)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Addendum Fix to support FIPS enabled machines with MD5 hashing (vllm-project#17043)

Signed-off-by: sydarb <areebsyed237@gmail.com>

* [Bugfix][Core] add seq_id_to_seq_group clearing to avoid memory leak when s… (vllm-project#16472)

Signed-off-by: 开哲 <kaizhe.zy@alibaba-inc.com>
Co-authored-by: 开哲 <kaizhe.zy@alibaba-inc.com>

* [V1] Update structured output (vllm-project#16812)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [doc] update to hyperlink (vllm-project#17096)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* Add docs for runai_streamer_sharded (vllm-project#17093)

Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* [Chore] Remove Sampler from Model Code (vllm-project#17084)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* Disable enforce_eager for V1 TPU sampler and structured output tests (vllm-project#17016)

Signed-off-by: mgoin <mgoin64@gmail.com>

* Simplify `TokenizerGroup` (vllm-project#16790)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Fix OOT registration test (vllm-project#17099)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [V1][PP] Optimization: continue scheduling prefill chunks (vllm-project#17080)

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>

* [Misc] Remove OLMo2 config copy (vllm-project#17066)

Signed-off-by: Isotr0py <2037008807@qq.com>

* Improve static type checking in `LoRAModelRunnerMixin` (vllm-project#17104)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [V1][Structured Output] Clear xgrammar compiler object when engine core shut down to avoid nanobind leaked warning (vllm-project#16954)

Signed-off-by: shen-shanshan <467638484@qq.com>

* [Frontend] Using matryoshka_dimensions control the allowed output dimensions. (vllm-project#16970)

* Add missing rocm_skinny_gemms kernel test to CI (vllm-project#17060)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Misc] refactor example series - structured outputs (vllm-project#17040)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_position metrics (vllm-project#16665)

Signed-off-by: Mark McLoughlin <markmc@redhat.com>

* [CI] Add automation for the `tool-calling` github label (vllm-project#17118)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* Updating builkite job for IBM Power  (vllm-project#17111)

Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com>

* existing torch installation pip command fix for docs (vllm-project#17059)

* Molmo Requirements (vllm-project#17026)

Signed-off-by: Eyshika Agarwal <eyshikaengineer@gmail.com>
Signed-off-by: eyshika <eyshikaengineer@gmail.com>

* Add `:markdownhelp:` to `EngineArgs` docs so markdown docstrings render properly (vllm-project#17124)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Improve configs - `LoRAConfig` + `PromptAdapterConfig` (vllm-project#16980)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Docs] Generate correct github links for decorated functions (vllm-project#17125)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* Add collective_rpc to llm engine (vllm-project#16999)

Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>

* Add chat template for Llama 4 models (vllm-project#16428)

Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

* [Misc] Add example to run DeepSeek with Ray Serve LLM (vllm-project#17134)

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>

* Better error message for missing mistral params.json (vllm-project#17132)

Signed-off-by: mgoin <mgoin64@gmail.com>

* Use custom address for listening socket (vllm-project#15988)

Signed-off-by: Jens Glaser <glaserj@ornl.gov>

* [FEAT] [ROCm]: AITER Fused MOE V1 Support (vllm-project#16752)

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>

* [Attention] FA3 decode perf improvement - single mma warp group support for head dim 128 (vllm-project#16864)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* fix float16 support for kimi-vl (vllm-project#17156)

Co-authored-by: zhouzaida <zhouzaida@msh.team>

* [Doc] V1 : Update LoRA status (vllm-project#17133)

Signed-off-by: varun sundar rabindranath <vsundarr@redhat.com>
Co-authored-by: varun sundar rabindranath <vsundarr@redhat.com>

* [Docs] Fix True->true in supported_models.md (vllm-project#17141)

* Move missed `SchedulerConfig` args into scheduler config group in `EngineArgs` (vllm-project#17131)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Misc] Clean up redundant code in uniproc_executor.py (vllm-project#16762)

Signed-off-by: Lifu Huang <lifu.hlf@gmail.com>

* [Bugfix][Misc] Use TritonPlaceholderModule to defensively import triton (vllm-project#15099)

Signed-off-by: Mengqing Cao <cmq0113@163.com>

* [Misc] Benchmark Serving Script Support Appending Results (vllm-project#17028)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* [Perf]Optimize rotary_emb implementation to use Triton operator for improved inference performance (vllm-project#16457)

Signed-off-by: cynthieye <yexin93@qq.com>
Co-authored-by: MagnetoWang <magnetowang@outlook.com>

* [Bugfix] remove fallback in guided_json (int range, patterns) (vllm-project#16725)

Signed-off-by: csy1204 <josang1204@gmail.com>
Co-authored-by: 조상연[플레이스 AI] <sang-yeon.cho@navercorp.com>

* [Quantization][FP8] Add support for FP8 models with input_scale for output projection and QK quantization (vllm-project#15734)

Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Co-authored-by: Luka Govedič <lgovedic@redhat.com>

* [Doc] Add headings to improve gptqmodel.md (vllm-project#17164)

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

* Only turn on FastIncrementalDetokenizer when tokenizers >= 0.21.1 (vllm-project#17158)

* [Doc] Add two links to disagg_prefill.md (vllm-project#17168)

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

* [Doc] Move todo out of beam search docstring (vllm-project#17183)

Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>

* [Bugfix] Fix mistral model tests (vllm-project#17181)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Bugfix] Fix Mistral ChatCompletionRequest Body Exception (vllm-project#16769)

Signed-off-by: Jasmond Loh <Jasmond.Loh@hotmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* Bump Transformers to 4.51.3 (vllm-project#17116)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Use Transformers helper `get_text_config()` instead of checking for `text_config` (vllm-project#17105)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [doc] update wrong hf model links (vllm-project#17184)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Misc] Inline Molmo requirements (vllm-project#17190)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Security] Use safe serialization and fix zmq setup for mooncake pipe (vllm-project#17192)

Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com>

* [V1] Move usage stats to worker and start logging TPU hardware (vllm-project#16211)

* [Bugfix] Fix hybrid model tests (vllm-project#17182)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Fix Python packaging edge cases (vllm-project#17159)

Signed-off-by: Christian Heimes <christian@python.org>

* [BugFix][Frontend] Fix `LLM.chat()` tokenization (vllm-project#16081)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [V1][Spec Decode] EAGLE-3 Support (vllm-project#16937)

Signed-off-by: Bryan Lu <yuzhelu@amazon.com>
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Co-authored-by: Bryan Lu <yuzhelu@amazon.com>

* [Misc] Refine ray_serve_deepseek example (vllm-project#17204)

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>

* [Bugfix] gemma[2,3] interleaved attention when sliding window is disabled (vllm-project#17180)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [AMD][FP8][BugFix] Remove V1 check in arg_utils.py for FP8 since it is not necessary (vllm-project#17215)

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

* [v1] [P/D] Adding LMCache KV connector for v1 (vllm-project#16625)

* [Bugfix] [pytorch] Patch AOTAutogradCache._get_shape_env (vllm-project#17142)

Signed-off-by: James Wu <jjwu@meta.com>

* [MISC][AMD] Add unused annotation to rocm kernel file (vllm-project#17097)

Signed-off-by: Lu Fang <lufang@fb.com>

* [doc] add Anything LLM integration (vllm-project#17216)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Minor][Spec Decode] Add use_eagle to SpeculativeConfig (vllm-project#17213)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [Doc] Minor fix for the vLLM TPU setup page (vllm-project#17206)

Signed-off-by: Yarong Mu <ymu@google.com>

* [Minor][Models] Fix Return Types of Llama & Eagle (vllm-project#17220)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* Allocate kv_cache with stride order (vllm-project#16605)

Signed-off-by: shuw <shuw@nvidia.com>

* [ROCm][Misc] Follow-ups for Skinny Gemms on ROCm. (vllm-project#17011)

Signed-off-by: charlifu <charlifu@amd.com>

* [V1][Metrics] Allow V1 AsyncLLM to use custom logger (vllm-project#14661)

Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>

* [BugFix] Avoid race conditions in zero-copy tensor transmission (vllm-project#17203)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [CI/test] Fix Eagle Correctness Test (vllm-project#17209)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [Core] Remove prompt string from engine core data structures (vllm-project#17214)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Bugfix] Fix missing int type for `-n` in multi-image example (vllm-project#17223)

* [Bugfix] Fix standard models tests (vllm-project#17217)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Hardware][Intel-Gaudi] Update hpu-extension and update bucketing system for HPU device (vllm-project#17186)

Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>

* [V1] Add `structural_tag` support using xgrammar (vllm-project#17085)

* [BUGFIX] use random for NONE_HASH only when PYTHONHASHSEED not set (vllm-project#17088)

Signed-off-by: Andy Xie <andy.xning@gmail.com>

* [Chore] added stubs for `vllm_flash_attn` during development mode (vllm-project#17228)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [Docs] Update structured output doc for V1 (vllm-project#17135)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [Bugfix] fix error due to an uninitialized tokenizer when using `skip_tokenizer_init` with `num_scheduler_steps` (vllm-project#9276)

Signed-off-by: changjun.lee <pord7457@gmail.com>

* Disable the torch.compile cache checks when VLLM_DISABLE_COMPILE_CACHE=1 (vllm-project#16573)

Signed-off-by: Lu Fang <lufang@fb.com>

* [MISC] rename interval to max_recent_requests (vllm-project#14285)

* [Bugfix] Fix Qwen2.5-Omni M-RoPE position ids generation (vllm-project#16878)

Signed-off-by: imkero <kerorek@outlook.com>

* [Minor] Fix lint error in main branch (vllm-project#17233)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

* [CI/Build] remove -t for run-lm-eval-gsm-hf-baseline.sh (vllm-project#16271)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* Update test_flash_attn.py (vllm-project#17102)

Signed-off-by: ShuaibinLi <lishuaibin@live.cn>

* [Kernel][Triton][FP8] Adding fp8 and variable length sequence support to Triton FAv2 kernel (vllm-project#12591)

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

* [Misc] Make cached tokenizer pickle-compatible (vllm-project#17048)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Bugfix] Fix QWen2 VL multimodal mapping (vllm-project#17240)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Bugfix] Get a specific type of layer from forward context (vllm-project#17222)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [MISC] Use string annotation types for class definitions (vllm-project#17244)

Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>

* [Misc] Change buckets of histogram_iteration_tokens to [1, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8096] to represent number of tokens (vllm-project#17033)

Signed-off-by: sfc-gh-zhwang <flex.wang@snowflake.com>

* [Bugfix] Fix Lora Name Parsing (vllm-project#17196)

Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

* [NVIDIA] Support Cutlass MLA for Blackwell GPUs (vllm-project#16032)

Signed-off-by: kaixih <kaixih@nvidia.com>

* [Feature] support sequence parallelism using compilation pass (vllm-project#16155)

Signed-off-by: cascade812 <cascade812@outlook.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>

* [doc] Add feature status legend (vllm-project#17257)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Metrics] Fix minor inconsistencies in bucket progression (vllm-project#17262)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [V1][Spec Decode] Make eagle compatible with prefix caching. (vllm-project#17137)

Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>

* [BugFix] Fix vllm_flash_attn install issues (vllm-project#17267)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz>

* [Bugfix] Fix missing ARG in Dockerfile for arm64 platforms (vllm-project#17261)

Signed-off-by: lkm-schulz <44176356+lkm-schulz@users.noreply.github.com>

* [Bugfix] Fix cutlass dispatch for fp8/int8 to properly invoke M<=16 c… (vllm-project#16751)

Signed-off-by: Ther-LF <2639852836@qq.com>

* [Bugfix] Fix Mistral3 spatial merge error (vllm-project#17270)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Doc] Fix wrong github link in LMCache examples (vllm-project#17274)

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

* [Doc] small fix (vllm-project#17277)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Misc] Validate `stop_token_ids` contents (vllm-project#17268)

Signed-off-by: Nick Hill <nhill@redhat.com>

* [Minor][Models] Pass partial_rotary_factor parameter to rope (vllm-project#17266)

Signed-off-by: evian <eviantai@u.nus.edu>
Co-authored-by: evian <eviantai@u.nus.edu>

* [Core] Remove legacy input mapper/processor from V0 (vllm-project#15686)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Model] Add Granite Speech Support (vllm-project#16246)

Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>

* Update tpu_worker.py 's typo (vllm-project#17288)

* Add missing class docstring for `PromptAdapterConfig` (vllm-project#17302)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Bugfix] Add missing `get_language_model` to new MLLMs (vllm-project#17300)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [doc] update wrong model id (vllm-project#17287)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Misc] Minor typo/grammar in `platforms/interface.py` (vllm-project#17307)

Signed-off-by: NickLucche <nlucches@redhat.com>

* [Misc] Clean up Qwen2.5-Omni code (vllm-project#17301)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Docs] Add a security guide (vllm-project#17230)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* Improve conversion from dataclass configs to argparse arguments (vllm-project#17303)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Make name of `compressed-tensors` quant method consistent across vLLM (vllm-project#17255)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Explicitly explain quant method override ordering and ensure all overrides are ordered (vllm-project#17256)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Security] Don't bind tcp zmq socket to all interfaces (vllm-project#17197)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [Chore] cleanup license indicators in light of SPDX (vllm-project#17259)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Russell Bryant <rbryant@redhat.com>

* [BugFix] Fix cascade attention - RuntimeError: scheduler_metadata must have shape (metadata_size) (vllm-project#17283)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* [Bugfix] Fix moe weight losing all extra attrs after `process_weights_after_loading`. (vllm-project#16854)

Signed-off-by: charlifu <charlifu@amd.com>

* [Model] Qwen3 Dense FP8 Compat Fixes (vllm-project#17318)

Signed-off-by: simon-mo <xmo@berkeley.edu>

* Support loading transformers models with named parameters (vllm-project#16868)

Signed-off-by: Alex <alexwu@character.ai>

* [Model] Add tuned triton fused_moe configs for Qwen3Moe (vllm-project#17328)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Benchmark] Add single turn MTBench to Serving Bench (vllm-project#17202)

* [Optim] Compute multimodal hash only once per item (vllm-project#17314)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* implement Structural Tag with Guidance backend (vllm-project#17333)

Signed-off-by: Michal Moskal <michal@moskal.me>

* [V1][Spec Decode] Make Eagle model arch config driven (vllm-project#17323)

* [model] make llama4 compatible with pure dense layers (vllm-project#17315)

Signed-off-by: Lucia Fang <fanglu@fb.com>

* [Bugfix] Fix `numel()` downcast in fused_layernorm_dynamic_per_token_quant.cu (vllm-project#17316)

* Ignore `'<string>'` filepath (vllm-project#17330)

Signed-off-by: rzou <zou3519@gmail.com>

* [Bugfix] Add contiguous call inside rope kernel wrapper (vllm-project#17091)

Signed-off-by: 苏政渊 <suzhengyuan@moonshot.cn>
Co-authored-by: 苏政渊 <suzhengyuan@moonshot.cn>

* [Misc] Add a Jinja template to support Mistral3 function calling (vllm-project#17195)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Model] support MiniMax-VL-01 model (vllm-project#16328)

Signed-off-by: qingjun <qingjun@minimaxi.com>

* [Misc] Move config fields to MultiModalConfig (vllm-project#17343)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Misc]Use a platform independent interface to obtain the device attributes (vllm-project#17100)

* [Fix] Documentation spacing in compilation config help text (vllm-project#17342)

Signed-off-by: Zerohertz <ohg3417@gmail.com>

* [Build][Bugfix] Restrict setuptools version to <80 (vllm-project#17320)

Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>

* [Model] Ignore rotary embed load for Cohere model (vllm-project#17319)

* Update docs requirements (vllm-project#17379)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Doc] Fix QWen3MOE info (vllm-project#17381)

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

* [Bugfix] Clean up MiniMax-VL and fix processing (vllm-project#17354)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* `pre-commit autoupdate` (vllm-project#17380)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Frontend] Support `chat_template_kwargs` in `LLM.chat` (vllm-project#17356)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* Transformers backend tweaks (vllm-project#17365)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Fix: Spelling of inference (vllm-project#17387)

* Improve literal dataclass field conversion to argparse argument (vllm-project#17391)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [V1] Remove num_input_tokens from attn_metadata (vllm-project#17193)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* [Bugfix] add qwen3 reasoning-parser fix content is None when disable … (vllm-project#17369)

Signed-off-by: mofanke <mofanke@gmail.com>

* fix gemma3 results all zero (vllm-project#17364)

Signed-off-by: mayuyuace <qiming1.zhang@intel.com>

* [Misc][ROCm] Exclude `cutlass_mla_decode` for ROCm build (vllm-project#17289)

Signed-off-by: Tianyuan Wu <Tianyuan.Wu@amd.com>

* Enabling multi-group kernel tests. (vllm-project#17115)

Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>

* [Docs] Propose a deprecation policy for the project (vllm-project#17063)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* [Doc][Typo] Fixing label in new model requests link in overview.md (vllm-project#17400)

* [TPU][V1][CI] Replace `python3 setup.py develop` with standard `pip install --e` on TPU (vllm-project#17374)

Signed-off-by: NickLucche <nlucches@redhat.com>

* [CI] Uses Python 3.11 for TPU (vllm-project#17359)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [CI/Build] Add retry mechanism for add-apt-repository (vllm-project#17107)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [Bugfix] Fix Minicpm-O-int4 GPTQ model inference (vllm-project#17397)

Signed-off-by: Isotr0py <2037008807@qq.com>

* Simplify (and fix) passing of guided decoding backend options (vllm-project#17008)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Remove Falcon3 2x7B from CI (vllm-project#17404)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Fix: Python package installation for opentelmetry (vllm-project#17049)

Signed-off-by: Dilip Gowda Bhagavan <dilip.bhagavan@ibm.com>

* [V1][Spec Decode] Apply torch.compile & cudagraph to EAGLE (vllm-project#17211)

Signed-off-by: Bryan Lu <yuzhelu@amazon.com>

* Remove Bamba 9B from CI (vllm-project#17407)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [V1][Feature] Enable Speculative Decoding with Structured Outputs (vllm-project#14702)

Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>

* [release] Always git fetch all to get latest tag on TPU release (vllm-project#17322)

* Truncation control for embedding models (vllm-project#14776)

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Max de Bayser <mbayser@br.ibm.com>

* Update PyTorch to 2.7.0 (vllm-project#16859)

* Improve configs - `ModelConfig` (vllm-project#17130)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Fix call to `logger.info_once` (vllm-project#17416)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* Fix some speculative decode tests with tl.dot (vllm-project#17371)

Signed-off-by: Huy Do <huydhn@gmail.com>

* Support LoRA for Mistral3 (vllm-project#17428)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [Intel GPU] [CI]Fix XPU ci, setuptools >=80.0 have build issue (vllm-project#17298)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* [Hardware][Intel GPU] Upgrade to torch 2.7 (vllm-project#17444)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Qiming Zhang <qiming1.zhang@intel.com>

* [Bugfix] Fix AttributeError: 'State' object has no attribute 'engine_client' (vllm-project#17434)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [MODEL ADDITION] Ovis2 Model Addition (vllm-project#15826)

Signed-off-by: Marco <121761685+mlinmg@users.noreply.github.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

* Make the _apply_rotary_emb compatible with dynamo (vllm-project#17435)

* [Misc] Remove deprecated files (vllm-project#17447)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [V1][Bugfix]: vllm v1 verison metric num_gpu_blocks is None (vllm-project#15755)

Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>

* [TPU][V1][CI] Update regression test baseline for v6 CI (vllm-project#17064)

Signed-off-by: NickLucche <nlucches@redhat.com>

* [Core] Prevent side-channel attacks via cache salting (vllm-project#17045)

Signed-off-by: Marko Rosenmueller <5467316+dr75@users.noreply.github.com>

* [V1][Metrics] add support for kv event publishing (vllm-project#16750)

Signed-off-by: alec-flowers <aflowers@nvidia.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>

* [Feature] The Qwen3 reasoning parser supports  guided decoding (vllm-project#17466)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [Docs] Add command for running mypy tests from CI (vllm-project#17475)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [Fix] Support passing args to logger (vllm-project#17425)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* [Bugfix] Fixed mistral tokenizer path when pointing to file (vllm-project#17457)

Signed-off-by: Pete Savage <psavage@redhat.com>

* [V1] Allow turning off pickle fallback in vllm.v1.serial_utils (vllm-project#17427)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

* [Docs] Update optimization.md doc (vllm-project#17482)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [BugFix] Fix authorization of openai_transcription_client.py (vllm-project#17321)

Signed-off-by: zh Wang <rekind133@outlook.com>

* [Bugfix][ROCm] Restrict ray version due to a breaking release (vllm-project#17480)

Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>

* [doc] add install tips (vllm-project#17373)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* doc: fix bug report Github template formatting (vllm-project#17486)

Signed-off-by: David Xia <david@davidxia.com>

* [v1][Spec Decode] Make sliding window compatible with eagle prefix caching (vllm-project#17398)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>

* Bump Compressed Tensors version to 0.9.4 (vllm-project#17478)

Signed-off-by: Rahul Tuli <rtuli@redhat.com>
Co-authored-by: mgoin <mgoin64@gmail.com>

* [Misc] Rename Audios -> Audio in Qwen2audio Processing (vllm-project#17507)

Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>

* [CI][TPU] Skip Multimodal test (vllm-project#17488)

Signed-off-by: Siyuan Liu <lsiyuan@google.com>

* [Bugfix][ROCm] Fix import error on ROCm (vllm-project#17495)

Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>

* [Bugfix] Temporarily disable gptq_bitblas on ROCm (vllm-project#17411)

Signed-off-by: Yan Cangang <nalanzeyu@gmail.com>

* [CI][TPU] Skip structured outputs+spec decode tests on TPU (vllm-project#17510)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [CI][Bugfix] Fix failing V1 Test due to missing 'cache_salt' arg (vllm-project#17500)

Signed-off-by: mgoin <mgoin64@gmail.com>

* [CI/Build] Reorganize models tests (vllm-project#17459)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* FIxing the AMD test failures caused by PR#16457 (vllm-project#17511)

Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>

* [Build] Require setuptools >= 77.0.3 for PEP 639 (vllm-project#17389)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [ROCm] Effort to reduce the number of environment variables in command line (vllm-project#17229)

Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>

* [BugFix] fix speculative decoding memory leak when speculation is disabled (vllm-project#15506)

Signed-off-by: Noah Yoshida <noahcy117@gmail.com>

* [BugFix] Fix mla cpu - missing 3 required positional arguments (vllm-project#17494)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

* Avoid overwriting vllm_compile_cache.py (vllm-project#17418)

Signed-off-by: Keyun Tong <tongkeyun@gmail.com>

* [Core] Enable IPv6 with vllm.utils.make_zmq_socket() (vllm-project#16506)

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* [Misc] Optimize the Qwen3_ReasoningParser extract_reasoning_content (vllm-project#17515)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* Improve configs - `ObservabilityConfig` (vllm-project#17453)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

* [Bugfix][Benchmarks] Allow benchmark of deepspeed-mii backend to select a model (vllm-project#17285)

Signed-off-by: Teruaki Ishizaki <teruaki.ishizaki@ntt.com>

* [Frontend] Show progress bar for adding requests (vllm-project#17525)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [Misc] Clean up test docstrings and names (vllm-project#17521)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

* [FEAT] [ROCm]: Add Qwen/Qwen3-30B-A3B-FP8 fused moe config for MI300X (vllm-project#17530)

Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>

* Fix more broken speculative decode tests (vllm-project#17450)

Signed-off-by: Huy Do <huydhn@gmail.com>

* [doc] add streamlit integration (vllm-project#17522)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

* [FEAT] [ROCm]: Add Qwen/Qwen3-235B-A22B-FP8 TP4 triton fused moe config (vllm-project#17535)

Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>

* [Feature][Frontend]: Deprecate --enable-reasoning (vllm-project#17452)

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

* [ROCm] remove unsupported archs from rocm triton flash-attention supported list (vllm-project#17536)

Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>

* [torch.compile] Add torch inductor pass for fusing silu_and_mul with subsequent scaled_fp8_quant operations (vllm-project#10867)

Signed-off-by: Sage Moore <sage@neuralmagic.com>

* [Misc] refactor example - cpu_offload_lmcache (vllm-project#17460)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

---------

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Michal Adamczyk <michal.adamczyk@intel.com>
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: gitover22 <qidizou88@gmail.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: csy1204 <josang1204@gmail.com>
Signed-off-by: sydarb <areebsyed237@gmail.com>
Signed-off-by: 开哲 <kaizhe.zy@alibaba-inc.com>
Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: shen-shanshan <467638484@qq.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Signed-off-by: Aaruni Aggarwal <aaruniagg@gmail.com>
Signed-off-by: Eyshika Agarwal <eyshikaengineer@gmail.com>
Signed-off-by: eyshika <eyshikaengineer@gmail.com>
Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Jens Glaser <glaserj@ornl.gov>
Signed-off-by: varun sundar rabindranath <vsundarr@redhat.com>
Signed-off-by: Lifu Huang <lifu.hlf@gmail.com>
Signed-off-by: Mengqing Cao <cmq0113@163.com>
Signed-off-by: cynthieye <yexin93@qq.com>
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Jasmond Loh <Jasmond.Loh@hotmail.com>
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Signed-off-by: Christian Heimes <christian@python.org>
Signed-off-by: Bryan Lu <yuzhelu@amazon.com>
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: James Wu <jjwu@meta.com>
Signed-off-by: Lu Fang <lufang@fb.com>
Signed-off-by: Yarong Mu <ymu@google.com>
Signed-off-by: shuw <shuw@nvidia.com>
Signed-off-by: charlifu <charlifu@amd.com>
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: changjun.lee <pord7457@gmail.com>
Signed-off-by: imkero <kerorek@outlook.com>
Signed-off-by: ShuaibinLi <lishuaibin@live.cn>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
Signed-off-by: sfc-gh-zhwang <flex.wang@snowflake.com>
Signed-off-by: kaixih <kaixih@nvidia.com>
Signed-off-by: cascade812 <cascade812@outlook.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
Signed-off-by: lkm-schulz <44176356+lkm-schulz@users.noreply.github.com>
Signed-off-by: Ther-LF <2639852836@qq.com>
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
Signed-off-by: evian <eviantai@u.nus.edu>
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Signed-off-by: Alex <alexwu@character.ai>
Signed-off-by: Michal Moskal <michal@moskal.me>
Signed-off-by: Lucia Fang <fanglu@fb.com>
Signed-off-by: rzou <zou3519@gmail.com>
Signed-off-by: 苏政渊 <suzhengyuan@moonshot.cn>
Signed-off-by: qingjun <qingjun@minimaxi.com>
Signed-off-by: Zerohertz <ohg3417@gmail.com>
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Signed-off-by: mofanke <mofanke@gmail.com>
Signed-off-by: mayuyuace <qiming1.zhang@intel.com>
Signed-off-by: Tianyuan Wu <Tianyuan.Wu@amd.com>
Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>
Signed-off-by: Dilip Gowda Bhagavan <dilip.bhagavan@ibm.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>
Signed-off-by: Huy Do <huydhn@gmail.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Marco <121761685+mlinmg@users.noreply.github.com>
Signed-off-by: isotr0py <2037008807@qq.com>
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
Signed-off-by: Marko Rosenmueller <5467316+dr75@users.noreply.github.com>
Signed-off-by: alec-flowers <aflowers@nvidia.com>
Signed-off-by: Pete Savage <psavage@redhat.com>
Signed-off-by: zh Wang <rekind133@outlook.com>
Signed-off-by: David Xia <david@davidxia.com>
Signed-off-by: Rahul Tuli <rtuli@redhat.com>
Signed-off-by: Siyuan Liu <lsiyuan@google.com>
Signed-off-by: Yan Cangang <nalanzeyu@gmail.com>
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>
Signed-off-by: Noah Yoshida <noahcy117@gmail.com>
Signed-off-by: Keyun Tong <tongkeyun@gmail.com>
Signed-off-by: Teruaki Ishizaki <teruaki.ishizaki@ntt.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
Co-authored-by: Michal Adamczyk <madamczyk@habana.ai>
Co-authored-by: Reid <61492567+reidliu41@users.noreply.github.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: huafeng <qidizou88@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Michael Yao <haifeng.yao@daocloud.io>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: Yong Hoon Shin <48474650+sarckk@users.noreply.github.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Sangyeon Cho <josang1204@gmail.com>
Co-authored-by: Chen Xia <cxia0209@gmail.com>
Co-authored-by: Areeb Syed <areebsyed237@gmail.com>
Co-authored-by: 张宇 <zhangyuygss@outlook.com>
Co-authored-by: 开哲 <kaizhe.zy@alibaba-inc.com>
Co-authored-by: omer-dayan <omdayan@nvidia.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Rui Qiao <161574667+ruisearch42@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Shanshan Shen <467638484@qq.com>
Co-authored-by: wang.yuqi <noooop@126.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Aaruni Aggarwal <47731267+AaruniAggarwal@users.noreply.github.com>
Co-authored-by: Atilla <48064466+atilla00@users.noreply.github.com>
Co-authored-by: Eyshika Agarwal <eyshikaengineer@gmail.com>
Co-authored-by: Yinghai Lu <yinghai@thinkingmachines.ai>
Co-authored-by: Maximilien de Bayser <mbayser@br.ibm.com>
Co-authored-by: jglaser <glaserj@ornl.gov>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
Co-authored-by: zhouzaida <zhouzaida@msh.team>
Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: varun sundar rabindranath <vsundarr@redhat.com>
Co-authored-by: Lifu Huang <lifu.hlf@gmail.com>
Co-authored-by: Mengqing Cao <cmq0113@163.com>
Co-authored-by: yexin(叶鑫) <yexin93@qq.com>
Co-authored-by: MagnetoWang <magnetowang@outlook.com>
Co-authored-by: 조상연[플레이스 AI] <sang-yeon.cho@navercorp.com>
Co-authored-by: rasmith <Randall.Smith@amd.com>
Co-authored-by: Luka Govedič <lgovedic@redhat.com>
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
Co-authored-by: Alex Brooks <alex.brooks@ibm.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Jasmond L <120363110+JasmondL@users.noreply.github.com>
Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com>
Co-authored-by: Daniel Li <dyli@google.com>
Co-authored-by: Christian Heimes <christian@python.org>
Co-authored-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Co-authored-by: Bryan Lu <yuzhelu@amazon.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Yihua Cheng <jc4xvyp@outlook.com>
Co-authored-by: James Wu <james.jz.wu@gmail.com>
Co-authored-by: yarongmu-google <150371854+yarongmu-google@users.noreply.github.com>
Co-authored-by: Shu Wang <shuw@nvidia.com>
Co-authored-by: Charlie Fu <charlifu@amd.com>
Co-authored-by: Zijing Liu <liuzijing2014@users.noreply.github.com>
Co-authored-by: Agata Dobrzyniewicz <160237065+adobrzyn@users.noreply.github.com>
Co-authored-by: Ning Xie <andy.xning@gmail.com>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: changjun.lee <pord7457@gmail.com>
Co-authored-by: Kero Liang <kerorek@outlook.com>
Co-authored-by: Happy <lsb19@tsinghua.org.cn>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com>
Co-authored-by: Flex Wang <flex.wang@snowflake.com>
Co-authored-by: Kaixi Hou <kaixih@nvidia.com>
Co-authored-by: cascade <guijuzhang11@gmail.com>
Co-authored-by: Lily Liu <lilyliupku@gmail.com>
Co-authored-by: Lennart K. M. Schulz <44176356+lkm-schulz@users.noreply.github.com>
Co-authored-by: TherLF <54900723+Ther-LF@users.noreply.github.com>
Co-authored-by: Kuntai Du <kuntai@uchicago.edu>
Co-authored-by: Wanrui Dai <daiwanrui@u.nus.edu>
Co-authored-by: evian <eviantai@u.nus.edu>
Co-authored-by: idouba <zhangchaomeng@huawei.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Alex Wu <alexwu@character.ai>
Co-authored-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Michał Moskal <michal@moskal.me>
Co-authored-by: Lucia Fang <116399278+luccafong@users.noreply.github.com>
Co-authored-by: Richard Barnes <rbarnes@meta.com>
Co-authored-by: Richard Zou <zou3519@users.noreply.github.com>
Co-authored-by: Zhengyuan Su (苏政渊) <su-zy21@mails.tsinghua.edu.cn>
Co-authored-by: 苏政渊 <suzhengyuan@moonshot.cn>
Co-authored-by: qscqesze <qingjun@minimaxi.com>
Co-authored-by: ponix-j <55234879+ponix-j@users.noreply.github.com>
Co-authored-by: Hyogeun Oh (오효근) <ohg3417@gmail.com>
Co-authored-by: Gregory Shtrasberg <156009573+gshtras@users.noreply.github.com>
Co-authored-by: a2q1p <a2q1p.2025@gmail.com>
Co-authored-by: mofanke <54242816+mofanke@users.noreply.github.com>
Co-authored-by: Qiming Zhang <qiming1.zhang@intel.com>
Co-authored-by: TY-AMD <tianyuan.wu@amd.com>
Co-authored-by: Alexei-V-Ivanov-AMD <156011006+Alexei-V-Ivanov-AMD@users.noreply.github.com>
Co-authored-by: casinca <47400729+casinca@users.noreply.github.com>
Co-authored-by: Dilip Gowda Bhagavan <110233170+dilipgb@users.noreply.github.com>
Co-authored-by: Bryan Lu <55512809+luyuzhe111@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <kevin@anyscale.com>
Co-authored-by: Gabriel Marinho <104592062+gmarinho2@users.noreply.github.com>
Co-authored-by: Huy Do <huydhn@gmail.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Marco <121761685+mlinmg@users.noreply.github.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: rongfu.leng <rongfu.leng@daocloud.io>
Co-authored-by: Marko Rosenmueller <5467316+dr75@users.noreply.github.com>
Co-authored-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Co-authored-by: Pete Savage <psav@users.noreply.github.com>
Co-authored-by: zh Wang <rekind133@outlook.com>
Co-authored-by: David Xia <david@davidxia.com>
Co-authored-by: Rahul Tuli <rtuli@redhat.com>
Co-authored-by: Siyuan Liu <lsiyuan@google.com>
Co-authored-by: NaLan ZeYu <nalanzeyu@gmail.com>
Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com>
Co-authored-by: Noah Yoshida <noahcy117@gmail.com>
Co-authored-by: Keyun Tong <tongkeyun@gmail.com>
Co-authored-by: Teruaki Ishizaki <tell.ishi@gmail.com>
Co-authored-by: Sage Moore <sage@neuralmagic.com>

Author: 121 people

.buildkite/lm-eval-harness/configs/DeepSeek-V2-Lite-Chat.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash ./run-lm-eval-gsm-vllm-baseline.sh -m deepseek-ai/DeepSeek-V2-Lite-Chat -b "auto" -l 1000 -f 5 -t 2
 model_name: "deepseek-ai/DeepSeek-V2-Lite-Chat"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3-70B-Instruct-FBGEMM-nonuniform.yaml

@@ -1,3 +1,4 @@
+# For hf script, without -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m nm-testing/Meta-Llama-3-70B-Instruct-FBGEMM-nonuniform -b auto -l 1000 -f 5
 model_name: "nm-testing/Meta-Llama-3-70B-Instruct-FBGEMM-nonuniform"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3-70B-Instruct.yaml

@@ -1,3 +1,4 @@
+# For hf script, without -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m meta-llama/Meta-Llama-3-70B-Instruct -b 32 -l 250 -f 5
 model_name: "meta-llama/Meta-Llama-3-70B-Instruct"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-Channelwise-compressed-tensors.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-Instruct-W8A8-FP8-Channelwise-compressed-tensors -b auto -l 1000 -f 5 -t 1
 model_name: "nm-testing/Meta-Llama-3-8B-Instruct-W8A8-FP8-Channelwise-compressed-tensors"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-FBGEMM-nonuniform.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-Instruct-FBGEMM-nonuniform -b auto -l 1000 -f 5 -t 1
 model_name: "nm-testing/Meta-Llama-3-8B-Instruct-FBGEMM-nonuniform"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-FP8-compressed-tensors.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-FP8-compressed-tensors-test -b 32 -l 1000 -f 5 -t 1
 model_name: "nm-testing/Meta-Llama-3-8B-FP8-compressed-tensors-test"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-FP8.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m neuralmagic/Meta-Llama-3-8B-Instruct-FP8 -b 32 -l 250 -f 5 -t 1
 model_name: "neuralmagic/Meta-Llama-3-8B-Instruct-FP8"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-INT8-compressed-tensors-asym.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-Instruct-W8-Channel-A8-Dynamic-Asym-Per-Token-Test -b "auto" -l 250 -f 5 -t 1
 model_name: "nm-testing/Meta-Llama-3-8B-Instruct-W8-Channel-A8-Dynamic-Asym-Per-Token-Test"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-INT8-compressed-tensors.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-Instruct-W8-Channel-A8-Dynamic-Per-Token-Test -b "auto" -l 250 -f 5 -t 1
 model_name: "nm-testing/Meta-Llama-3-8B-Instruct-W8-Channel-A8-Dynamic-Per-Token-Test"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-nonuniform-compressed-tensors.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Meta-Llama-3-8B-Instruct-nonuniform-test -b auto -l 1000 -f 5 -t 1
 model_name: "nm-testing/Meta-Llama-3-8B-Instruct-nonuniform-test"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct.yaml

@@ -1,4 +1,5 @@
-# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m meta-llama/Meta-Llama-3-8B-Instruct -b 32 -l 250 -f 5 -t 1
+# For hf script, without -t option (tensor parallel size).
+# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m meta-llama/Meta-Llama-3-8B-Instruct -b 32 -l 250 -f 5
 model_name: "meta-llama/Meta-Llama-3-8B-Instruct"
 tasks:
 - name: "gsm8k"

.buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-QQQ.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m HandH1998/QQQ-Llama-3-8b-g128 -b 32 -l 1000 -f 5 -t 1
 model_name: "HandH1998/QQQ-Llama-3-8b-g128"
 tasks:

.buildkite/lm-eval-harness/configs/Meta-Llama-3.2-1B-Instruct-INT8-compressed-tensors.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m neuralmagic/Llama-3.2-1B-Instruct-quantized.w8a8 -b "auto" -l 1000 -f 5 -t 1
 model_name: "neuralmagic/Llama-3.2-1B-Instruct-quantized.w8a8"
 tasks:

.buildkite/lm-eval-harness/configs/Minitron-4B-Base-FP8.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m mgoin/Minitron-4B-Base-FP8 -b auto -l 1000 -f 5 -t 1
 model_name: "mgoin/Minitron-4B-Base-FP8"
 tasks:

.buildkite/lm-eval-harness/configs/Mixtral-8x22B-Instruct-v0.1-FP8-Dynamic.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash ./run-lm-eval-gsm-vllm-baseline.sh -m neuralmagic/Mixtral-8x22B-Instruct-v0.1-FP8-dynamic -b "auto" -l 250 -f 5 -t 8
 model_name: "neuralmagic/Mixtral-8x22B-Instruct-v0.1-FP8-dynamic"
 tasks:

.buildkite/lm-eval-harness/configs/Mixtral-8x7B-Instruct-v0.1-FP8.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash ./run-lm-eval-gsm-vllm-baseline.sh -m neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8 -b "auto" -l 250 -f 5 -t 4
 model_name: "neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8"
 tasks:

.buildkite/lm-eval-harness/configs/Mixtral-8x7B-Instruct-v0.1.yaml

@@ -1,4 +1,5 @@
-# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m neuralmagic/Mixtral-8x7B-Instruct-v0.1 -b 32 -l 250 -f 5 -t 4
+# For hf script, without -t option (tensor parallel size).
+# bash .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh -m neuralmagic/Mixtral-8x7B-Instruct-v0.1 -b 32 -l 250 -f 5
 model_name: "mistralai/Mixtral-8x7B-Instruct-v0.1"
 tasks:
 - name: "gsm8k"

.buildkite/lm-eval-harness/configs/Qwen1.5-MoE-W4A16-compressed-tensors.yaml

@@ -1,11 +1,12 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Qwen1.5-MoE-A2.7B-Chat-quantized.w4a16 -b auto -l 1319 -f 5 -t 1
 model_name: "nm-testing/Qwen1.5-MoE-A2.7B-Chat-quantized.w4a16"
 tasks:
 - name: "gsm8k"
   metrics:
   - name: "exact_match,strict-match"
-    value: 0.31
+    value: 0.30
   - name: "exact_match,flexible-extract"
-    value: 0.47
+    value: 0.465
 limit: 1319
 num_fewshot: 5

.buildkite/lm-eval-harness/configs/Qwen2-1.5B-Instruct-FP8W8.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Qwen2-1.5B-Instruct-FP8W8 -b auto -l 1000 -f 5 -t 1
 model_name: "nm-testing/Qwen2-1.5B-Instruct-FP8W8"
 tasks:

.buildkite/lm-eval-harness/configs/Qwen2-1.5B-Instruct-INT8-compressed-tensors.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m neuralmagic/Qwen2-1.5B-Instruct-quantized.w8a8 -b "auto" -l 1000 -f 5 -t 1
 model_name: "neuralmagic/Qwen2-1.5B-Instruct-quantized.w8a8"
 tasks:

.buildkite/lm-eval-harness/configs/Qwen2-1.5B-Instruct-W8A16-compressed-tensors.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/Qwen2-1.5B-Instruct-W8A16-Channelwise -b "auto" -l 1000 -f 5 -t 1
 model_name: "nm-testing/Qwen2-1.5B-Instruct-W8A16-Channelwise"
 tasks:

.buildkite/lm-eval-harness/configs/Qwen2-57B-A14-Instruct.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash ./run-lm-eval-gsm-vllm-baseline.sh -m Qwen/Qwen2-57B-A14B-Instruct -b "auto" -l 250 -f 5 -t 4
 model_name: "Qwen/Qwen2-57B-A14B-Instruct"
 tasks:

.buildkite/lm-eval-harness/configs/SparseLlama3.1_2of4_fp8_compressed.yaml

@@ -1,3 +1,4 @@
+# For vllm script, with -t option (tensor parallel size).
 # bash ./run-lm-eval-gsm-vllm-baseline.sh -m nm-testing/SparseLlama-3.1-8B-gsm8k-pruned.2of4-chnl_wts_per_tok_dyn_act_fp8-BitM -b "auto" -t 2
 model_name: "nm-testing/SparseLlama-3.1-8B-gsm8k-pruned.2of4-chnl_wts_per_tok_dyn_act_fp8-BitM"
 tasks:

.buildkite/lm-eval-harness/test_lm_eval_correctness.py

@@ -16,7 +16,7 @@
 import pytest
 import yaml
 
-RTOL = 0.05
+RTOL = 0.08
 TEST_DATA_FILE = os.environ.get(
     "LM_EVAL_TEST_DATA_FILE",
     ".buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct.yaml")

.buildkite/release-pipeline.yaml

@@ -1,20 +1,20 @@
 steps:
-  - label: "Build wheel - CUDA 12.4"
+  - label: "Build wheel - CUDA 12.8"
     agents:
       queue: cpu_queue_postmerge
     commands:
-      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.4.0 --tag vllm-ci:build-image --target build --progress plain -f docker/Dockerfile ."
+      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.8.1 --tag vllm-ci:build-image --target build --progress plain -f docker/Dockerfile ."
       - "mkdir artifacts"
       - "docker run --rm -v $(pwd)/artifacts:/artifacts_host vllm-ci:build-image bash -c 'cp -r dist /artifacts_host && chmod -R a+rw /artifacts_host'"
       - "bash .buildkite/scripts/upload-wheels.sh"
     env:
       DOCKER_BUILDKIT: "1"
 
-  - label: "Build wheel - CUDA 12.1"
+  - label: "Build wheel - CUDA 12.6"
     agents:
       queue: cpu_queue_postmerge
     commands:
-      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.1.0 --tag vllm-ci:build-image --target build --progress plain -f docker/Dockerfile ."
+      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.6.3 --tag vllm-ci:build-image --target build --progress plain -f docker/Dockerfile ."
       - "mkdir artifacts"
       - "docker run --rm -v $(pwd)/artifacts:/artifacts_host vllm-ci:build-image bash -c 'cp -r dist /artifacts_host && chmod -R a+rw /artifacts_host'"
       - "bash .buildkite/scripts/upload-wheels.sh"
@@ -48,7 +48,7 @@ steps:
       queue: cpu_queue_postmerge
     commands:
       - "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
-      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.4.0 --tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT --target vllm-openai --progress plain -f docker/Dockerfile ."
+      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.8.1 --tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT --target vllm-openai --progress plain -f docker/Dockerfile ."
       - "docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT"
 
   - label: "Build and publish TPU release image"
@@ -57,6 +57,7 @@ steps:
     agents:
       queue: tpu_queue_postmerge
     commands:
+      - "git fetch --all"
       - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --tag vllm/vllm-tpu:nightly --tag vllm/vllm-tpu:$BUILDKITE_COMMIT --progress plain -f docker/Dockerfile.tpu ."
       - "docker push vllm/vllm-tpu:nightly"
       - "docker push vllm/vllm-tpu:$BUILDKITE_COMMIT"

.buildkite/scripts/hardware_ci/run-amd-test.sh

@@ -75,30 +75,51 @@ HF_MOUNT="/root/.cache/huggingface"
 commands=$@
 echo "Commands:$commands"
 #ignore certain kernels tests
-if [[ $commands == *" kernels "* ]]; then
+if [[ $commands == *" kernels/core"* ]]; then
   commands="${commands} \
-  --ignore=kernels/test_attention_selector.py \
-  --ignore=kernels/test_blocksparse_attention.py \
-  --ignore=kernels/test_causal_conv1d.py \
-  --ignore=kernels/test_cutlass.py \
-  --ignore=kernels/test_encoder_decoder_attn.py \
-  --ignore=kernels/test_flash_attn.py \
-  --ignore=kernels/test_flashinfer.py \
-  --ignore=kernels/test_int8_quant.py \
-  --ignore=kernels/test_machete_gemm.py \
-  --ignore=kernels/test_mamba_ssm.py \
-  --ignore=kernels/test_marlin_gemm.py \
-  --ignore=kernels/test_moe.py \
-  --ignore=kernels/test_prefix_prefill.py \
-  --ignore=kernels/test_rand.py \
-  --ignore=kernels/test_sampler.py \
-  --ignore=kernels/test_cascade_flash_attn.py \
-  --ignore=kernels/test_mamba_mixer2.py \
-  --ignore=kernels/test_aqlm.py \
-  --ignore=kernels/test_machete_mm.py \
-  --ignore=kernels/test_mha_attn.py \
-  --ignore=kernels/test_block_fp8.py \
-  --ignore=kernels/test_permute_cols.py"
+  --ignore=kernels/core/test_fused_quant_layernorm.py \
+  --ignore=kernels/core/test_permute_cols.py"
+fi
+
+if [[ $commands == *" kernels/attention"* ]]; then
+  commands="${commands} \
+  --ignore=kernels/attention/stest_attention_selector.py \
+  --ignore=kernels/attention/test_blocksparse_attention.py \
+  --ignore=kernels/attention/test_encoder_decoder_attn.py \
+  --ignore=kernels/attention/test_attention_selector.py \
+  --ignore=kernels/attention/test_flash_attn.py \
+  --ignore=kernels/attention/test_flashinfer.py \
+  --ignore=kernels/attention/test_prefix_prefill.py \
+  --ignore=kernels/attention/test_cascade_flash_attn.py \
+  --ignore=kernels/attention/test_mha_attn.py \
+  --ignore=kernels/attention/test_lightning_attn.py \
+  --ignore=kernels/attention/test_attention.py"
+fi
+
+if [[ $commands == *" kernels/quantization"* ]]; then
+  commands="${commands} \
+  --ignore=kernels/quantization/test_int8_quant.py \
+  --ignore=kernels/quantization/test_aqlm.py \
+  --ignore=kernels/quantization/test_machete_mm.py \
+  --ignore=kernels/quantization/test_block_fp8.py \
+  --ignore=kernels/quantization/test_block_int8.py \
+  --ignore=kernels/quantization/test_marlin_gemm.py \
+  --ignore=kernels/quantization/test_cutlass_scaled_mm.py \
+  --ignore=kernels/quantization/test_int8_kernel.py"
+fi
+
+if [[ $commands == *" kernels/mamba"* ]]; then
+  commands="${commands} \
+  --ignore=kernels/mamba/test_mamba_mixer2.py \
+  --ignore=kernels/mamba/test_causal_conv1d.py \
+  --ignore=kernels/mamba/test_mamba_ssm_ssd.py"
+fi
+
+if [[ $commands == *" kernels/moe"* ]]; then
+  commands="${commands} \
+  --ignore=kernels/moe/test_moe.py \
+  --ignore=kernels/moe/test_cutlass_moe.py \
+  --ignore=kernels/moe/test_triton_moe_ptpc_fp8.py"
 fi
 
 #ignore certain Entrypoints/openai tests

.buildkite/scripts/hardware_ci/run-cpu-test-ppc64le.sh

@@ -5,25 +5,30 @@
 set -ex
 
 # Setup cleanup
-remove_docker_container() { podman rm -f cpu-test-ubi9-ppc || true; podman system prune -f; }
+remove_docker_container() {
+  if [[ -n "$container_id" ]]; then
+      podman rm -f "$container_id" || true
+  fi
+  podman system prune -f
+}
 trap remove_docker_container EXIT
 remove_docker_container
 
 # Try building the docker image
 podman build -t cpu-test-ubi9-ppc -f docker/Dockerfile.ppc64le .
 
 # Run the image
-podman run -itd --entrypoint /bin/bash -v /tmp/:/root/.cache/huggingface --privileged=true --network host -e HF_TOKEN --name cpu-test-ubi9-ppc cpu-test-ubi9-ppc
+container_id=$(podman run -itd --entrypoint /bin/bash -v /tmp/:/root/.cache/huggingface --privileged=true --network host -e HF_TOKEN cpu-test-ubi9-ppc)
 
 function cpu_tests() {
 
   # offline inference
-  podman exec cpu-test-ubi9-ppc bash -c "
+  podman exec -it "$container_id" bash -c "
     set -e
     python3 examples/offline_inference/basic/generate.py --model facebook/opt-125m"
 
   # Run basic model test
-  podman exec cpu-test-ubi9-ppc bash -c "
+  podman exec -it "$container_id" bash -c "
     set -e
     pip install pytest pytest-asyncio einops peft Pillow soundfile transformers_stream_generator matplotlib
     pip install sentence-transformers datamodel_code_generator
@@ -33,6 +38,8 @@ function cpu_tests() {
 }
 
 # All of CPU tests are expected to be finished less than 40 mins.
+
+export container_id
 export -f cpu_tests
 timeout 40m bash -c cpu_tests
 

.buildkite/scripts/hardware_ci/run-tpu-v1-test.sh

@@ -17,8 +17,9 @@ source /etc/environment
 docker run --privileged --net host --shm-size=16G -it \
     -e "HF_TOKEN=$HF_TOKEN" --name tpu-test \
     vllm-tpu /bin/bash -c "python3 -m pip install git+https://github.com/thuml/depyf.git \
-    && python3 -m pip install pytest tpu-info \
+    && python3 -m pip install pytest pytest-asyncio tpu-info \
     && python3 -m pip install lm_eval[api]==0.4.4 \
+    && export VLLM_XLA_CACHE_PATH= \
     && export VLLM_USE_V1=1 \
     && export VLLM_XLA_CHECK_RECOMPILATION=1 \
     && echo HARDWARE \
@@ -42,7 +43,11 @@ docker run --privileged --net host --shm-size=16G -it \
     && echo TEST_8 \
     && pytest -s -v /workspace/vllm/tests/v1/tpu/test_topk_topp_sampler.py \
     && echo TEST_9 \
-    && pytest -s -v /workspace/vllm/tests/v1/tpu/test_pallas.py" \
+    && pytest -s -v /workspace/vllm/tests/v1/tpu/test_multimodal.py \
+    && echo TEST_10 \
+    && pytest -s -v /workspace/vllm/tests/v1/tpu/test_pallas.py \
+    && echo TEST_11 \
+    && pytest -s -v /workspace/vllm/tests/v1/entrypoints/llm/test_struct_output_generate.py" \
 
 
 # TODO: This test fails because it uses RANDOM_SEED sampling

.buildkite/scripts/upload-wheels.sh

@@ -50,11 +50,11 @@ aws s3 cp "$normal_wheel" "s3://vllm-wheels/$BUILDKITE_COMMIT/"
 if [[ $normal_wheel == *"cu118"* ]]; then
     # if $normal_wheel matches cu118, do not upload the index.html
     echo "Skipping index files for cu118 wheels"
-elif [[ $normal_wheel == *"cu121"* ]]; then
-    # if $normal_wheel matches cu121, do not upload the index.html
-    echo "Skipping index files for cu121 wheels"
+elif [[ $normal_wheel == *"cu126"* ]]; then
+    # if $normal_wheel matches cu126, do not upload the index.html
+    echo "Skipping index files for cu126 wheels"
 else
-    # only upload index.html for cu124 wheels (default wheels)
+    # only upload index.html for cu128 wheels (default wheels)
     aws s3 cp index.html "s3://vllm-wheels/$BUILDKITE_COMMIT/vllm/index.html"
     aws s3 cp "s3://vllm-wheels/nightly/index.html" "s3://vllm-wheels/$BUILDKITE_COMMIT/index.html"
 fi
@@ -66,12 +66,12 @@ aws s3 cp "$normal_wheel" "s3://vllm-wheels/nightly/"
 if [[ $normal_wheel == *"cu118"* ]]; then
     # if $normal_wheel matches cu118, do not upload the index.html
     echo "Skipping index files for cu118 wheels"
-elif [[ $normal_wheel == *"cu121"* ]]; then
-    # if $normal_wheel matches cu121, do not upload the index.html
-    echo "Skipping index files for cu121 wheels"
+elif [[ $normal_wheel == *"cu126"* ]]; then
+    # if $normal_wheel matches cu126, do not upload the index.html
+    echo "Skipping index files for cu126 wheels"
 else
-    # only upload index.html for cu124 wheels (default wheels)
+    # only upload index.html for cu128 wheels (default wheels)
     aws s3 cp index.html "s3://vllm-wheels/nightly/vllm/index.html"
 fi
 
-aws s3 cp "$wheel" "s3://vllm-wheels/$version/"
+aws s3 cp "$wheel" "s3://vllm-wheels/$version/"