[Usage]: How vllm split the llm when pipeline parallel size>1, is it divided equally by the number of hidden layers?

[janelu9]

Your current environment

How vllm split the llm when pipeline parallel size>1, is it divided equally by the number of hidden layers?

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.