Tutorial: Deploy with vLLM

In this tutorial, we will deploy a vLLM endpoint in a few easy steps. vLLM has become one of the leading libraries for LLM-serving and inference, supporting many architectures and models that use them.

Prerequisites

Model Weights

vLLM depends on the model weights being fetched from Hugging Face and requires a User Access Token to fetch them.

You can obtain the Access Token in your Hugging Face account by clicking the Profile icon (top right corner) and selecting Access Tokens.

For deploying the vLLM endpoint, the READ permissions are sufficient.

Hugging Face Access Tokens panel

Please store the obtained token safely. You will need it for the next steps!

Some models on Hugging Face require the user to accept their usage policy, so make sure you verify this for the models you are deploying.

Create the Deployment

In this tutorial, we will deploy mistralai/Mistral-7B-v0.1 on a General Compute (24 GB VRAM) GPU type. For larger models, you may need to choose one of the other GPU types we offer.

1. Log in to the DataCrunch cloud dashboard, and go to Containers -> New deployment. Name your deployment and select the Compute Type.

2. We will be using the official vLLM Docker container, set Container Image to docker.io/vllm/vllm-openai

3. Toggle on the Public location for your image

4. Select the Tag to deploy

5. Set the Exposed HTTP port and Healthcheck port to 8000

6. Toggle Start Command on

7. Modify CMD to include the following entries: --model, mistralai/Mistral-7B-v0.1, --gpu-memory-utilization, 0.9, --max-model-len, 8192

8. Add your Hugging Face User Access Token to the Environment Variables as HUGGING_FACE_HUB_TOKEN

9. Deploy container

(You can leave the Scaling options to their default values.)

That's it you should now have a running deployment!

For production use, we recommend authenticating/using private registries to avoid potential rate limits imposed by public container registries.

Connect to the Endpoint

Before you can connect to the endpoint, you will need to generate an authentication token, by going to Keys -> Inference API Keys, and click Create.

The endpoint URL for your deployment is in the Containers API section in the top left of the screen.

Test Request

Below is an example cURL command for running your test request:

curl -X POST <YOUR_CONTAINERS_API_URL>/v1/completions \
--header 'Authorization: Bearer <YOUR_INFERENCE_API_KEY>' \
--header 'Content-Type: application/json' \
--data '{
  "model": "mistralai/Mistral-7B-v0.1",
  "prompt": "Write a short story about a robot learning to paint:",
  "max_tokens": 150
}'

Example Response

You should see a response that looks like this:

{
    "id": "cmpl-b5a4e4b56a8d40a5a51c49f173f2d13b",
    "object": "text_completion",
    "created": 1731672729,
    "model": "mistralai/Mistral-7B-v0.1",
    "choices": [
        {
            "index": 0,
            "text": "\n\n            “James” it is, for name, my flighty husband the Pweb-5XX billionaire and tech entrepreneur, couldn’t settle on a litany to fit a niche, so named me “James,” as a little joke, I guess. I’m his only little joke. Except, maybe, his little joke about coming back to life, constantly much reassured by freshly inked tats or laser cosmetic visages. Accordingly, I became James.\n\n            I was designed “Mouse” at the “James [s]” gaming corporation headquarters to be a machine companion to the preexisting preexisting preexisting 50,00",
            "logprobs": null,
            "finish_reason": "length",
            "stop_reason": null,
            "prompt_logprobs": null
        }
    ],
    "usage": {
        "prompt_tokens": 12,
        "total_tokens": 162,
        "completion_tokens": 150
    }
}