Deploy DocSum on Kubernetes: A Step-by-Step Guide with Helm Charts

This guide provides a comprehensive walkthrough on deploying the DocSum service using Helm charts. DocSum leverages LLM microservices, and this article details the installation, configuration, and verification processes. Whether you are using CPU, Gaudi, or AMD ROCm devices, this guide provides the necessary steps. Let's get started with DocSum deployment.

Prerequisites for DocSum Deployment

Before deploying DocSum, ensure you have the following:

• A Kubernetes cluster.
• Helm installed.
• kubectl configured to connect to your cluster.

Installing the DocSum Helm Chart

Follow these steps to install the DocSum Helm chart and get your document summarization service up and running.

1. Clone the GenAIInfra Repository:

   git clone https://github.com/opea-project/GenAIInfra.git
   cd GenAIInfra/helm-charts/
   

   Copy

2. Prepare the Model Directory:

   mkdir /mnt/opea-models && chmod -R 664 /mnt/opea-models
   

   Copy

3. Update Dependencies:

   ./update_dependency.sh
   helm dependency update docsum
   

   Copy

4. Export Environment Variables: Configure your Hugging Face token, model directory, and model name. Replace "insert-your-huggingface-token-here" with your actual token:

   export HFTOKEN="insert-your-huggingface-token-here"
   export MODELDIR="/mnt/opea-models"
   export MODELNAME="Intel/neural-chat-7b-v3-3"
   

   Copy

Deploying DocSum with Different Devices

The following commands detail deploying DocSum on various devices utilizing different YAML configurations.

Deploying DocSum on CPU

Use this command if you plan to utilize the CPU

5. Install DocSum using Helm:

   helm install docsum docsum --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} --set llm-uservice.LLM_MODEL_ID=${MODELNAME} --set vllm.LLM_MODEL_ID=${MODELNAME}
   

   Copy

Deploying DocSum on Gaudi

Use these commands if you plan to utilize Gaudi

# To use Gaudi device with vLLM
# helm install docsum docsum --set global.HUGGINGFACEHUB_API_TOKEN= ${HFTOKEN} --values docsum/gaudi-values.yaml
# To use Gaudi device with TGI
# helm install docsum docsum --set global.HUGGINGFACEHUB_API_TOKEN= ${HFTOKEN} --values docsum/gaudi-tgi-values.yaml

Deploying DocSum on AMD ROCm

Use these commands if you plan to utilize an AMD ROCm device

# To use AMD ROCm device with vLLM
# helm install docsum docsum --set global.HUGGINGFACEHUB_API_TOKEN= ${HFTOKEN} --values docsum/rocm-values.yaml
# To use AMD ROCm device with TGI
# helm install docsum docsum --set global.HUGGINGFACEHUB_API_TOKEN= ${HFTOKEN} --values docsum/rocm-tgi-values.yaml

Verifying the DocSum Installation

Ensuring your DocSum deployment is successful involves checking the pod status and testing the service via curl commands or the UI.

1. Check Pod Status: Verify all pods are running.

   kubectl get pod
   

   Copy

2. Port Forwarding: Expose the DocSum service for local access.

   kubectl port-forward svc/docsum 8888:8888
   

   Copy

Testing with Curl

To verify that the document summarization service is correctly deployed, use the following curl command:

3. Send a Test Request: Open a new terminal and run:

   curl http://localhost:8888/v1/docsum \
     -H 'Content-Type: multipart/form-data' \
     -F "type=text" \
     -F "messages=Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence classification models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5." \
     -F "max_tokens=32"
   

   Copy

Accessing the UI for Verification

The UI provides an interface to interact with the service.

1. Determine the NodePort:

   • For the default Gradio UI:

     export port=$(kubectl get service docsum-docsum-ui --output='jsonpath={.spec.ports[0].nodePort}')
     echo $port
     

     Copy
   • For other UIs (e.g., NGINX):

     # export port=$(kubectl get service docsum-nginx --output='jsonpath={.spec.ports[0].nodePort}')
     # echo $port
     

     Copy

2. Access the UI in a Browser: Open a browser and navigate to http://<k8s-node-ip-address>:${port}.

Now you can interact with the DocSum workload through the UI, testing document summarization with various inputs.

Conclusion

Successfully deploying DocSum with Helm charts involves careful configuration and verification steps. By following this guide, you can deploy and verify DocSum, ensuring it operates correctly within your Kubernetes environment. This enables efficient and scalable document summarization for your applications.