Deploy a RAG Workload on a TKG Cluster

As a DevOps engineer, on a TKG cluster in a Supervisor, you can deploy a RAG workload based on the RAG Sample Pipeline from NVIDIA that uses a pgvector PostgreSQL database managed by VMware Data Services Manager.

Prerequisites

• Verify that VMware Private AI Foundation with NVIDIA is available for the VI workload domain. See Preparing VMware Cloud Foundation for Private AI Workload Deployment.
• Deploy a Vector Database in VMware Private AI Foundation with NVIDIA.

Procedure

1. Provision a GPU-accelated TKG cluster.
   You can use one of the following workflows.

   Provisioning Workflow	Steps
   By using a catalog item in VMware Aria Automation	Deploy a GPU-accelerated Tanzu Kubernetes Grid RAG cluster.
   By using the kubectl comamnd	Provision a GPU-Accelerated TKG Cluster by using the kubectl command. For a connected environment, see Provision a GPU-Accelerated TKG Cluster by Using the kubectl Command in a Connected VMware Private AI Foundation with NVIDIA Environment . For a disconnected environment, see Provision a GPU-Accelerated TKG Cluster by Using the kubectl Command in a Disconnected VMware Private AI Foundation with NVIDIA Environment. Install the RAG LLM Operator. See Install the RAG LLM Operator.

2. If you used the kubectl command to provision the TKG cluster, install the NVIDIA RAG LLM Operator on the TKG cluster.

   See Install the RAG LLM Operator.

   During deployment, the AI Kubernetes RAG Cluster catalog item in VMware Aria Automation automatically installs the NVIDIA RAG LLM Operator on the TKG cluster.

3. Download the manifests for the NVIDIA sample RAG pipeline.
   See Sample RAG Pipeline.
4. Configure the sample RAG pipeline with the pgvector PostgreSQL database.
   1. Edit the sample pipeline YAML file.
      See Step 4 in Sample RAG Pipeline.
   2. In the YAML file, configure the sample pipeline with the pgvector PostgreSQL database by using the database's connection string.
      See Vector Database for RAG Sample Pipeline .
5. To provide an external IP for the sample chat application, in the YAML file, set frontend.service.type to loadBalancer.
6. Start the sample RAG pipeline.
   See Sample RAG Pipeline.
7. To access the sample chat application, run the following command to get the application's external IP address.

   kubectl -n rag-sample get service rag-playground

8. In a Web browser, open the sample chat application at http://application_external_ip:3001/orgs/nvidia/models/text-qa-chatbot.