Simplify your LLM Chatbot Demos with SAP BTP and Gradio

[AronMac]

NOTE: The views and opinions expressed in this blog are my own

In this blog I will demonstrate how you can quickly build a custom LLM Chatbot Demo with Gradio in under 100 lines of code, running on SAP BTP. Gradio's agility makes it a quick solution to present your machine learning models and share with colleagues.

Gradio is commonly used to demonstrate open source models on Hugging Face.

For an example see Meta's recent opensource Llama2 chatbot.

 

If you get to the end of this blog you will be able to build the following LLM Chatbot demo on BTP, using OpenAI API's:

Click to Enlarge

 

The pre-requistes are:

• BTP Account..  (even a free trial account works)


• Cloud foundry (cf) cli installed


• OpenAI API key


• Basic Python knowledge

 

The build includes the steps to make the following:

openai_service	Cloud Foundry User Provide Service to store your OpenAI API Key
manifest.yaml	Details of the App, need for deployment to Cloud Foundry
requirements.txt	Python library dependencies
server.py	100 lines of code

NOTE: in this example I've focussed on deploying on the BTP CF runtime environment, but with some minor tweaks to the steps the Kyma (K8S) runtime could also be used.

 

 

openai_service

Create the following:

#Command line (Mac flavour)



cf cups openai_service -p '{"username":"api_key","password":"<your OpenAI API Key goes here>"}'

 

In BTP you should see something like the following.

Click to Enlarge

 

manifest.yaml

In your local machine create a file with the following:

---

applications:

- name: llm-chatbot

  random-route: true

  domain: cfapps.ap21.hana.ondemand.com

  path: ./

  memory: 128M

  buildpack: python_buildpack

  command: python server.py

services:

 - openai_service

 

 

requirements.txt

In your local machine create a file with the following:

gradio==3.38.0

cfenv

openai

 

server.py

In your local machine create a file with the following:

import os, cfenv

import gradio as gr 

import openai



port = int(os.environ.get('PORT', 7860))

print("PORT:", port)



# Create a Cloud Foundry environment object

env = cfenv.AppEnv()



# Check if the app is running in Cloud Foundry

if env.app:

    try:

        # Get the specified user-provided service by name

        service_name = "openai_service"

        service = env.get_service(name=service_name)

        

        # Access the service credentials

        credentials = service.credentials

        

        # Access the "openai_api_key" 

        openai_api_key = credentials['password']

        os.environ['OPENAI_API_KEY'] = openai_api_key



        openai.api_key = openai_api_key



        print("OpenAI API Key assigned")

    except cfenv.AppEnvError:

        print(f"The service '{service_name}' is not found.")

else:

    print("The app is not running in Cloud Foundry.")





prompt = " "



#Call the LLM Api's with a prompt/question and return a response

def generate_response(llm, prompt):

   if llm == 'OpenAI':

      completion = openai.Completion.create(

            model = "text-davinci-003",

            prompt = prompt,

            temperature = 0,

            max_tokens= 500, 

            top_p=1,

            frequency_penalty=0, 

            presence_penalty=0, 

            stop=[" Human:", " AI:"]

        ) 

      return completion.choices[0].text

   else:

      return "Unknown LLM, please choose another and retry."



#Handle the Chatbox call back

def llm_chatbot_function(llm, input, history):

    history = history or []

    my_history = list(sum(history, ()))

    my_history.append(input)

    my_input = ' '.join(my_history)

    output = generate_response(llm, my_input)

    history.append((input, output))

    return history, history 



#Define the ChatBot using Gradio Elements

def create_llm_chatbot():

    with gr.Blocks(analytics_enabled=False) as interface:

        with gr.Column():

            top = gr.Markdown("""<h1><center>LLM Chatbot</center></h1>""")

            llm = gr.Dropdown(

                    ["OpenAI", "Llama-2-7b-chat-hf"], label="LLM Choice", info="Which LLM should be used?" , value="OpenAI"

                )

        

            chatbot = gr.Chatbot()

            state = gr.State()

            question = gr.Textbox(show_label=False, placeholder="Ask me a question and press enter.") #.style(container=False)

            with gr.Row():

              summbit_btn = gr.Button("Submit")

              clear = gr.ClearButton([question, chatbot, state ])



        question.submit(llm_chatbot_function, inputs=[llm, question, state], outputs=[chatbot, state])

        summbit_btn.click(llm_chatbot_function, inputs=[llm, question, state], outputs=[chatbot, state])



    return interface



llm_chatbot = create_llm_chatbot()

    

if __name__ == '__main__': 

    llm_chatbot.launch(server_name='0.0.0.0',server_port=port)   

 

 

Deploy the App to SAP BTP

With the above service and files created your are ready to deploy the app.

From the same directory you created the files, run the following:

cf push

 

In BTP you should now see your app running:

Click to Enlarge

 

You can find  the url (application route) to your app when you navigate further on the llm-chatbot:

Click to Enlarge

 

 

Where to Next?

As you can see it only takes a few simple steps to deploy a machine learning demo app onto the BTP platform. Perfect for iterative testing and development.

So where do you you think it make sense to explore this further?

Ideas I have are:

• Add additional Open Source LLM's


• Add Human Feedback to the Chatbot, to be used with fine tuning of open source models


• Add user level security (application routing)


• Integrating with Corporate Data

 

I welcome your additional ideas in the comments below.

 

SAP notes that posts about potential uses of generative AI and large language models are merely the individual poster’s ideas and opinions, and do not represent SAP’s official position or future development roadmap. SAP has no legal obligation or other commitment to pursue any course of business, or develop or release any functionality, mentioned in any post or related content on this website.