For data security reasons, you may want to enable AI Assistant with a self-deployed LLM. Here we chose the powerful open source model Llama3. We used Ollama and One API as a relay to convert between Bytebase’s OpenAI API-compliant requests and Llama3 API requests.
Prerequisites
Before you begin, make sure you have:
Get Llama3 running in Docker
Run the following command in terminal to get a Docker container running:
docker run -d -p 11434:11434 --name ollama --restart always ollama/ollama
docker exec -it ollama bash
gpt-3.5-turbo (or mapped in One-API). After renaming, the model name is gpt-3.5-turbo, but indeed it is still Llama3.
ollama pull Llama3
ollama cp Llama3 gpt-3.5-turbo
ollama run gpt-3.5-turbo
curl http://localhost:11434/api/generate -d '{
"model": "gpt-3.5-turbo",
"prompt":"Why is the sky blue?"
}'
YOUR_PATH in the following command) to save data and logs. For example, you can use the pwd command in the mac terminal to view the current path and replace YOUR_PATH with it.
docker run --name one-api -d --restart always -p 3000:3000 -e TZ=Asia/Shanghai -v YOUR_PATH/one-api:/data justsong/one-api-en
In Docker dashboard, you can see one-api container and its address as well. You can access localhost:3000 here to log in to One API dashboard.
The initial account username is root, password is 123456.
- Type:
ollama
- Name:
Llama3
- Group:
default
- Model:
gpt-3.5-turbo
- Key: Anything (for example
SSSS|sssss|1111) with format APPID|APISecret|APIKey if ollama has not set up for key
- Proxy: the IP address of the ollama container
http://host.docker.internal:11434
Furthermore, we mentioned above that the model name can be mapped in One-API. This can be done in the Model redirection bar on this page using a JSON string.
In the API keys page, click Add New Token, and fill in the Name (for example Llama3) and Model scope (for example gpt-3.5-turbo).
After clicking Submit, you will see the new API key in My keys list within API keys page. Click Copy to get a token starting with sk-, with witch you can repalce YOUR_TOKEN in the code below. If the code runs successfully in your terminal, it means that One API configuration is complete.
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_TOKEN" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "user",
"content": "Tell me a joke."
}
],
"temperature": 0.7
}'
YOUR_TOKEN we generated in One API into OpenAI API Key bar, and fill the OpenAI API Endpoint bar with http://localhost:3000. Click Update.
Enter SQL Editor from top of any page. You can see an OpenAI icon on top right corner. Click it to start conversation with AI assistant, ask questions in natural language and get SQL results.
