Add CustomProvider for custom OpenAI compatible LLM providers#166
Open
dinesy wants to merge 5 commits into
Open
Add CustomProvider for custom OpenAI compatible LLM providers#166dinesy wants to merge 5 commits into
dinesy wants to merge 5 commits into
Conversation
|
|
Collaborator
|
This MR looks good to me with the exception of missing docs! It is currently in draft. Feel free to add docs and set to "ready" and will merge |
Collaborator
|
Hey @dinesy There are allot of changes here that are unrelated to the work you did. Could you rebase this branch onto main? Thanks! |
# Summary For Issue AcademySoftwareFoundation#77 (Add Ollama LLM Provider). While ollama technically has its own REST API, it also supports standard OpenAI compatible calls (the `/v1` base url instead of `/api`). This means we can still use the openai module, and support any other local LLM provider (thus the "custom" name instead of "ollama"). Signed-off-by: Ben Dines <benjamin.dines@gmail.com>
llama3.2:latest is a basic 3B model host.docker.internal points to docker host on Docker Desktop for Mac Signed-off-by: Ben Dines <benjamin.dines@gmail.com>
…oviderBase class It shouldn't be necessary, given that any provider that requires one would fail anyway, and a Custom provider would likely not have one at all. Signed-off-by: Ben Dines <benjamin.dines@gmail.com>
- Remove description of error for missing credentials - Add entry for `custom` to llm provider table in README.md - Add yaml examples for configuring custom llm provider to README.md and QUICKSTART.md - Add explanation of local hostnames and docker containers to README.md - Add link from QUICKSTART.md to local hostname explanation in README.md Signed-off-by: Ben Dines <benjamin.dines@gmail.com>
Add option 3 (Custom/OpenAI-compatible) to the LLM provider setup in bootstrap.sh. When selected, the wizard prompts for: - Custom LLM URL (default: http://host.docker.internal:11434/v1) - Custom LLM model (default: llama3.2:latest) - Whether an API key is required (optional) Writes LLM_PROVIDER=custom, CUSTOM_LLM_URL, CUSTOM_LLM_MODEL, and optionally CUSTOM_LLM_API_KEY to docker-compose.local.yml. On Linux with a host.docker.internal URL, automatically adds the extra_hosts mapping so containers can reach the host. Warns if the user provides a localhost URL since it won't work from within Docker. Signed-off-by: Ben Dines <benjamin.dines@gmail.com>
dinesy
force-pushed
the
issue77-custom-llm-provider
branch
from
27f76f1 to
5b6de10
Compare
Author
|
Fixed! Sorry about that! I must have picked the wrong commit to rebase onto last time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
For Issue #77 (Add Ollama LLM Provider). While ollama technically has its own REST API, it also supports standard OpenAI compatible calls (the
/v1base url instead of/api). This means we can still use the openai module, and support any other local LLM provider (thus the "custom" name instead of "ollama").Testing
How I Tested
Tested via the web UI
Note: I've only confirmed this works on a Mac. Still needs testing on other OS's