Integrates LiteLLM for Unified Access to Multiple LLM Models #5925
Conversation
WalkthroughThe changes involve replacing the Changes
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (2)
apiserver/requirements/base.txt (1)
40-40: Consider upgrading to latest litellm version.
The current version (1.51.0) is a few versions behind the latest release (1.55.0). While the chosen version should work fine, newer versions include additional features and bug fixes.
-litellm==1.51.0
+litellm==1.55.0apiserver/plane/app/views/external/base.py (1)
59-61: Add rate limiting and token limits.
Consider implementing these security measures:
- Add rate limiting to prevent abuse
- Set maximum token limits to control costs
Example implementation:
from django.core.cache import cache
from django.conf import settings
def get_rate_limit_key(request):
return f"gpt_rate_limit_{request.user.id}"
def check_rate_limit(request):
key = get_rate_limit_key(request)
current = cache.get(key, 0)
if current >= settings.GPT_RATE_LIMIT:
return False
cache.incr(key)
return True
# In your view:
if not check_rate_limit(request):
return Response(
{"error": "Rate limit exceeded"},
status=status.HTTP_429_TOO_MANY_REQUESTS,
)
# Add to completion call:
response = completion(
model=GPT_ENGINE,
messages=[{"role": "user", "content": final_text}],
api_key=OPENAI_API_KEY,
max_tokens=settings.GPT_MAX_TOKENS, # Add this
)Also applies to: 119-121
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
- apiserver/plane/app/views/external/base.py (5 hunks)
- apiserver/requirements/base.txt (1 hunks)
🔇 Additional comments (4)
apiserver/requirements/base.txt (1)
40-40: LGTM! Verify removal of openai package.
The switch to litellm is appropriate for the stated objectives. However, let's verify that no other packages depend on openai.
apiserver/plane/app/views/external/base.py (3)
1-17: LGTM! Import statements are well-organized.
The imports follow the standard pattern (stdlib → third-party → local) and correctly include the required litellm package.
94-94:
Apply the same fixes as GPTIntegrationEndpoint.
The same issues are present in this endpoint:
- Model validation
- Global API key setting
- Missing error handling
Please apply the same fixes as suggested for the GPTIntegrationEndpoint.
Also applies to: 118-121
Line range hint 1-161: Verify the migration from OpenAI to LiteLLM across the codebase.
Let's ensure all OpenAI client usage has been migrated to LiteLLM.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (3)
apiserver/plane/app/views/external/base.py (3)
21-37: Add type hints and improve docstring.The configuration helper function is well-structured, but could benefit from type hints and a more detailed docstring.
-def get_gpt_config(): +def get_gpt_config() -> tuple[str | None, str | None]: """Helper to get GPT configuration values + + Returns: + tuple: A tuple containing (api_key, engine) where both values could be None + if configuration is missing """
39-51: Enhance error handling and type safety.The response helper has good error handling but could be improved with type hints and more specific error handling.
-def get_gpt_response(task, prompt, api_key, engine): +def get_gpt_response(task: str, prompt: str | None, api_key: str, engine: str) -> tuple[str | None, str | None]: """Helper to get GPT completion response + + Args: + task: The main task description + prompt: Additional prompt context (optional) + api_key: The LiteLLM API key + engine: The LLM engine to use + + Returns: + tuple: (response_text, error_message) where one will always be None """ final_text = task + "\n" + prompt try: response = litellm.completion( model=engine, messages=[{"role": "user", "content": final_text}], api_key=api_key, ) text = response.choices[0].message.content.strip() return text, None - except Exception as e: + except litellm.exceptions.OpenAIError as e: return None, str(e)
Line range hint
54-136: Consider architectural improvements.
- The GPT integration endpoints share significant code. Consider extracting common logic into a base class or mixin.
- There's no rate limiting for the LLM API calls, which could lead to excessive costs.
Example base class approach:
class BaseGPTEndpoint(BaseAPIView): def handle_gpt_request(self, request, additional_response_data=None): OPENAI_API_KEY, GPT_ENGINE = get_gpt_config() if not OPENAI_API_KEY or not GPT_ENGINE: return Response( {"error": "OpenAI API key and engine is required"}, status=status.HTTP_400_BAD_REQUEST, ) task = request.data.get("task", False) if not task: return Response( {"error": "Task is required"}, status=status.HTTP_400_BAD_REQUEST, ) text, error = get_gpt_response(task, request.data.get("prompt", False), OPENAI_API_KEY, GPT_ENGINE) if error: return Response( {"error": "An error occurred while processing your request"}, status=status.HTTP_500_INTERNAL_SERVER_ERROR, ) response_data = { "response": text, "response_html": text.replace("\n", "<br/>"), } if additional_response_data: response_data.update(additional_response_data) return Response(response_data, status=status.HTTP_200_OK)Would you like me to provide a complete implementation of the rate limiting solution?
🧰 Tools
🪛 GitHub Check: CodeQL
[warning] 83-83: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.
[warning] 91-96: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
apiserver/plane/app/views/external/base.py(2 hunks)
🧰 Additional context used
🪛 GitHub Check: CodeQL
apiserver/plane/app/views/external/base.py
[warning] 83-83: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.
[warning] 125-125: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.
🔇 Additional comments (1)
apiserver/plane/app/views/external/base.py (1)
1-19: LGTM! Clean import organization.
The imports are well-organized and include all necessary dependencies for the LiteLLM integration.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (2)
apiserver/plane/app/views/external/base.py (2)
21-36: Move model validation to configuration level.Consider validating the GPT_ENGINE value when retrieving configuration to fail fast if an unsupported model is configured.
def get_gpt_config(): """Helper to get GPT configuration values""" + SUPPORTED_MODELS = ["gpt-4o-mini", "gpt-4o"] OPENAI_API_KEY, GPT_ENGINE = get_configuration_value([ { "key": "OPENAI_API_KEY", "default": os.environ.get("OPENAI_API_KEY", None), }, { "key": "GPT_ENGINE", "default": os.environ.get("GPT_ENGINE", "gpt-4o-mini"), }, ]) if not OPENAI_API_KEY or not GPT_ENGINE: return None, None + if GPT_ENGINE not in SUPPORTED_MODELS: + return None, None return OPENAI_API_KEY, GPT_ENGINE
References to gpt-3.5-turbo need to be updated in configuration files
The following files still contain references to the deprecated
gpt-3.5-turbomodel and need to be updated to usegpt-4o-mini:
apiserver/plane/license/management/commands/configure_instance.py: Default value for GPT_ENGINE is set to "gpt-3.5-turbo"admin/app/ai/form.tsx: Uses "gpt-3.5-turbo" in placeholder textENV_SETUP.md: Contains deprecated example showing "gpt-3.5-turbo"🔗 Analysis chain
Line range hint
1-165: Verify removal of all gpt-3.5-turbo references.Let's ensure all references to the deprecated model have been removed from the codebase.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Search for any remaining references to gpt-3.5-turbo echo "Searching for gpt-3.5-turbo references..." rg -l "gpt-3.5-turbo" # Search for any OpenAI direct client usage that might need updating echo "Searching for OpenAI client usage..." rg -l "openai\.ChatCompletion" rg -l "openai\.Completion"Length of output: 360
Script:
#!/bin/bash # Check the content of files that still reference gpt-3.5-turbo echo "=== Content of configure_instance.py ===" cat apiserver/plane/license/management/commands/configure_instance.py echo -e "\n=== Content of form.tsx ===" cat admin/app/ai/form.tsx echo -e "\n=== Content of ENV_SETUP.md ===" cat ENV_SETUP.mdLength of output: 18565
🧰 Tools
🪛 GitHub Check: CodeQL
[warning] 91-96: Information exposure through an exception
Stack trace information flows to this location and may be exposed to an external user.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
apiserver/plane/app/views/external/base.py(2 hunks)
🔇 Additional comments (2)
apiserver/plane/app/views/external/base.py (2)
1-19: LGTM! Clean import organization.
The imports are well-organized and include all necessary dependencies for the LiteLLM integration.
58-65: 🛠️ Refactor suggestion
Avoid duplicating model validation logic.
The supported models list is hardcoded and duplicated. Consider moving it to a shared constant or configuration.
+# At module level
+SUPPORTED_MODELS = ["gpt-4o-mini", "gpt-4o"]
+
class GPTIntegrationEndpoint(BaseAPIView):
@allow_permission([ROLE.ADMIN, ROLE.MEMBER])
def post(self, request, slug, project_id):
OPENAI_API_KEY, GPT_ENGINE = get_gpt_config()
- supported_models = ["gpt-4o-mini", "gpt-4o"]
- if GPT_ENGINE not in supported_models:
+ if GPT_ENGINE not in SUPPORTED_MODELS:
return Response(
- {"error": f"Unsupported model. Please use one of: {', '.join(supported_models)}"},
+ {"error": f"Unsupported model. Please use one of: {', '.join(SUPPORTED_MODELS)}"},
status=status.HTTP_400_BAD_REQUEST,
)Likely invalid or redundant comment.
Model Replacement:
Performance and Cost Improvements:
Summary by CodeRabbit
New Features
litellmlibrary for AI completions, replacing the previousOpenAIclient.Chores
openaipackage from project dependencies and added thelitellmpackage.