Skip to content
Blame Permalink
  • Berkay Orhan's avatar
    9bb24b14
    feat: Implement vLLM handler and WebSocket server for LLM service · 9bb24b14
    Berkay Orhan authored 
    - Added VLLMHandler class for synchronous and asynchronous text generation using vLLM API.
- Integrated tool calling support in VLLMWithToolsHandler for enhanced functionality.
- Developed WebSocketLLMServer for real-time token streaming with support for both vLLM and Ollama backends.
- Configured Docker Compose for vLLM service with GPU optimizations and health checks.
- Updated requirements for vLLM and PydanticAI dependencies.
    9bb24b14
    feat: Implement vLLM handler and WebSocket server for LLM service
    Berkay Orhan authored 
    - Added VLLMHandler class for synchronous and asynchronous text generation using vLLM API.
- Integrated tool calling support in VLLMWithToolsHandler for enhanced functionality.
- Developed WebSocketLLMServer for real-time token streaming with support for both vLLM and Ollama backends.
- Configured Docker Compose for vLLM service with GPU optimizations and health checks.
- Updated requirements for vLLM and PydanticAI dependencies.