Llmops Open Source Projects

Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data. 🐳Docker-friendly.⚡Always in sync with Sharepoint, Google Drive, S3, Kafka, PostgreSQL, real-time data APIs, and more.

The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]

Build AI Assistants using function calling

🪢 Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets. Integrates with OpenTelemetry, LangChain, OpenAI SDK, LiteLLM, and more. 🍊YC W23

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

☁️ Build multimodal AI applications with cloud-native stack

<⚡️> SuperAGI - A dev-first open source autonomous AI agent framework. Enabling developers to build, manage & run useful autonomous agents quickly and reliably.

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines

A blazing fast AI Gateway with integrated guardrails. Route to 1,600+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

AI Observability & Evaluation

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

The programming language for agents

Open-source observability for your GenAI or LLM application, based on OpenTelemetry

An awesome & curated list of best LLMOps tools for developers

🐢 The testing framework for ML models, from tabular to LLMs

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalable model training and vector search.

cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持国产cpu/gpu/npu 昇腾生态，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式

The open-source workspace for building and running AI agents. Build agents through chat, share them with your team, and run background agents on schedules or app events.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Evaluation and Tracking for LLM Experiments and AI Agents

🦖 𝗟𝗲𝗮𝗿𝗻 about 𝗟𝗟𝗠𝘀, 𝗟𝗟𝗠𝗢𝗽𝘀, and 𝘃𝗲𝗰𝘁𝗼𝗿 𝗗𝗕𝘀 for free by designing, training, and deploying a real-time financial advisor LLM system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 𝘷𝘪𝘥𝘦𝘰 & 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴

🕹️ Open-source, developer-first LLMOps platform designed to streamline prompt design, version management, instant delivery, collaboration, troubleshooting, observability and more.

AGiXT is a dynamic AI Agent Automation Platform that seamlessly orchestrates instruction management and complex task execution across diverse AI providers. Combining adaptive memory, smart features, and a versatile plugin system, AGiXT delivers efficient and comprehensive AI solutions.

The Security Toolkit for LLM Interactions

An open-source visual programming environment for battle-testing prompts to LLMs.

Turn expensive prompts into cheap fine-tuned models

Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

🤖Self-Modifying Framework from the Future 🔮 World's First AMS

Bionic is an on-premise replacement for ChatGPT, offering the advantages of Generative AI while maintaining strict data confidentiality

No-code multi-agent framework to build LLM Agents, workflows and applications with your data

🏕️ Reproducible development environment for humans and agents

The LLM Anti-Framework

LLM Prompt Injection Detector

The agent framework where every run is durable, replayable, and resumable by default.

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

RayLLM - LLMs on Ray (Archived). Read README for more info.

A database of SDKs, frameworks, libraries, and tools for creating, monitoring, debugging and deploying autonomous AI agents

OpenTelemetry Instrumentation for AI Observability

Dynamiq is an orchestration framework for agentic AI and LLM applications

📊 llm.report is an open-source logging and analytics platform for OpenAI: Log your ChatGPT API requests, analyze costs, and improve your prompts.

The web framework for building LLM microservices [deprecated]

⛓️ Serving LangChain LLM apps and agents automagically with FastApi. LLMops

Large Language Models (LLMs) tutorials & sample scripts, ft. langchain, openai, llamaindex, gpt, chromadb & pinecone

🧠 Motorhead is a memory and information retrieval server for LLMs.

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

利用LLM构建应用实践笔记

LLM全栈优质资源汇总

LLMFlows - Simple, Explicit and Transparent LLM Apps

Deprecated historical repo. Superlinked now develops SIE, a self-hosted inference engine for embeddings, reranking, OCR, extraction, and document processing.

Data-Driven Evaluation for LLM-Powered Applications

🕵️‍♂️ Library designed for developers eager to explore the potential of Large Language Models (LLMs) and other generative AI through a clean, effective, and Go-idiomatic approach.

⚡ Vigil ⚡ Detect prompt injections, jailbreaks, and other potentially risky Large Language Model (LLM) inputs

A fast and minimal framework for building agentic systems

Smoothly Manage Multiple LLMs (OpenAI, Anthropic, Azure) and Image Models (Dall-E, SDXL), Speed Up Responses, and Ensure Non-Stop Reliability.

LLMs and Machine Learning done easily

The Python package for phospho, the LLM analytics platform