Pipeline Open Source Projects

A new type of shell

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Declarative Continuous Deployment for Kubernetes

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

A high-performance observability data pipeline.

Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io

A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.

☁️ Build multimodal AI applications with cloud-native stack

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

Turns Data and AI algorithms into production-ready web applications in no time.

Easy and Repeatable Kubernetes Development

An orchestration platform for the development, production, and observation of data assets.

Always know what to expect from your data.

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.

Install Jenkins, configure Docker as slave, set up cicd, deploy applications to k8s using Argo CD in GitOps way.

A cloud-native Pipeline resource.

Ultimate Plumber is a tool for writing Linux pipes with instant live preview

🧙 Build, run, and manage data pipelines for integrating and transforming data.

Image loading system

Apache Beam is a unified programming model for Batch and Streaming data processing.

A proposal for adding a useful pipe operator to JavaScript.

Concourse is a container-based automation system written in Go. It's mostly used for CI/CD.

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

Making it easy to write shell-like scripts in Go

Integrate React.js with Rails views and controllers, the asset pipeline, or webpacker.

🍴 Web applications made easy. Since 2011.

A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin

📚 Parameterize, execute, and analyze notebooks

最全面的游戏开发技术图谱(Game Development Map)。帮助游戏开发者们在已知问题上节省时间，省出更多的精力投入到更有创造性的工作中去。

Build powerful pipelines in any programming language.

cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持国产cpu/gpu/npu 昇腾生态，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式

Enterprise-Grade Continuous Delivery & DevOps Automation Open Source Platform

An in-depth machine learning tutorial introducing readers to a whole machine learning pipeline from scratch.

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton

A guideline for building practical production-level deep learning systems to be deployed in real world applications.

prototype of stream based programming language

A collection of examples, tips and tricks and snippets of scripting for the Jenkins Pipeline plugin

Open Source Data Science Resources.

Machine Learning Pipelines for Kubeflow

Digging Deeper....

Tired of pushing to test your .gitlab-ci.yml?

A Ruby client library for Redis

中文nlp解决方案(大模型、数据、模型、训练、推理)

It's a small library to provide the I18n translations on the Javascript. It comes with Rails support.

Open source Structure-from-Motion pipeline

A procedural Blender pipeline for photorealistic training image generation

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

3D Computer Vision Framework

Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.

A DSL for data-driven computational pipelines

pySLAM is a hybrid Python/C++ Visual SLAM pipeline supporting monocular, stereo, and RGB-D cameras. It provides a broad set of modern local and global feature extractors, multiple loop-closure strategies, a volumetric reconstruction module, integrated depth-prediction models, and semantic segmentation capabilities for enhanced scene understanding.

Browser compilation library – an asset pipeline for applications that run in the browser

Wangle is a framework providing a set of common client/server abstractions for building services in a consistent, modular, and composable way.

Common solutions and tools developed by Google Cloud's Professional Services team. This repository and its contents are not an officially supported Google product.

Dynamic sub-dir volume provisioner on a remote NFS server.

Compose web applications with functions

Unity Graphics - Including Scriptable Render Pipeline

SaaS Boilerplate - Open Source and free SaaS stack that lets you build SaaS products faster in React, Django and AWS. Focus on essential business logic instead of coding repeatable features!