Data Pipelines Open Source Projects

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

An orchestration platform for the development, production, and observation of data assets.

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

🧙 Build, run, and manage data pipelines for integrating and transforming data.

Lean and mean distributed stream processing system written in rust and web assembly.

Build data pipelines, the easy way 🛠️

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

The best place to learn data engineering. Built and maintained by the data engineering community.

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

MLeap: Deploy ML Pipelines to Production

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

Dataform is a framework for managing SQL based data operations in BigQuery

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

This dbt package captures metadata, artifacts, and test results so you can detect anomalies, monitor data quality, and build metadata tables. It powers Elementary OSS and feeds the wider context layer used by Elementary Cloud’s full Data & AI Control Plane.

Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.

One framework to develop, deploy and operate data workflows with Python and SQL.

Drop-in replacement for Apache Spark UI

Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

Work with your web service, database, and streaming schemas in a single format.

A curated list of awesome projects and resources related to Kubeflow (a CNCF incubating project)

A lightweight CLI tool for versioning data alongside source code and building data pipelines.

Relational Workflows: where database schemas define executable data pipelines.

Cloud-native, data onboarding architecture for Google Cloud Datasets

An Open Source PHP Reporting Framework that helps you to write perfect data reports or to construct awesome dashboards in PHP. Working great with all PHP versions from 5.6 to latest 8.0. Fully compatible with all kinds of MVC frameworks like Laravel, CodeIgniter, Symfony.

Developer platform for production ML.

Pipebird is open source infrastructure for securely sharing data with customers.

Smart Automation Tool for building modern Data Lakes and Data Pipelines

Multi-hop declarative data pipelines

Move your data with ease.

Data pipelines from re-usable components