Etl Open Source Projects

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. No data silos. No noisy neighbors. No infrastructure ceiling.

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

An orchestration platform for the development, production, and observation of data assets.

Apache Doris is a real-time analytics and hybrid search database for AI agents.

🧙 Build, run, and manage data pipelines for integrating and transforming data.

Fancy stream processing made operationally mundane

Pentaho Data Integration ( ETL ) a.k.a Kettle

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Privacy and Security focused Segment-alternative, in Golang and React

Open Source Data Security Platform for Developers to Monitor and Detect PII, Anonymize Production Data and Sync it across environments.

Build data pipelines, the easy way 🛠️

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

A curated list with resources about node-based UIs

A curated list of awesome ETL frameworks, libraries, and software.

visualized crawler & ETL IDE written with C#/WPF

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

SQLMesh is a data transformation framework that brings the benefits of DevOps to data teams. It enables data scientists, analysts, and engineers to efficiently run and deploy data transformations written in SQL or Python.

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

Embedded Template Library

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.

Quadratic | Data Science Spreadsheet with Python & SQL

💧 Instill VDP (Versatile Data Pipeline) is an open-source tool to seamlessly integrate AI to process unstructured data in the modern data stack

Actively curated list of awesome BI tools. PRs welcome!

A lightweight stream processing library for Go

Implementing best practices for PySpark ETL jobs and applications.

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

The best place to learn data engineering. Built and maintained by the data engineering community.

The Feldera Incremental Computation Engine

DeepIE: Deep Learning for Information Extraction

Data processing & ETL framework for Ruby

Modern C++ high level GBA engine

A Python stream processing engine modeled after Yahoo! Pipes

Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.

AWS Glue code samples

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Sync data between persistence engines, like ETL only not stodgy

A fast and versatile ETL tool that can transfer data between RDBMS and NoSQL seamlessly

Postgres to Elasticsearch/OpenSearch sync

ETL best practices with airflow, with examples

This repository is a getting started guide to Singer.

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.

Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI

Scalable master data management, identity resolution, entity resolution, and deduplication using ML

Logical Replication extension for PostgreSQL 17, 16, 15, 14, 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

Open Source research tool to search, browse, analyze and explore large document collections by Semantic Search Engine and Open Source Text Mining & Text Analytics platform (Integrates ETL for document processing, OCR for images & PDF, named entity recognition for persons, organizations & locations, metadata management by thesaurus & ontologies, search user interface & search apps for fulltext search, faceted search & knowledge graph)

React components to build CSV files on the fly basing on Array/literal object of data

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

Personal Data Engineering Projects

Dataform is a framework for managing SQL based data operations in BigQuery

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

The most advanced data processing framework allowing to build scalable data processing pipelines and move data between various data sources and destinations.

Open Source .NET libraries that make your life easier.

Mass processing data with a complete ETL for .net developers