Big Data Open Source Projects

The Patterns of Scalable, Reliable, and Performant Large-Scale Systems

ClickHouse® is a real-time analytics database management system

Apache Spark - A unified analytics engine for large-scale data processing

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

Apache Flink

📚 A public list of APIs from round the web.

An open source cybersecurity protocol for syncing decentralized graph data.

QuestDB is a high performance, open-source, time-series database

大数据入门指南 :star:

The official home of the Presto distributed SQL query engine for big data

The Data Engineering Cookbook

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

PredictionIO, a machine learning server for developers and ML engineers.

A distributed, fast open-source graph database featuring horizontal scalability and high availability

Open-Source Web UI for Apache Kafka Management

CMAK is a tool for managing Apache Kafka clusters

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

Cloud-native OSS search engine for observability

The most widely used Python to C compiler

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

Apache Arrow DataFusion SQL Query Engine

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

Apache Beam is a unified programming model for Batch and Streaming data processing.

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

Arkime is an open source, large scale, full packet capturing, indexing, and database system.

The Open Source Feature Store for AI/ML

The AI search platform

Seamless multi-primary syncing database with an intuitive HTTP/JSON API, designed for reliability

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

Apache IoTDB

Data-Centric Pipelines and Data Versioning

Apache Hive

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Simple and Distributed Machine Learning

Apache Calcite

Apache Ignite

Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:

⚡️A vue component support big amount data list with high render performance and efficient.

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

CrateDB is a distributed and scalable SQL database for storing and analyzing massive amounts of data in near real-time, even with complex queries. It is PostgreSQL-compatible, and based on Lucene.

🚄 FASTJSON2 is a Java JSON library with excellent performance.

Data Science Roadmap from A to Z

🔨 🍇 💻 🚀 GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba | 一站式图计算系统

Koalas: pandas API on Apache Spark

Apache Paimon(incubating) is a streaming data lake platform that supports high-speed data ingestion, change data tracking and efficient real-time analytics.

The official repository for ROOT: analyzing, storing and visualizing big data, scientifically

LakeSoul is an end-to-end, realtime cloud-native Lakehouse framework for fast data ingestion, concurrent updates, incremental analytics, multimodal data processing and vector search — powering next-generation BI and AI workloads.

A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)

An easy to use, self-service open BI reporting and BI dashboard platform.

Apache Parquet

🔥🔥 AllData可定义数据中台，以数据平台为底座，以数据中台为桥梁，以机器学习平台为工厂，以大模型应用为上游产品，提供全链路数字化解决方案。产品正式演示体验、社群咨询、商务采购：https://docs.qq.com/doc/DVHlkSEtvVXVCdEFo

Mirror of Apache Flume

A crazy fast analytical database, built on bitmaps. Perfect for ML applications. Learn more at: http://docs.featurebase.com/. Start a Docker instance: https://hub.docker.com/r/featurebasedb/featurebase

Apache Parquet Format

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.

Bare bone examples of machine learning in TensorFlow

Apache Ambari simplifies provisioning, managing, and monitoring of Apache Hadoop clusters.

YTsaurus is a scalable and fault-tolerant open-source big data platform.