Bigdata Open Source Projects

This is a repo with links to everything you'd ever want to learn about data engineering

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

大数据入门指南 :star:

A curated list of awesome big data frameworks, ressources and other awesomeness.

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. Cloud: https://databend.com

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

Upserts, Deletes And Incremental Processing on Big Data.

A Cloud Native Batch System (Project under CNCF)

100+套大数据可视化炫酷大屏Html5模板；包含行业：社区、物业、政务、交通、金融银行等，全网最新、最多，最全、最酷、最炫大数据可视化模板。陆续更新中

A data integration framework

🔨 用 JSON 来生成结构化的 SQL 语句，基于 Vue3 + TypeScript + Vite + Ant Design + MonacoEditor 实现，项目简单（重逻辑轻页面）、适合练手~

Apache Avro is a data serialization system.

大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料

Python clone of Spark, a MapReduce alike framework in Python

GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.

Please visit https://github.com/h2oai/h2o-3 for latest H2O

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

基于开源的flink，对其实时sql进行扩展；主要实现了流与维表的join，支持原生flink SQL所有的语法

An easy-to-use BI server built for SQL lovers. Power data analysis in SQL and gain faster business insights.

Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF)

Byzer (former MLSQL): A low-code open-source programming language for data pipeline, analytics and AI.

大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

Distributed Big Data Orchestration Service

Google, Naver multiprocess image web crawler (Selenium)

:dart: :star2:[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

TensorBase is a new big data warehousing with modern efforts.

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

Amoro is a Lakehouse management system built on open data lake formats.

A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPC

A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPC

scala、spark使用过程中，各种测试用例以及相关资料整理

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.

学习记录的一些笔记，以及所看得一些电子书eBooks、视频资源和平常收纳的一些自己认为比较好的博客、网站、工具。涉及大数据几大组件、Python机器学习和数据分析、Linux、操作系统、算法、网络等

Data syncing in golang for ClickHouse.

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

C# and F# language binding and extensions to Apache Spark

GUI-based Python code generator for data science, extension to Jupyter Lab, Jupyter Notebook and Google Colab.

TiSpark is built for running Apache Spark on top of TiDB/TiKV

An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset

懒松鼠Flink-Boot 脚手架让Flink全面拥抱Spring生态体系，使得开发者可以以Java WEB开发模式开发出分布式运行的流处理程序，懒松鼠让跨界变得更加简单。懒松鼠旨在让开发者以更底上手成本（不需要理解分布式计算的理论知识和Flink框架的细节）便可以快速编写业务代码实现。为了进一步提升开发者使用懒松鼠脚手架开发大型项目的敏捷的度，该脚手架默认集成Spring框架进行Bean管理，同时将微服务以及WEB开发领域中经常用到的框架集成进来，进一步提升开发速度。比如集成Mybatis ORM框架，Hibernate Validator校验框架,Spring Retry重试框架等，具体见下面的脚手架特性。

A book about running Elasticsearch

Lightweight real-time big data streaming engine over Akka

可视化大屏解决方案, 提供一套可视化编辑引擎, 助力个人或企业轻松定制自己的可视化大屏应用.

DataWave is an ingest/query framework that leverages Apache Accumulo to provide fast, secure data access.

WeDataSphere is a financial grade, one-stop big data platform suite.

Fast topic modeling platform

Data Lineage Tracking And Visualization Solution

Datafaker is a large-scale test data and flow test data generation tool. Datafaker fakes data and inserts to varied data sources. 测试数据生成工具

Multi-Modal Database replacing MongoDB, Neo4J, and Elastic with 1 faster ACID solution, with NetworkX and Pandas interfaces, and bindings for C 99, C++ 17, Python 3, Java, GoLang 🗄️

CORTX Community Object Storage is 100% open source object storage uniquely optimized for mass capacity storage devices.

基于Vue.js的百万级数据表格组件，支持编辑、筛选、过滤、粘贴、拖动调整列宽等多种功能

High Performance HTTP Sidecar Load Balancer

BigData Ecosystem Dataset

A simple spark standalone cluster for your testing environment purposses