Avro Open Source Projects

Upserts, Deletes And Incremental Processing on Big Data.

Apache Avro is a data serialization system.

Apache Parquet

Confluent Schema Registry for Kafka

Record Query - A tool for doing record analysis and transformation

Apache Kafka, Apache Flink and Confluent Platform examples and demos

More than 2000+ Data engineer interview questions.

What's in your data? Extract schema, statistics and entities from datasets

Change data capture from PostgreSQL into Kafka

Avro for JavaScript :zap:

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks

pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].

Goavro is a library that encodes and decodes Avro data.

Command Line Tool for managing Apache Kafka

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Benchmark comparing various data serialization libraries (thrift, protobuf etc.) for C++

Avro schema generation and serialization / deserialization for Scala

[PROJECT IS NO LONGER MAINTAINED] Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

Fast Avro for Python

Reversible conversions between types

Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.

An OpenSource, Unicode compliant Bengali Input Method

Learning Apache spark,including code and data .Most part can run local.

Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

A collection of tools, datasets and resources on Bangla computing

Map-Reduce for Clojure

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.

Avro Data Source for Apache Spark

A fast Go Avro codec

Iceberg is a table format for large, slow-moving tabular data

Generic Data Ingestion & Dispersal Library for Hadoop

A Clojure library for Apache Spark: fast, fully-features, and developer friendly

Sparser: Raw Filtering for Faster Analytics over Raw Data

MySQL binary log consumer with the ability to act on changed rows and publish changes to different systems with emphasis on Apache Kafka.

Web tool for Avro Schema Registry |

Apache Avro RPC Quick Start.

Use Kafka as a remote storage database for Prometheus (remote write only)

MongoDB Kafka Connector

Flexible, Fast & Compact Serialization with RPC

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

A tool for data sampling, data generation, and data diffing

🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile

Replicate data from MySQL, Postgres and MongoDB to ClickHouse®

ADMM based large scale logistic regression

Mu (μ) is a purely functional framework for building micro services.

A Gradle plugin to allow easily performing Java code generation for Apache Avro. It supports JSON schema declaration files, JSON protocol declaration files, and Avro IDL files.

A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.

JSON to Avro conversion tool designed to make migration to Avro easier.

Avro phonetic bangla typing layout for ibus

Divolte Collector

大数据面试题，从0到1走向架构师之路。Flink、Spark、Hive、HBase、Hadoop、Kettle、Kafka...

API and command line interface for HDFS

Confluent's Apache Kafka .NET client

Golang Client for Schema Registry

Json Wikipedia, contains code to convert the Wikipedia xml dump into a json/avro dump

Unicode compliant native Bangla/Bengali Input Method Editor (IME) for Windows

Generate avro schemas from python dataclasses, Pydantic models and Faust Records. Code generation from avro schemas. Serialize/Deserialize python instances with avro schemas.