Quantization Open Source Projects

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Faster Whisper transcription with CTranslate2

中文LLaMA&Alpaca大语言模型+本地CPU/GPU训练部署 (Chinese LLaMA & Alpaca LLMs)

[🔥updating ...] AI 自动量化交易机器人(完全本地部署) AI-powered Quantitative Investment Research Platform. 📃 online docs: https://ufund-me.github.io/Qbot ✨ :news: qbot-mini: https://github.com/Charmve/iQuant

Lossy PNG compressor — pngquant command based on libimagequant library

Improved JPEG encoder.

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

Fast inference engine for Transformer models

A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.

Neural Network Distiller by Intel AI Lab: a Python package for neural network compression research. https://intellabs.github.io/distiller

Vector (and Scalar) Quantization, in Pytorch

RepVGG: Making VGG-style ConvNets Great Again

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

Sparsity-aware deep learning inference runtime for CPUs

A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications.

Base pretrained models and datasets in pytorch (MNIST, SVHN, CIFAR10, CIFAR100, STL10, AlexNet, VGG16, VGG19, ResNet, Inception, SqueezeNet)

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

A list of papers, docs, codes about model quantization. This repo is aimed to provide the info for model quantization research, we are continuously improving the project. Welcome to PR the works (papers, repositories) that are missed by the repo.

Run Mixtral-8x7B models in Colab or consumer desktops

micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape

Content pipeline tools for optimizing glTF assets. :globe_with_meridians:

Leptonica is an open source library containing software that is broadly useful for image processing and image analysis applications. The official github repository for Leptonica is: danbloomberg/leptonica. See leptonica.org for more documentation.

A Python package for extending the official PyTorch that can easily obtain performance on Intel platform

PPL Quantization Tool (PPQ) is a powerful offline neural network quantization tool.

OpenMMLab Model Compression Toolbox and Benchmark.

PaddleSlim is an open-source library for deep model compression and architecture search.

A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.

INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model

Brevitas: neural network quantization in PyTorch

Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization

Efficient computing methods developed by Huawei Noah's Ark Lab

Train, Evaluate, Optimize, Deploy Computer Vision Models via OpenVINO™

Neural Network Compression Framework for enhanced OpenVINO™ inference

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

A pytorch Quantization Toolkit

Dataflow compiler for QNN inference on FPGAs

This is a list of interesting papers and projects about TinyML.

A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.

TinyChatEngine: On-Device LLM Inference Library

[NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning; [NeurIPS 2022] MCUNetV3: On-Device Training Under 256KB Memory

Official implementation of Half-Quadratic Quantization (HQQ)

Palette quantization library that powers pngquant and other PNG optimizers

[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.

A list of high-quality (newest) AutoML works and lightweight models including 1.) Neural Architecture Search, 2.) Lightweight Structures, 3.) Model Compression, Quantization and Acceleration, 4.) Hyperparameter Optimization, 5.) Automated Feature Engineering.

List of papers related to neural network quantization in recent AI conferences and journals.

Embedded and mobile deep learning research resources

[ICML 2024] SqueezeLLM: Dense-and-Sparse Quantization

PyTorch Project Specification.

The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment

A pure Java library implementing the GIF89a specification. Suitable for use on Android.

Deep Compression on AlexNet

A Pytorch Knowledge Distillation library for benchmarking and extending works in the domains of Knowledge Distillation, Pruning, and Quantization.

Complete-Life-Cycle-of-a-Data-Science-Project

Code for: "And the bit goes down: Revisiting the quantization of neural networks"

🤗 Optimum Intel: Accelerate inference with Intel optimization tools

⚡ boost inference speed of T5 models by 5x & reduce the model size by 3x.

Offline semantic Text-to-Image and Image-to-Image search on Android powered by quantized state-of-the-art vision-language pretrained CLIP model and ONNX Runtime inference engine