Attention Mechanism Open Source Projects

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

此项目是机器学习(Machine Learning)、深度学习(Deep Learning)、NLP面试中常考到的知识点和代码实现，也是作为一个算法工程师必会的理论基础知识。

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

all kinds of text classification models and more with deep learning

The Enterprise-Grade Production-Ready Multi-Agent Orchestration Framework. Website: https://swarms.ai

A concise but complete full-attention transformer with a set of promising experimental features from various papers

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

An ultimately comprehensive paper list of Vision Transformer/Attention, including papers, codes, and related websites

A collection of important graph embedding, classification and representation learning papers with implementations.

A TensorFlow Implementation of the Transformer: Attention Is All You Need

Graph Attention Networks (https://arxiv.org/abs/1710.10903)

Automatic Speech Recognition (ASR), Speaker Verification, Speech Synthesis, Text-to-Speech (TTS), Language Modelling, Singing Voice Synthesis (SVS), Voice Conversion (VC)

Pytorch implementation of the Graph Attention Network model by Veličković et. al (2017, https://arxiv.org/abs/1710.10903)

Show, Attend, and Tell | a PyTorch Tutorial to Image Captioning

Multilingual Automatic Speech Recognition with word-level timestamps and confidence

Keras Attention Layer (Luong and Bahdanau scores).

My implementation of the original GAT paper (Veličković et al.). I've additionally included the playground.py file for visualizing the Cora dataset, GAT embeddings, an attention mechanism, and entropy histograms. I've supported both Cora (transductive) and PPI (inductive) examples!

Reformer, the efficient Transformer, in Pytorch

Hopfield Networks is All You Need

To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released

Implementation of SoundStorm, Efficient Parallel Audio Generation from Google Deepmind, in Pytorch

Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute

基于金融-司法领域(兼有闲聊性质)的聊天机器人，其中的主要模块有信息抽取、NLU、NLG、知识图谱等，并且利用Django整合了前端展示,目前已经封装了nlp和kg的restful接口

Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

Sequence-to-sequence framework with a focus on Neural Machine Translation based on PyTorch

Implementation of various self-attention mechanisms focused on computer vision. Ongoing repository.

Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch

An implementation of Performer, a linear attention-based transformer, in Pytorch

A Deep Learning library for EEG Tasks (Signals) Classification, based on TensorFlow.

A curated list of NLP resources focused on Transformer networks, attention mechanism, GPT, BERT, ChatGPT, LLMs, and transfer learning.

OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing otherwise seemingly hard concepts. Currently included IWSLT pretrained models.

Implementation of TabTransformer, attention network for tabular data, in Pytorch

Text classifier for Hierarchical Attention Networks for Document Classification

[CVPR2022] Geometric Transformer for Fast and Robust Point Cloud Registration

pytorch implementation of "Get To The Point: Summarization with Pointer-Generator Networks"

TensorFlow Implementation of "Show, Attend and Tell"

Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

Transformer based on a variant of attention that is linear complexity in respect to sequence length

Implementation of the specific Transformer architecture from PaLM - Scaling Language Modeling with Pathways

A PyTorch implementation of "SimGNN: A Neural Network Approach to Fast Graph Similarity Computation" (WSDM 2019).

Visualizing RNNs using the attention mechanism

Tensorflow implementation of attention mechanism for text classification tasks.

Sequence to Sequence Models with PyTorch

Implementation of TimeSformer from Facebook AI, a pure attention-based solution for video classification

[ECCV 2020] Learning Enriched Features for Real Image Restoration and Enhancement. SOTA results for image denoising, super-resolution, and image enhancement.

A Pytorch Implementation of "Neural Speech Synthesis with Transformer Network"

Implementation of Bottleneck Transformer in Pytorch

based on yolo-high-level project (detect\pose\classify\segment\):include yolov5\yolov7\yolov8\ core ,improvement research ,SwintransformV2 and Attention Series. training skills, business customization, engineering deployment C

Attention mechanism for processing sequential data that considers the context for each timestamp.

MORAN: A Multi-Object Rectified Attention Network for Scene Text Recognition

Tensorflow seq2seq Implementation of Text Summarization.

Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch

FlashAttention (Metal Port)

[IGARSS'22]: A Transformer-Based Siamese Network for Change Detection

[CVPR 2021, Oral] PREDATOR: Registration of 3D Point Clouds with Low Overlap.

Implementation of the Point Transformer layer, in Pytorch

End-to-end ASR/LM implementation with PyTorch

Democratization of RT-2 "RT-2: New model translates vision and language into action"