Awesome Open Source

Programming Languages

Search results for artificial intelligence attention mechanism

artificial-intelligence x

attention-mechanism x

99 search results found

Vit Pytorch ⭐ 16,298

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Palm Rlhf Pytorch ⭐ 7,496

Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

Dalle Pytorch ⭐ 5,477

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

X Transformers ⭐ 3,840

A simple but complete full-attention transformer with a set of promising experimental features from various papers

Musiclm Pytorch ⭐ 2,686

Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch

Audiolm Pytorch ⭐ 2,112

Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

Reformer Pytorch ⭐ 1,917

Reformer, the efficient Transformer, in Pytorch

Toolformer Pytorch ⭐ 1,788

Implementation of Toolformer, Language Models That Can Use Tools, by MetaAI

Make A Video Pytorch ⭐ 1,449

Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch

Lambda Networks ⭐ 1,110

Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute

Alphafold2 ⭐ 1,086

To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released

Soundstorm Pytorch ⭐ 1,054

Implementation of SoundStorm, Efficient Parallel Audio Generation from Google Deepmind, in Pytorch

Perceiver Pytorch ⭐ 980

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

Coca Pytorch ⭐ 900

Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch

Retro Pytorch ⭐ 784

Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

Performer Pytorch ⭐ 777

An implementation of Performer, a linear attention-based transformer, in Pytorch

Muse Maskgit Pytorch ⭐ 739

Implementation of Muse: Text-to-Image Generation via Masked Generative Transformers, in Pytorch

Phenaki Pytorch ⭐ 674

Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch

Longnet ⭐ 613

Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

Tab Transformer Pytorch ⭐ 609

Implementation of TabTransformer, attention network for tabular data, in Pytorch

Memorizing Transformers Pytorch ⭐ 556

Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch

Self Attention Cv ⭐ 550

Implementation of various self-attention mechanisms focused on computer vision. Ongoing repository.

Flamingo Pytorch ⭐ 549

Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

Bottleneck Transformer Pytorch ⭐ 523

Implementation of Bottleneck Transformer in Pytorch

Parti Pytorch ⭐ 487

Implementation of Parti, Google's pure attention-based text-to-image neural network, in Pytorch

Timesformer Pytorch ⭐ 483

Implementation of TimeSformer from Facebook AI, a pure attention-based solution for video classification

Nuwa Pytorch ⭐ 466

Implementation of NÜWA, state of the art attention network for text to video synthesis, in Pytorch

Megabyte Pytorch ⭐ 458

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

Openstl ⭐ 450

OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

Lamda Rlhf Pytorch ⭐ 444

Open-source pre-training implementation of Google's LaMDA in PyTorch. Adding RLHF similar to ChatGPT.

Point Transformer Pytorch ⭐ 402

Implementation of the Point Transformer layer, in Pytorch

Meshgpt Pytorch ⭐ 394

Implementation of MeshGPT, SOTA Mesh generation using Attention, in Pytorch

Build, Deploy, and Scale Reliable Swarms of Autonomous Agents for Workflow Automation. Join our Community: https://discord.gg/DbjBMJTSWD

Enformer Pytorch ⭐ 359

Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch

Recurrent Memory Transformer Pytorch ⭐ 335

Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch

Linformer Pytorch ⭐ 323

My take on a practical implementation of Linformer for Pytorch.

Multimodalmamba ⭐ 321

A novel implementation of fusing ViT with Mamba into a fast, agile, and high performance Multi-Modal Model. Powered by Zeta, the simplest AI framework ever.

Robotic Transformer Pytorch ⭐ 306

Implementation of RT1 (Robotic Transformer) in Pytorch

Medical Chatgpt ⭐ 300

Implementation of ChatGPT, but tailored towards primary care medicine, with the reward being able to collect patient histories in a thorough and efficient manner and come up with a reasonable differential diagnosis

Flash Pytorch ⭐ 298

Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"

Slot Attention ⭐ 286

Implementation of Slot Attention from GoogleAI

Linear Attention Transformer ⭐ 278

Transformer based on a variant of attention that is linear complexity in respect to sequence length

Itransformer ⭐ 275

Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group

Local Attention ⭐ 270

An implementation of local windowed attention for language modeling

Memory Efficient Attention Pytorch ⭐ 267

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

Eqtransformer ⭐ 260

EQTransformer, a python package for earthquake signal detection and phase picking using AI.

Q Transformer ⭐ 253

Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind

Metal Flash Attention ⭐ 252

Faster alternative to Metal Performance Shaders

Magvit2 Pytorch ⭐ 244

Implementation of MagViT2 Tokenizer in Pytorch

Democratization of RT-2 "RT-2: New model translates vision and language into action"

Equiformer Pytorch ⭐ 207

Implementation of the Equiformer, SE3/E3 equivariant attention network that reaches new SOTA, and adopted for use by EquiFold for protein folding

Colt5 Attention ⭐ 207

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

Se3 Transformer Pytorch ⭐ 205

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repository is geared towards integration with eventual Alphafold2 replication.

Routing Transformer ⭐ 202

Fully featured implementation of Routing Transformer

Mega Pytorch ⭐ 195

Implementation of Mega, the Single-head Attention with Multi-headed EMA architecture that currently holds SOTA on Long Range Arena

Linformer ⭐ 194

Implementation of Linformer for Pytorch

En Transformer ⭐ 192

Implementation of E(n)-Transformer, which incorporates attention mechanisms into Welling's E(n)-Equivariant Graph Neural Network

Deformable Attention ⭐ 190

Implementation of Deformable Attention in Pytorch from the paper "Vision Transformer with Deformable Attention"

Attention ⭐ 187

This repository will house a visualization that will attempt to convey instant enlightenment of how Attention works to someone not working in artificial intelligence, with 3Blue1Brown as inspiration

Sinkhorn Transformer ⭐ 178

Sinkhorn Transformer - Practical implementation of Sparse Sinkhorn Attention

Flash Cosine Sim Attention ⭐ 173

Implementation of fused cosine similarity attention in the same style as Flash Attention

Simple Hierarchical Transformer ⭐ 172

Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT

Recurrent Interface Network Pytorch ⭐ 170

Implementation of Recurrent Interface Network (RIN), for highly efficient generation of images and video without cascading networks, in Pytorch

Block Recurrent Transformer Pytorch ⭐ 164

Implementation of Block Recurrent Transformer - Pytorch

Bs Roformer ⭐ 144

Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs

Implementation of "PaLM-E: An Embodied Multimodal Language Model"

Axial Attention ⭐ 140

Implementation of Axial attention - attending to multi-dimensional data efficiently

Fast Transformer ⭐ 132

An implementation of Fastformer: Additive Attention Can Be All You Need, a Transformer Variant in TensorFlow

Triton Transformer ⭐ 125

Implementation of a Transformer, but completely in Triton

Flash Attention Jax ⭐ 123

Implementation of Flash Attention in Jax

Nystrom Attention ⭐ 111

Implementation of Nyström Self-attention, from the paper Nyströmformer

Segformer Pytorch ⭐ 111

Implementation of Segformer, Attention + MLP neural network for segmentation, in Pytorch

H Transformer 1d ⭐ 100

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning

Bidirectional Cross Attention ⭐ 95

A simple cross attention that updates both the source and target in one step

Simplednn ⭐ 95

SimpleDNN is a machine learning lightweight open-source library written in Kotlin designed to support relevant neural network architectures in natural language processing tasks

Calm Pytorch ⭐ 93

Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind

Transganformer ⭐ 91

Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GanFormer and TransGan paper

Long Short Transformer ⭐ 91

Implementation of Long-Short Transformer, combining local and global inductive biases for attention over long sequences, in Pytorch

Compressive Transformer Pytorch ⭐ 86

Pytorch implementation of Compressive Transformers, from Deepmind

Zorro Pytorch ⭐ 83

Implementation of Zorro, Masked Multimodal Transformer, in Pytorch

Mirasol Pytorch ⭐ 74

Implementation of 🌻 Mirasol, SOTA Multimodal Autoregressive model out of Google Deepmind, in Pytorch

Mixture Of Attention ⭐ 74

Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts

Uniformer Pytorch ⭐ 74

Implementation of Uniformer, a simple attention and 3d convolutional net that achieved SOTA in a number of video classification tasks, debuted in ICLR 2022

Fast Transformer Pytorch ⭐ 73

Implementation of Fast Transformer in Pytorch

Taylor Series Linear Attention ⭐ 61

Explorations into the recently proposed Taylor Series Linear Attention

Stam Pytorch ⭐ 59

Implementation of STAM (Space Time Attention Model), a pure and simple attention model that reaches SOTA for video classification

Memory Compressed Attention ⭐ 58

Implementation of Memory-Compressed Attention, from the paper "Generating Wikipedia By Summarizing Long Sequences"

AReLU: Attention-based-Rectified-Linear-Unit

Agent Attention Pytorch ⭐ 57

Implementation of Agent Attention in Pytorch

Perceiver ⭐ 50

Implementation of Perceiver, General Perception with Iterative Attention in TensorFlow

Flash Genomics Model ⭐ 49

My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hierarchical methods)

Global Self Attention Network ⭐ 49

A Pytorch implementation of Global Self-Attention Network, a fully-attention backbone for vision tasks

Lie Transformer Pytorch ⭐ 49

Implementation of Lie Transformer, Equivariant Self-Attention, in Pytorch

Rq Transformer ⭐ 49

Implementation of RQ Transformer, proposed in the paper "Autoregressive Image Generation using Residual Quantization"

Diffusion Policy ⭐ 46

Implementation of Diffusion Policy, Toyota Research's supposed breakthrough in leveraging DDPMs for learning policies for real-world Robotics

Adjacent Attention Network ⭐ 45

Graph neural network message passing reframed as a Transformer with local attention

Halonet Pytorch ⭐ 45

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Tf Bind Transformer ⭐ 45

A repository with exploration into using transformers to predict DNA ↔ transcription factor binding

Isab Pytorch ⭐ 44

An implementation of (Induced) Set Attention Block, from the Set Transformers paper

Flash Attention Softmax N ⭐ 41

CUDA and Triton implementations of Flash Attention with SoftmaxN.

Related Searches

Python Artificial Intelligence (9,831)

Machine Learning Artificial Intelligence (5,690)

Deep Learning Artificial Intelligence (3,273)

Jupyter Notebook Artificial Intelligence (2,652)

Artificial Intelligence Neural Network (1,732)

Java Artificial Intelligence (1,340)

Artificial Intelligence Tensorflow (1,225)

Artificial Intelligence Chatgpt (1,141)

Artificial Intelligence Openai (997)

C Sharp Artificial Intelligence (978)

1-99 of 99 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.