Alexandria Library

A curated collection of awesome articles

Here are some ressources related to Transformer, Language, Memory. Hoping you will find learning material, alternative tools or just some interesting stuff.

Home > Programming, Machine Learning, Quan... > Machine Learning, Deep Learning, Ne... > NLP, Summarization, Machine Learnin... > Transformer, Language, Memory

Articles

A Mechanistic Interpretability Analysis of Grokking
See discussion (53 comments, 202 points) - Archive
RWKV: Reinventing RNNs for the Transformer Era
See discussion (171 comments, 358 points) - Archive
Why the original transformer figure is wrong, and some other tidbits about LLMs
See discussion (49 comments, 237 points) - Archive
Patent for attention-based sequence transduction neural networks (2019)
See discussion (96 comments, 100 points) - Archive
Revealing example of self-attention, the building block of transformer AI models
See discussion (35 comments, 98 points) - Archive
Scaling Transformer to 1M tokens and beyond with RMT
See discussion (132 comments, 277 points) - Archive
Mass editing memory in a transformer
See discussion (51 comments, 142 points) - Archive
Understanding large language models: A cross-section of the relevant literature
See discussion (31 comments, 307 points) - Archive
What are transformer models and how do they work?
See discussion (108 comments, 441 points) - Archive
From deep to long learning?
See discussion (117 comments, 499 points) - Archive
RWKV RNN: Better than ChatGPT?
See discussion (107 comments, 328 points) - Archive
The RWKV language model: An RNN with the advantages of a transformer
See discussion (45 comments, 184 points) - Archive
Vision Transformers (ViT) Explained
See discussion (30 comments, 107 points) - Archive
Transformer models: an introduction and catalog
See discussion (25 comments, 188 points) - Archive
Understanding and coding the self-attention mechanism of large language models
See discussion (37 comments, 158 points) - Archive
Understanding Large Language Models – A Transformative Reading List
See discussion (16 comments, 81 points) - Archive
The paper that made ChatGPT possible
See discussion (55 comments, 173 points) - Archive
The Transformer Family
See discussion (46 comments, 254 points) - Archive
A stack of feed-forward layers does surprisingly well on ImageNet
See discussion (25 comments, 97 points) - Archive
Formal Algorithms for Transformers
See discussion (13 comments, 106 points) - Archive
Mental time travel: Hierarchical memory for reinforcement learning agents (2021)
See discussion (8 comments, 82 points) - Archive
Building a transformer in Julia
See discussion (4 comments, 132 points) - Archive
Memorizing Transformers
See discussion (32 comments, 179 points) - Archive
Transformers for software engineers
See discussion (20 comments, 175 points) - Archive
DeepNet: Scaling Transformers to 1k Layers
See discussion (38 comments, 194 points) - Archive
Transformers from Scratch
See discussion (17 comments, 199 points) - Archive
Hierarchical transformers are more efficient language models
See discussion (5 comments, 91 points) - Archive
Thinking Like Transformers
See discussion (42 comments, 161 points) - Archive
Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
See discussion (2 comments, 85 points) - Archive
The Unreasonable Syntactic Expressivity of RNNs
See discussion (34 comments, 137 points) - Archive
Rethinking Attention with Performers
See discussion (52 comments, 212 points) - Archive
Google ‘BigBird’ Achieves SOTA Performance on Long-Context NLP Tasks
See discussion (27 comments, 96 points) - Archive
Recipes for automatic text summarization using Google BERT and Microsoft UniLM
See discussion (4 comments, 84 points) - Archive
A new model and dataset for long-range memory
See discussion (13 comments, 166 points) - Archive
Reformer, the Efficient Transformer
See discussion (43 comments, 158 points) - Archive
Single Headed Attention RNN
See discussion (37 comments, 212 points) - Archive
A Machine Learning Guide to Hierarchical Temporal Memory
See discussion (6 comments, 105 points) - Archive
On Extractive and Abstractive Summarization with Transformer Language Models
See discussion (12 comments, 106 points) - Archive
Open-sourcing PyTorch-BigGraph for faster embeddings of extremely large graphs
See discussion (7 comments, 86 points) - Archive
RNN-Based Handwriting Recognition in Gboard
See discussion (21 comments, 109 points) - Archive
Text Normalization using Memory Augmented Neural Networks
See discussion (16 comments, 152 points) - Archive
One model to learn them all
See discussion (47 comments, 289 points) - Archive
Attention Is All You Need
See discussion (30 comments, 225 points) - Archive
Automated Crowdturfing Attacks and Defenses in Online Review Systems
See discussion (21 comments, 88 points) - Archive
Transformer: A Novel Neural Network Architecture for Language Understanding
See discussion (19 comments, 280 points) - Archive
Generating music with expressive timing and dynamics
See discussion (22 comments, 126 points) - Archive
Taming Recurrent Neural Networks for Better Summarization
See discussion (8 comments, 135 points) - Archive
SILT: A Memory-Efficient, High-Performance Key-Value Store [pdf]
See discussion (7 comments, 81 points) - Archive
Dynamic Memory Networks for Visual and Textual Question Answering
See discussion (15 comments, 122 points) - Archive
Attention and Memory in Deep Learning and NLP
See discussion (7 comments, 97 points) - Archive
Simplifying Transformer Blocks
See discussion (49 comments, 142 points) - Archive
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
See discussion (32 comments, 107 points) - Archive
Paving the way to efficient architectures: StripedHyena-7B
See discussion (65 comments, 221 points) - Archive
Can a transformer represent a Kalman filter?
See discussion (62 comments, 180 points) - Archive
"Attention", "Transformers", in Neural Network "Large Language Models"
See discussion (76 comments, 300 points) - Archive
Understand how transformers work by demystifying the math behind them
See discussion (134 comments, 453 points) - Archive
Ten Noteworthy AI Research Papers of 2023
See discussion (19 comments, 127 points) - Archive
Turing Complete Transformers: Two Transformers Are More Powerful Than One
See discussion (77 comments, 190 points) - Archive
MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
See discussion (38 comments, 129 points) - Archive
Coding Self-Attention, Multi-Head Attention, Cross-Attention, Causal-Attention
See discussion (11 comments, 142 points) - Archive
Linear transformers are faster after all
See discussion (22 comments, 116 points) - Archive
Efficient LLM inference solution on Intel GPU
See discussion (25 comments, 103 points) - Archive
LoMA: Lossless Compressed Memory Attention
See discussion (8 comments, 99 points) - Archive
Eagle 7B: Soaring past Transformers
See discussion (81 comments, 400 points) - Archive
How do transformers work?
See discussion (39 comments, 134 points) - Archive
The Annotated S4
See discussion (12 comments, 111 points) - Archive
World model on million-length video and language with RingAttention
See discussion (60 comments, 196 points) - Archive
LWM – Open LLM with 1M Tokens Context Window
See discussion (36 comments, 161 points) - Archive
Mamba Explained: The State Space Model Taking On Transformers
See discussion (93 comments, 270 points) - Archive
Mamba: The Easy Way
See discussion (60 comments, 279 points) - Archive
Based: Simple linear attention language models
See discussion (14 comments, 165 points) - Archive
Show HN: Manta – A tool for FPGA Debugging and Rapid Prototyping
See discussion (5 comments, 86 points) - Archive
How Chain-of-Thought Reasoning Helps Neural Networks Compute
See discussion (153 comments, 270 points) - Archive
Implementation of mixture of experts language model in a single file of PyTorch
See discussion (14 comments, 88 points) - Archive
DenseFormer: Enhancing Information Flow in Transformers
See discussion (33 comments, 123 points) - Archive
Mamba Explained
See discussion (44 comments, 201 points) - Archive
How do machines ‘grok’ data?
See discussion (16 comments, 102 points) - Archive
A Visual Guide to Vision Transformers
See discussion (26 comments, 237 points) - Archive
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
See discussion (28 comments, 168 points) - Archive
Visualizing Attention, a Transformer's Heart [video]
See discussion (172 comments, 999 points) - Archive
Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
See discussion (32 comments, 159 points) - Archive
Energy-Efficient Llama 2 Inference on FPGAs via High Level Synthesis
See discussion (29 comments, 94 points) - Archive
HMT: Hierarchical Memory Transformer for Long Context Language Processing
See discussion (6 comments, 87 points) - Archive
26× Faster Inference with Layer-Condensed KV Cache for Large Language Models
See discussion (19 comments, 127 points) - Archive
Grokked Transformers Are Implicit Reasoners
See discussion (61 comments, 239 points) - Archive
New attention mechanisms that outperform standard multi-head attention
See discussion (49 comments, 233 points) - Archive
Transformers Can Do Arithmetic with the Right Embeddings
See discussion (211 comments, 207 points) - Archive
Transformers Represent Belief State Geometry in Their Residual Stream
See discussion (19 comments, 85 points) - Archive
Grokfast: Accelerated Grokking by Amplifying Slow Gradients
See discussion (39 comments, 117 points) - Archive
Mamba-2 – State Space Duality
See discussion (33 comments, 151 points) - Archive
Meta Open-Sources Megalodon LLM for Efficient Long Sequence Modeling
See discussion (10 comments, 129 points) - Archive
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision
See discussion (60 comments, 287 points) - Archive
Transformer Layers as Painters
See discussion (10 comments, 99 points) - Archive
Codestral Mamba
See discussion (138 comments, 485 points) - Archive
FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention
See discussion (24 comments, 210 points) - Archive
Hardware Acceleration of LLMs: A comprehensive survey and comparison
See discussion (67 comments, 255 points) - Archive