Alexandria Library

A curated collection of awesome articles

Here are some ressources related to Machine learning, Model training, Transformer architecture. Hoping you will find learning material, alternative tools or just some interesting stuff.

Home > Programming, Machine Learning, Quan... > Machine Learning, Deep Learning, Ne... > NLP, Summarization, Machine Learnin... > Machine learning, Model training, T...

Articles

GGML – AI at the Edge

See discussion (104 comments, 322 points) - Archive

Meta AI Unleashes Megabyte, a Scalable Model Architecture

See discussion (205 comments, 240 points) - Archive

BigCode Project Releases StarCoder: A 15B Code LLM

See discussion (37 comments, 185 points) - Archive

Basic math related to computation and memory usage for transformers

See discussion (13 comments, 168 points) - Archive

MMC4: An open, billion-scale corpus of images interleaved with text

See discussion (20 comments, 132 points) - Archive

Monolith: The Recommendation System Behind TikTok

See discussion (74 comments, 220 points) - Archive

Stanford benchmarks and compares numerous Large Language Models

See discussion (10 comments, 114 points) - Archive

Using mmap to make LLaMA load faster

See discussion (169 comments, 269 points) - Archive

Finetuning LLMs on a Single GPU Using Gradient Accumulation

See discussion (3 comments, 104 points) - Archive

High-Throughput Generative Inference of Large Language Models with a Single GPU

See discussion (36 comments, 116 points) - Archive

Training Deep Networks with Data Parallelism in Jax

See discussion (37 comments, 122 points) - Archive

How to train large models on many GPUs? (2021)

See discussion (33 comments, 216 points) - Archive

Flame: A small language model for spreadsheet formulas

See discussion (13 comments, 106 points) - Archive

Large Transformer Model Inference Optimization

See discussion (20 comments, 136 points) - Archive

Tinygrad: A simple and powerful neural network framework

See discussion (142 comments, 434 points) - Archive

Emergent Features in 6.7B+ Param Transformers

See discussion (15 comments, 185 points) - Archive

Google ML Compiler Inlining Achieves 3-7% Reduction in Size

See discussion (81 comments, 184 points) - Archive

See discussion (20 comments, 141 points) - Archive

OPT: Open Pre-trained Transformer Language Models

See discussion (224 comments, 461 points) - Archive

DeepMind’s New Language Model, Chinchilla

See discussion (142 comments, 229 points) - Archive

It looks like you’re trying to take over the world

See discussion (274 comments, 334 points) - Archive

A new programming language for high-performance computers

See discussion (102 comments, 173 points) - Archive

T0* – Series of encoder-decoder models trained on a large set of different tasks

See discussion (153 comments, 218 points) - Archive

Ray: A Distributed Framework for Emerging AI Applications

See discussion (34 comments, 134 points) - Archive

Distributed Cloud Builds for Everyone

See discussion (16 comments, 86 points) - Archive

NNCP: Lossless Data Compression with Neural Networks (2019)

See discussion (160 comments, 397 points) - Archive

Flashlight: Fast and flexible machine learning in C++

See discussion (91 comments, 150 points) - Archive

Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

See discussion (75 comments, 275 points) - Archive

Scaling Kubernetes to 7,500 Nodes

See discussion (53 comments, 240 points) - Archive

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

See discussion (60 comments, 223 points) - Archive

Deep Learning for Compilers

See discussion (17 comments, 81 points) - Archive

Learning@home hivemind – train large neural networks across the internet

See discussion (38 comments, 148 points) - Archive

GShard: Scaling giant models with conditional computation and automatic sharding

See discussion (35 comments, 112 points) - Archive

The rise of embarrassingly parallel serverless compute

See discussion (131 comments, 132 points) - Archive

A commit history of BERT and its forks

See discussion (43 comments, 103 points) - Archive

Researchers unveil a pruning algorithm to shrink deep learning models

See discussion (59 comments, 208 points) - Archive

Distilling knowledge from neural networks to build smaller and faster models

See discussion (16 comments, 230 points) - Archive

Sorting 2 Tons of Lego (2017)

See discussion (63 comments, 127 points) - Archive

MLIR Primer: A Compiler Infrastructure for the End of Moore’s Law

See discussion (23 comments, 127 points) - Archive

Open-Sourcing Ax and BoTorch: New AI Tools for Adaptive Experimentation

See discussion (3 comments, 144 points) - Archive

NNCP: Lossless Data Compression with Neural Networks

See discussion (33 comments, 214 points) - Archive

Google open sourced GPipe, a library for training large ML models

See discussion (12 comments, 195 points) - Archive

TensorFlow Code for Google Research's BERT: Pre-Training Method for NLP Tasks

See discussion (13 comments, 129 points) - Archive

Edward – A Turing-Complete Language for Deep Probabilistic Programming

See discussion (30 comments, 206 points) - Archive

NNVM Compiler: A New Open End-To-End Compiler for AI Frameworks

See discussion (14 comments, 124 points) - Archive

Sorting 2 Tons of Lego, Many Questions, Results

See discussion (205 comments, 322 points) - Archive

XLA: linear algebra library for TensorFlow

See discussion (28 comments, 176 points) - Archive

Keras-based molecular autoencoder

See discussion (41 comments, 127 points) - Archive

Gorgonia: a library like Theano or TensorFlow, mainly written in Go

See discussion (53 comments, 180 points) - Archive

Stacked Approximated Regression Machine: A Simple Deep Learning Approach

See discussion (27 comments, 123 points) - Archive

User-friendly language for programming efficient simulations

See discussion (17 comments, 88 points) - Archive

Generating Recommendations at Amazon Scale with Apache Spark and Amazon DSSTNE

See discussion (18 comments, 131 points) - Archive

Show HN: Initial release of the distributed TensorFlow runtime

See discussion (3 comments, 85 points) - Archive

SmileMiner – A Java library of state-of-art machine learning algorithms

See discussion (25 comments, 118 points) - Archive

Dice-O-Matic hopper and elevator

See discussion (4 comments, 81 points) - Archive

A guide to open-source LLM inference and performance

See discussion (14 comments, 113 points) - Archive

Show HN: Stella Nera – Maddness Hardware Accelerator

See discussion (26 comments, 154 points) - Archive

Ultra Fast Bert

See discussion (1 comments, 237 points) - Archive

Fast Llama 2 on CPUs with Sparse Fine-Tuning and DeepSparse

See discussion (26 comments, 238 points) - Archive

Exponentially faster language modelling

See discussion (137 comments, 301 points) - Archive

QuIP#: 2-bit Quantization for LLMs

See discussion (58 comments, 201 points) - Archive

PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU [pdf]

See discussion (9 comments, 82 points) - Archive

Show HN: I made a GPU VRAM calculator for transformer-based models

See discussion (38 comments, 135 points) - Archive

How to make LLMs go fast

See discussion (54 comments, 215 points) - Archive

Distributed Inference and Fine-Tuning of Large Language Models over the Internet

See discussion (22 comments, 102 points) - Archive

TinyLlama: An Open-Source Small Language Model

See discussion (44 comments, 143 points) - Archive

Chess-GPT's Internal World Model

See discussion (2 comments, 121 points) - Archive

Mixtral 8x7B: A sparse Mixture of Experts language model

See discussion (151 comments, 358 points) - Archive

Efficient LLM Inference (2023)

See discussion (11 comments, 94 points) - Archive

Gemma: New Open Models

See discussion (509 comments, 1129 points) - Archive

Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

See discussion (130 comments, 422 points) - Archive

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

See discussion (447 comments, 1040 points) - Archive

GGUF, the Long Way Around

See discussion (30 comments, 249 points) - Archive

The Pile is a 825 GiB diverse, open-source language modelling data set (2020)

See discussion (234 comments, 332 points) - Archive

Fixing Gemma Bugs

See discussion (63 comments, 166 points) - Archive

Show HN: Matrix Multiplication with Half the Multiplications

See discussion (78 comments, 310 points) - Archive

Micrograd-CUDA: adapting Karpathy's tiny autodiff engine for GPU acceleration

See discussion (13 comments, 178 points) - Archive

Towards 1-bit Machine Learning Models

See discussion (157 comments, 351 points) - Archive

Jamba: Production-grade Mamba-based AI model

See discussion (81 comments, 346 points) - Archive

Octopus v2: On-device language model for super agent

See discussion (17 comments, 91 points) - Archive

See discussion (24 comments, 136 points) - Archive

JetMoE: Reaching LLaMA2 performance with 0.1M dollars

See discussion (90 comments, 272 points) - Archive

Mixture-of-Depths: Dynamically allocating compute in transformers

See discussion (83 comments, 281 points) - Archive

Implementation of Google's Griffin Architecture – RNN LLM

See discussion (38 comments, 218 points) - Archive

See discussion (63 comments, 162 points) - Archive

Phi-3 Technical Report

See discussion (130 comments, 411 points) - Archive

Lossless Acceleration of LLM via Adaptive N-Gram Parallel Decoding

See discussion (23 comments, 136 points) - Archive

Fine tune LLAMA3 on million scale dataset in consumer GPU using QLora, DeepSpeed

See discussion (26 comments, 145 points) - Archive

Apple/OpenELM: Efficient Open-Source Family Language Models

See discussion (13 comments, 82 points) - Archive

See discussion (15 comments, 145 points) - Archive

PaliGemma: Open-Source Multimodal Model by Google

See discussion (101 comments, 121 points) - Archive

“Imprecise” language models are smaller, speedier, and nearly as accurate

See discussion (169 comments, 302 points) - Archive

How Meta trains large language models at scale

See discussion (194 comments, 404 points) - Archive

Researchers run high-performing LLM on the energy needed to power a lightbulb

See discussion (70 comments, 147 points) - Archive

Gemma 2: Improving Open Language Models at a Practical Size [pdf]

See discussion (172 comments, 328 points) - Archive

MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

See discussion (55 comments, 260 points) - Archive

SpreadsheetLLM: Encoding Spreadsheets for Large Language Models

See discussion (69 comments, 190 points) - Archive

A Visual Guide to LLM Quantization

See discussion (18 comments, 310 points) - Archive

Calculating the cost of a Google DeepMind paper

See discussion (150 comments, 303 points) - Archive

Self-Compressing Neural Networks

See discussion (57 comments, 233 points) - Archive

Hermes 3: The First Fine-Tuned Llama 3.1 405B Model

See discussion (67 comments, 146 points) - Archive

GPU utilization can be a misleading metric

See discussion (36 comments, 144 points) - Archive