tinygradgpuemulation№01
running an adreno 630 in numpy
on emulating a phone gpu i never had, from the kernel driver down to how its alu rounds a float.
READ⟶
i'm pawan. i build database and inference stuff and break it a lot. this is where i write down why.
- good at
- ml infra, databases, inference
- languages
- rust, go, typescript, c
HELLO9 POSTS
i build database and inference stuff and break it a lot.
this is where i write down why. every number in a post was printed by a program that ran.
- good at
- ml infra, databases, inference
- languages
- rust, go, typescript, c
- status
- alive, compiling
- written
- 9 posts, 34.1k words
llm inferencegpucuda№02
why speculative decoding changed my output
greedy speculative decoding is supposed to reproduce the big model exactly. on my laptop it changed the text in 16 of 24 runs. the cause was three places where a gpu kernel adds the same numbers in a different order when tokens share a pass.
READ⟶
csimdarm№03
how i made a 27b ternary model 3.8x faster on a laptop cpu by reordering its bits
a 27b ternary model on a snapdragon x cpu, no gpu. prompts get read 3.8x faster and answers get written 1.5x faster, with the same output. how the kernel works, with the code.
READ⟶
ccuda№04
pointers before cuda: what they are, what changes between machines, and where cuda uses them
what an address is, what changes between machines, and where cuda needs both.
READ⟶
gpuperformanceml compilers№05
the inference tax
inference larping maybe?
READ⟶
machine learningoptimizationpython№06
gradient descent is one line of code. here is everything that line hides.
the line almost every neural network is trained with, built from nothing and measured: when it is fast, slow, or explodes, and what sgd, momentum, adam and backprop each change.
READ⟶
ai agentsmemorymachine learning№07
your ai has amnesia, and a bigger context window will not fix it
three turns after a user removed their card, i watched the agent tell them it was still on file. same session.
READ⟶
c++simdmachine learning№08
ciot: a cpu inference engine for ternary networks
0.54 ms per 1024x1024 ternary matvec on a snapdragon x laptop. zero dependencies, one binary, ~3,000 lines of c++ and python.
READ⟶
distributed systemsgoarchitecture№09
Building a Membership System That Says "I Don't Know"
Most failure detectors lie. STYX is a distributed membership system that returns a probability distribution instead of a boolean, handling uncertainty with honesty.
READ⟶