Hey HN, I'm Orazio. I built microcrad (with a 'c'), a tiny scalar-valued automatic differentiation engine, with a small multi-layer perceptron implementation on top. It's reimplementation of Andrej Karpathy's micrograd in C. For me, this was a learning project to revisit backpropagation from first principles, with the additional difficulties that come with programming in C.
The basic idea is the same as micrograd: each number is a `Value` node in a computation graph, ops connect nodes, and the `backward` function topologically sorts the graph before applying the chain rule in reverse. The C-specific parts are memory management and two simple data structures I needed to implement backprop: sets and vectors.
The source code is about 1,350 lines, MIT licensed, and well documented. Dependencies are just the standard library and libm. In addition, the repo contains two examples to showcase how the engine works: a toy regression and an MNIST task.
What this is not: a framework to build and train neural networks in production. Being scalar-valued makes it slow, and it wasn't built for numerical robustness or large datasets. There's no commercial aim here; it's a learning project.
If you read through it, I'd like to hear thoughts, both on the ML engineering aspect and on anything that reads as un-idiomatic C.
Two things stick out as un-idiomatic for C. First, the casts before malloc are unnecessary. This you do in C++ but not in C. Second, names with beginning underscore are reserved, and the underscore + capital letter is specifically problematic.
The rest looks fairly nice but there are a couple of things I would do differently: I would not have the tests for NULL, use signed integers for indices and dimensions, use a flexible array member to integrate the data into the vector type directly, and omit the capacity field (as long as benchmarking does not show it is really needed). I would also use variably modified types for bounds checking, and with C23 the include guards become largely unnecessary.
Interesting project. Do you think manual memory management help understand computational graph lifecycle better, or does it distract from backprop itself?
btw, I went down the micrograd path with numpy-primitives all the way to building a PyTorch clone that can pre-train and post-train LLMs (https://github.com/workofart/ml-by-hand). My learning focus was on the math/calculus <-> high-level APIs, instead of efficiency. I'm glad to see more people tackling this problem from different angles.
I guess I interpreted this part of their README as implying that the author found RC too fragile
> Reference counting buys correctness and composability, but at a cost.
> Disadvantage #1: you must balance every reference. Each value_create, value_retain, and each operation's implicit retain of its operands has to be matched by a value_release. Forget one and you leak; do one too many and you free memory that is still in use. The training examples in examples/ are verbose precisely because they are scrupulous about this in their error paths; that verbosity is the price of leak-free C.
I’d put it this way: though the idea of ref counting felt very natural at the beginning for my use case, at some point I realized there were probably better techniques to achieve the same goal. I found myself multiple times writing nonsense like:
sum = value_add(v1, v2)
mul = value_mul(sum, v3)
[...]
value_release(sum)
value_release(mul)
so that later I could release the sum. When you only have one intermediate value it’s still acceptable, but at 3-4 it starts getting cumbersome.
After asking for feedback, someone rightfully pointed out that the better and faster approach for an autograd engine is using an arena allocator. My reason for saying “rightfully” is that arenas are ideal when you have many “objects” that have the same lifespan, such as the values involved in the forward/backward pass. RC is better when you have a lot of “objects” with independent lifespans.
I did a similar project, but my approach to the topology definition was declaring perceptron structs with inputs as pointer arrays and output as a regular variable. With this scheme, perceptrons can reference directly to the outputs from other perceptrons — or even their own output (I haven't implemented that yet).
Hey HN, I'm Orazio. I built microcrad (with a 'c'), a tiny scalar-valued automatic differentiation engine, with a small multi-layer perceptron implementation on top. It's reimplementation of Andrej Karpathy's micrograd in C. For me, this was a learning project to revisit backpropagation from first principles, with the additional difficulties that come with programming in C.
The basic idea is the same as micrograd: each number is a `Value` node in a computation graph, ops connect nodes, and the `backward` function topologically sorts the graph before applying the chain rule in reverse. The C-specific parts are memory management and two simple data structures I needed to implement backprop: sets and vectors.
The source code is about 1,350 lines, MIT licensed, and well documented. Dependencies are just the standard library and libm. In addition, the repo contains two examples to showcase how the engine works: a toy regression and an MNIST task.
What this is not: a framework to build and train neural networks in production. Being scalar-valued makes it slow, and it wasn't built for numerical robustness or large datasets. There's no commercial aim here; it's a learning project.
If you read through it, I'd like to hear thoughts, both on the ML engineering aspect and on anything that reads as un-idiomatic C.
Two things stick out as un-idiomatic for C. First, the casts before malloc are unnecessary. This you do in C++ but not in C. Second, names with beginning underscore are reserved, and the underscore + capital letter is specifically problematic.
The rest looks fairly nice but there are a couple of things I would do differently: I would not have the tests for NULL, use signed integers for indices and dimensions, use a flexible array member to integrate the data into the vector type directly, and omit the capacity field (as long as benchmarking does not show it is really needed). I would also use variably modified types for bounds checking, and with C23 the include guards become largely unnecessary.
Names beginning with double underbar (or single underbar + capital letter) are reserved. Single underbar + lowercase is not. C23 §6.4.2.1.
Interesting project. Do you think manual memory management help understand computational graph lifecycle better, or does it distract from backprop itself?
btw, I went down the micrograd path with numpy-primitives all the way to building a PyTorch clone that can pre-train and post-train LLMs (https://github.com/workofart/ml-by-hand). My learning focus was on the math/calculus <-> high-level APIs, instead of efficiency. I'm glad to see more people tackling this problem from different angles.
Is there a reason you didn't go with something like Boehm for a library gc, instead of writing your reference counting implementation?
Mainly did it for learning, as dgellow correctly presumed, but also there’s something intrinsically beautiful in writing code with zero dependencies
Learning, I presume?
Also refcounting is not a very difficult thing to implement
I guess I interpreted this part of their README as implying that the author found RC too fragile
> Reference counting buys correctness and composability, but at a cost.
> Disadvantage #1: you must balance every reference. Each value_create, value_retain, and each operation's implicit retain of its operands has to be matched by a value_release. Forget one and you leak; do one too many and you free memory that is still in use. The training examples in examples/ are verbose precisely because they are scrupulous about this in their error paths; that verbosity is the price of leak-free C.
In general yes, but not if you are disciplined.
I’d put it this way: though the idea of ref counting felt very natural at the beginning for my use case, at some point I realized there were probably better techniques to achieve the same goal. I found myself multiple times writing nonsense like:
sum = value_add(v1, v2) mul = value_mul(sum, v3) [...] value_release(sum) value_release(mul)
so that later I could release the sum. When you only have one intermediate value it’s still acceptable, but at 3-4 it starts getting cumbersome.
After asking for feedback, someone rightfully pointed out that the better and faster approach for an autograd engine is using an arena allocator. My reason for saying “rightfully” is that arenas are ideal when you have many “objects” that have the same lifespan, such as the values involved in the forward/backward pass. RC is better when you have a lot of “objects” with independent lifespans.
I did a similar project, but my approach to the topology definition was declaring perceptron structs with inputs as pointer arrays and output as a regular variable. With this scheme, perceptrons can reference directly to the outputs from other perceptrons — or even their own output (I haven't implemented that yet).