This article is only about understanding the implementation of positional embeddings, but to really understand positional embeddings you have to understand the theory. Positional embeddings are group representations. This is how you find them, and this is also how you parameterize them to make them differentiable/learnable. To give a few examples:
1. Rotary positional embeddings are representations of the translation group. The n-dimensional translation group ℝⁿ is abelian, so its irreducible representations are all one-dimensional ρ: ℝⁿ → ℂ, specified by a momentum vector m: ρₘ(x) = exp(i⟨m, x⟩). You can get a multi-dimensional representation by choosing several m, which are your learnable parameters.
2. If you wanted RoPE embeddings to also preserve rotation (perhaps you have scenes viewed at arbitrary angles), you are looking for representations of the special Euclidean group. The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc., and in general are parameterized by a momentum vector and a discrete spin vector. You would have to do some kind of annealing process to train the spin vector (or you could just make it constant).
3. If you want to do attention on quadtrees, well, what is the group? Suppose every node in the quadtree has local coordinates scaled to [0, 1]. Moving up/down/to a neighbor in the quadtree requires an affine transformation of the local coordinates, which compose, so we can label every node by its affine transformation to the root (x, d): y → x + 2⁻ᵈy, where x is the node's translation in the image and d is its depth in the tree. The finite-dimensional irreducible representations are parameterized by momentum vectors m as well as a modulus M and basis vectors {e₁, e₂, ..., eₖ} where k = ord₂(M) and look like ρₘ,ₑ(x, d) = DPᵈ where D is the diagonal matrix with entries Dⱼⱼ = exp(2πi⟨2ʲm, x⟩ / M) and P permutes the basis vectors.
As I understand, this is post-factum rationalization of what positional embedding is, not the actual theory used to build neural networks.
From what I usually see, ML researches just throw everything at the wall and see what sticks. Whatever approach turns out to work better, gets used more often, and sometimes theoretical background is developed for why it works. RoPE works not because it’s good math, just it turned out to work better than some other equally arbitrary approaches at some point, and then other approaches have also been developed that work better than RoPE.
This is the most frustrating part of trying to understand machine learning. You can read a bunch of really great math books that rigorously build concepts from the ground up (calculus, linear algebra), then you start to get into the more hand wavey math (_some_ statistical concepts feel like this) and then you start getting into some of the theory behind machine learning systems and expecting the same amount of rigor and ability to understand _why_ something works and it’s always an unsatisfying explanation. I really wish there would be more disclosure like “this is what we tried and this seems to work, we think this is why” rather than present the method, give some mathematical formalization of the method, then go “it works great” and then move on. It leaves people like me wondering if there is something that I’m missing or some hole in my knowledge that is obvious to everyone else, OR does no one actually know why this works and just don’t want to say. Just give us a hint, make the shrug emoji or something, the more I learn about machine learning, the more I get that it really is more art than science.
Most people who write ML papers are at the stage of alchemy—try a bunch of things and see if you can create gold out of lead. Usually their papers are not worth reading, because even if they do something right, they do not know what made it right, and it cannot lead to further or generalizable insights.
A few ML researchers do actually have a lot of this theoretical knowledge. You can see this in papers like Variational Diffusion Models, Continuous Normalizing Flows, <?> Delta Attention. These papers are usually also better written, because the authors understand why things work so they can deprioritize irrelevant details.
I presume your comment is in reference to this speech, as he famously states: "Machine Learning has become alchemy" in it. For anyone who has some time and hasn't watched it, I'd highly recommend it.
Both things can be true and useful. You can search for solutions in a fairly arbitrary manner, and when you find something that works you can attempt to develop a theory to explain it in the hope that said theory will enable more efficient search efforts going forward.
There is no enlightenment in this. Are you going to make the next breakthrough by throwing yet more mud at the wall, or are you going to take educated steps based on a "post-factum" analysis of what worked well so far?
Most people who publish papers in ML are alchemists, but please do not assume all of us are. The original authors of the RoPE paper knowingly designed their embedding to be translation invariant. I do not know if they were aware it was a group representation, but what they were trying to design was in essence a group representation.
That seems to skip a few steps that explain why any of that is important for a positional encoding. I think the original author of RoPE explains the motivation behind it in his own blogpost quite well ( https://spaces.ac.cn/archives/8265 ). And it doesn’t seem to treat group theoretic understanding as something foundational. It mostly relies on engineering goals and geometrical intuition.
> The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc
Can you explain more what you mean here? What do electrons (elementary particles) have to do with protons (complex composite particles) and with general principles of four-dimensional spatial transformation? The electrical force is not a direct consequence of space-time geometry, and even less so the specific particles we happen to find in our universe.
But still - the basic space-time group properties and symmetries don't give you the actual particles, not even a little bit. You could maybe say they give you a concept of a particle, and you can describe the actual particles that happen to exist in our universe using the properties that this group defines - sure. But you can't simply write the group symmetries and discover that a particle with spin 1/2 and mass 0.15MeV has to exist - you just discover that particles in this theory are described by mass and spin, with a certain relation between possible combinations of mass and spin. But there's still an infinite number of such possible combinations, and only 30 or so actual elementary particles.
Plus, calling the particles that you get out of such a theory "the electron" or "the quark" is profoundly misleading. If you only start with a space-time related symmetry, you don't get "electrons", you maybe get particles with the spin and mass of an electron. But it doesn't have charge, nor color nor weak hypercharge. It doesn't decay and doesn't absorb photons nor do anything else that an electron does - it's not an electron. You need to combine this with a lot more symmetry groups to start getting the actual particles, and even then you still need to add the experimentally discovered values to match physics - the math allows for infinite numbers of various particles with properties similar to the ones our universe actually contains.
I'd also note that apparently the Euclidean group doesn't have anything to do with particles anyway, it's the Poincarre group whose irreducible symmetries lead to the (mass, spin) description of particles.
The Euclidean group is a Wick rotation off from the Poincaire group. I think it is valid to call these particles. You can tack on the gauge group later to get your 'qualia'.
Particles yes, but not electrons, quarks, etc. The math gives you "point-like things that have momentum and spin". The gauge groups add charge, color, weak hypercharge (and the associated interactions). And then you add actual parameters you have to get from experiments to discover which of the infinite number of possible particles that have these characteristics actually exist. For example, nothing in the pure math will tell you that there has to exist a particle with electrical charge but no color charge, like the electron and the positron - this is a fact about the universe that comes out of experiments.
I dont think the target audience for the blog is abstract algebra grad students
However would be good to add that the solution can be structural(like in NoPE), then you dont need explicit representations and it just falls out of your architecture.
When I see a title "Understanding..." I assume it will be about understanding theory not implementation or the theory of implementation. When I see a title "Surveying..." or "Understanding Implementation..." I assume it will be about surveying the implementation of that theory. There was just a title mismatch so I felt it was prudent to fill in the gap in understanding the blog did not cover.
Also, I expect the target audience of the blog is ML researchers, since he goes into some of the math, and umm... the kind of ML researcher that is not at least aware of the basics of abstract algebra is closer to an alchemist than a researcher.
At the quantum level, nature seems to use the half spin ireps, but I fail to see if they are useful for 2D images or maybe 3D shapes as inputs for LLMs. Any ideas?
It is called representation theory, and once you have seen even the basics of representation theory it is obvious the authors of the original RoPE paper were trying to rederive a representation for the translation group.
The reason you want to know the general theory behind positional embeddings is it lets you work with other kinds of relative positions. For example, quad/octtrees are useful for faster attention on large images/videos, or a Gaussian splat might want 3D translation + rotation invariance.
I believe you can also go further and start adding in gauge groups and whatnot, but I just thought about that off the top of my head.
This article is only about understanding the implementation of positional embeddings, but to really understand positional embeddings you have to understand the theory. Positional embeddings are group representations. This is how you find them, and this is also how you parameterize them to make them differentiable/learnable. To give a few examples:
1. Rotary positional embeddings are representations of the translation group. The n-dimensional translation group ℝⁿ is abelian, so its irreducible representations are all one-dimensional ρ: ℝⁿ → ℂ, specified by a momentum vector m: ρₘ(x) = exp(i⟨m, x⟩). You can get a multi-dimensional representation by choosing several m, which are your learnable parameters.
2. If you wanted RoPE embeddings to also preserve rotation (perhaps you have scenes viewed at arbitrary angles), you are looking for representations of the special Euclidean group. The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc., and in general are parameterized by a momentum vector and a discrete spin vector. You would have to do some kind of annealing process to train the spin vector (or you could just make it constant).
3. If you want to do attention on quadtrees, well, what is the group? Suppose every node in the quadtree has local coordinates scaled to [0, 1]. Moving up/down/to a neighbor in the quadtree requires an affine transformation of the local coordinates, which compose, so we can label every node by its affine transformation to the root (x, d): y → x + 2⁻ᵈy, where x is the node's translation in the image and d is its depth in the tree. The finite-dimensional irreducible representations are parameterized by momentum vectors m as well as a modulus M and basis vectors {e₁, e₂, ..., eₖ} where k = ord₂(M) and look like ρₘ,ₑ(x, d) = DPᵈ where D is the diagonal matrix with entries Dⱼⱼ = exp(2πi⟨2ʲm, x⟩ / M) and P permutes the basis vectors.
As I understand, this is post-factum rationalization of what positional embedding is, not the actual theory used to build neural networks.
From what I usually see, ML researches just throw everything at the wall and see what sticks. Whatever approach turns out to work better, gets used more often, and sometimes theoretical background is developed for why it works. RoPE works not because it’s good math, just it turned out to work better than some other equally arbitrary approaches at some point, and then other approaches have also been developed that work better than RoPE.
This is the most frustrating part of trying to understand machine learning. You can read a bunch of really great math books that rigorously build concepts from the ground up (calculus, linear algebra), then you start to get into the more hand wavey math (_some_ statistical concepts feel like this) and then you start getting into some of the theory behind machine learning systems and expecting the same amount of rigor and ability to understand _why_ something works and it’s always an unsatisfying explanation. I really wish there would be more disclosure like “this is what we tried and this seems to work, we think this is why” rather than present the method, give some mathematical formalization of the method, then go “it works great” and then move on. It leaves people like me wondering if there is something that I’m missing or some hole in my knowledge that is obvious to everyone else, OR does no one actually know why this works and just don’t want to say. Just give us a hint, make the shrug emoji or something, the more I learn about machine learning, the more I get that it really is more art than science.
Most people who write ML papers are at the stage of alchemy—try a bunch of things and see if you can create gold out of lead. Usually their papers are not worth reading, because even if they do something right, they do not know what made it right, and it cannot lead to further or generalizable insights.
A few ML researchers do actually have a lot of this theoretical knowledge. You can see this in papers like Variational Diffusion Models, Continuous Normalizing Flows, <?> Delta Attention. These papers are usually also better written, because the authors understand why things work so they can deprioritize irrelevant details.
Ali Rahimi's NIPS 2017 Test of Time Award speech lays out that case especially well:
https://youtube.com/watch?v=x7psGHgatGM
I presume your comment is in reference to this speech, as he famously states: "Machine Learning has become alchemy" in it. For anyone who has some time and hasn't watched it, I'd highly recommend it.
Both things can be true and useful. You can search for solutions in a fairly arbitrary manner, and when you find something that works you can attempt to develop a theory to explain it in the hope that said theory will enable more efficient search efforts going forward.
This is exactly correct.
There is no enlightenment in this. Are you going to make the next breakthrough by throwing yet more mud at the wall, or are you going to take educated steps based on a "post-factum" analysis of what worked well so far?
> As I understand, this is post-factum rationalization of what positional embedding is, not the actual theory used to build neural networks
It's always like this because people can't admit ML is just spray and pray. So they superimpose theory.
Most people who publish papers in ML are alchemists, but please do not assume all of us are. The original authors of the RoPE paper knowingly designed their embedding to be translation invariant. I do not know if they were aware it was a group representation, but what they were trying to design was in essence a group representation.
That seems to skip a few steps that explain why any of that is important for a positional encoding. I think the original author of RoPE explains the motivation behind it in his own blogpost quite well ( https://spaces.ac.cn/archives/8265 ). And it doesn’t seem to treat group theoretic understanding as something foundational. It mostly relies on engineering goals and geometrical intuition.
Here's a nice write-up on group theory and positional encodings: https://blog.janestreet.com/using-group-theory-to-explore-po...
This is better than TFA, thank you!
> The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc
Can you explain more what you mean here? What do electrons (elementary particles) have to do with protons (complex composite particles) and with general principles of four-dimensional spatial transformation? The electrical force is not a direct consequence of space-time geometry, and even less so the specific particles we happen to find in our universe.
Oops, should say quarks not protons. I meant just the elementary particles. I cannot edit the comment anymore or I would fix that.
But still - the basic space-time group properties and symmetries don't give you the actual particles, not even a little bit. You could maybe say they give you a concept of a particle, and you can describe the actual particles that happen to exist in our universe using the properties that this group defines - sure. But you can't simply write the group symmetries and discover that a particle with spin 1/2 and mass 0.15MeV has to exist - you just discover that particles in this theory are described by mass and spin, with a certain relation between possible combinations of mass and spin. But there's still an infinite number of such possible combinations, and only 30 or so actual elementary particles.
Plus, calling the particles that you get out of such a theory "the electron" or "the quark" is profoundly misleading. If you only start with a space-time related symmetry, you don't get "electrons", you maybe get particles with the spin and mass of an electron. But it doesn't have charge, nor color nor weak hypercharge. It doesn't decay and doesn't absorb photons nor do anything else that an electron does - it's not an electron. You need to combine this with a lot more symmetry groups to start getting the actual particles, and even then you still need to add the experimentally discovered values to match physics - the math allows for infinite numbers of various particles with properties similar to the ones our universe actually contains.
I'd also note that apparently the Euclidean group doesn't have anything to do with particles anyway, it's the Poincarre group whose irreducible symmetries lead to the (mass, spin) description of particles.
The Euclidean group is a Wick rotation off from the Poincaire group. I think it is valid to call these particles. You can tack on the gauge group later to get your 'qualia'.
Particles yes, but not electrons, quarks, etc. The math gives you "point-like things that have momentum and spin". The gauge groups add charge, color, weak hypercharge (and the associated interactions). And then you add actual parameters you have to get from experiments to discover which of the infinite number of possible particles that have these characteristics actually exist. For example, nothing in the pure math will tell you that there has to exist a particle with electrical charge but no color charge, like the electron and the positron - this is a fact about the universe that comes out of experiments.
I dont think the target audience for the blog is abstract algebra grad students
However would be good to add that the solution can be structural(like in NoPE), then you dont need explicit representations and it just falls out of your architecture.
When I see a title "Understanding..." I assume it will be about understanding theory not implementation or the theory of implementation. When I see a title "Surveying..." or "Understanding Implementation..." I assume it will be about surveying the implementation of that theory. There was just a title mismatch so I felt it was prudent to fill in the gap in understanding the blog did not cover.
Also, I expect the target audience of the blog is ML researchers, since he goes into some of the math, and umm... the kind of ML researcher that is not at least aware of the basics of abstract algebra is closer to an alchemist than a researcher.
I think you're constraining your language to a specific academic dialect and misapplying to other circumstances.
I don't expect a blog post to use the same phrasing as an academic article in several regards, including what you've described.
At the quantum level, nature seems to use the half spin ireps, but I fail to see if they are useful for 2D images or maybe 3D shapes as inputs for LLMs. Any ideas?
This comment is precisely why I opened this article when it had 0 comments in a new tab to come back to.
It's wrong though lol
This is just basic lie theory shoe-horned where it doesn't belong lol.
It is called representation theory, and once you have seen even the basics of representation theory it is obvious the authors of the original RoPE paper were trying to rederive a representation for the translation group.
The reason you want to know the general theory behind positional embeddings is it lets you work with other kinds of relative positions. For example, quad/octtrees are useful for faster attention on large images/videos, or a Gaussian splat might want 3D translation + rotation invariance.
I believe you can also go further and start adding in gauge groups and whatnot, but I just thought about that off the top of my head.