If you use -ffast-math when linking an executable but not a shared library, both gcc and clang will link in crtfastmath.o which has the bit of code to set the DAZ/FTZ flags.
right, but the issue was that if an application was linked with a library that happened to link with a fast-math shared library, it would unknowingly bring along the crt code. Now the only way to get it is to use the fast-math flag when linking the final binary, which at least is an explicit request.
It used to be quite normal to add a low level random signal to inputs when writing DSP code so as to avoid dropping into subnormal territory. Careful analysis of the algorithm would identify any points where this was also necessary (e.g. feedback paths when running delays).
Obviously those lucky/unlucky enough to be writing 56k fixed precision code wouldn't have this concern, but other ones instead :)
I think flush to zero is probably the preferred strategy these days.
That's what I mean though- in the Core 2 Duo vs K8 / Athlon days, Intel was much slower for denormals and subnormals than AMD. I'm not sure why but I thought this was common knowledge.
Probably different teams. Plus you can not underestimate the role that momentum plays in semiconductor engineering teams. If some respected person determined that subnormals are either Hard (tm) or not a Real Problem (tm), it will take a long time to correct this mistaken belief. I believe there have been some recent academic papers on FP implementation from the Intel E-Core team, which is a sign that they are a bit more with the times.
They are designed by different teams and have different ways of doing math. You could just as easily say “why are the efficiency cores wasting an extra cycle for each multiplication doing fixups of an uncommon case”.
I'm still trying to understand what a subnormal number is; IE, I'm looking for the TLDR so I know just enough to know if I'm using them and need to learn more.
Unfortunately, the Wikipedia article, while probably being accurate, doesn't give a clear and concise answer.
IE, is 0.0001 a subnormal? Or is it 0.000000000000000000001?
where sign, mantissa and exponent are fixed bit width integers. The 1. before the number is normally implicit because it would be a waste of a bit to encode it when you could just use a diferent exponent to represent such a number.
However with this simple scheme the number zero and a relatively large gap around it cannot be represented (relatively large to the gap between the smallest and next smalles number that can be represented).
So there is a special case where for the smallest encodeable exponent the mantissa must also specify that 1. or 0. prefix. Because its a special case it needs special handling that clever silicon engineers might think is unimportant enough to handle in microcode instead of dedicated silicon.
x86 has a mode to assume that all such small numbers are actually equal to zero which can then be handle without microcode fallback. Technically its even a bit more complicated because x86 has two different float implementations and for at least SSE floats you can control the denormals-are-zero and flush-(denormals)-to-zero-(when writing) modes independently. GCC -ffast-math actual enables that mode for the entire main thread.
AFAIK ARM NEON always works in that mode so the Gravion and Apple benchmarks might be unfair here undless you compare with DAZ and FTZ enabled on Intel. No idea if the AMD benchmarks might have used different modes. Because the flags are global per thread you can easily have unrelated loaded libraries messing the benchmark up.
However, in IEEE, the exponent cannot be made arbitrary small. Because of that, some very small numbers cannot be represented that way.
In those cases the standard says operations can return numbers with the value closest to the correct value with a significant less than 1. Those number representations are called subnormals.
The 32-bit subnormals are all the non-zero 32-bit floating point values between but not including
-0.000000000000000000000000000000000000011754943508222875079687365372222456778186655567720875215087517062784172594547271728515625 and +0.000000000000000000000000000000000000011754943508222875079687365372222456778186655567720875215087517062784172594547271728515625
Does that help you?
[Edited: correct decimal after noticing that my calculator defaulted to the wrong setting]
[And again because I think there's a bug in the last few digits, so debugging that's a fun activity for the weekend]
[And a third time because nope, those were correct and I can't type]
Very interesting, you are right. The IEEE 754 standard defines positive/negative zero as not subnormal numbers.
Mechanically speaking, the two zeros use the subnormal number format, so in that sense they are subnormal (but definitionally they aren't). Also, I guess FPUs treat zero differently from other subnormal numbers, which is why zero doesn't have a performance penalty.
Binary floating-point numbers are scientific notation except the pieces are all in binary. In proper scientific notation, the only time the digit before the decimal point can be 0 is when the number itself is 0. Since the only other digit in binary notation is 1, there is no need to store the digit before the decimal point, since it's always 1... except now you can't store 0.
This problem is fixed by reserving one of the exponents for the representation of 0. Some of the formats (e.g. VAX floating point) that introduced this implicit-1-bit for the binary format said that every number with this special-0-exponent was a zero. But IEEE 754 introduced the concept of gradual underflow, and says instead that it is a bit string with the implicit digit before the decimal point as a 0 instead of 1.
Putting it differently and more succinctly: a subnormal number is a number that has fewer digits of precision than is normally implied by the format. Which numbers are subnormal numbers is entirely dependent on the floating-point format.
For 32-bit floats, subnormals are the numbers closer to 0 than 2**(-126) == 0.0000000000000000000000000000000000000117549.
For 64-bit doubles, it's 2**(-1022), a number starting with 308 decimal zeroes.
Usually IEEE floats have an implied 1 in the front. So for the standard represented numbers, there's some minimum number 1.bbbbbb.. * 2^-N. This allows 1bit more precision than is actually stored.
between any two numbers, there's basically the same epsilon difference, but from the smallest number to zero it's bigger.
A subnormal number breaks that convention, it just becomes 0.bbbbb... * 2^-N. As the numbers get smaller, the relative difference between the numbers gets larger. That also means their precision is smaller than the normal floats.
Subnormal numbers have a different, basically fixed-point, representation. They exist in order to bridge the large (relatively speaking; indeed "infinite" in a sense) gap between the least positive normal number, zero, and the greatest negative normal number, caused by the usual significand-exponent representation.
Most "mundane" uses of floating point have no need for subnormal numbers, and results that underflow could just be flushed to zero. But they’re sometimes important in scientific computing to ensure sufficient smoothness around zero, avoiding precision issues.
I can think of one useful property of subnormal numbers off the top of my head. If subnormal processing is enabled, then for all finite values of `a` and `b`, `a != b` if and only if `a - b != 0`. But if subnormals are flushed to zero, then two tiny normal distinct values `a` and `b` would have a subnormal difference that is flushed to zero.
Isn't that just a scale issue that exists with or without subnormals? If a and b are closer to zero than the smallest representable number, a and b compare as the same. With subnormals your smallest possible number is smaller than without, but it's still the same issue.
But that same statement applies to normal values too, right? With normal numbers you might get the oddity of a-b -> a even if b is nonzero, but you don't get the oddity of a-b -> 0 unless the same number is represented, IIUC. A and B might not be bit identical, but they represent the same number if the difference is 0.
One nice thing that subnormals get you is the property that if x-y == 0 then x == y. If you want to guard against division by 0, and your denominator is a difference of two terms, it’s nice to be able to check equality of those terms and know that if they are not equal, then their difference will not be 0.
I don’t know how useful they are in scientific computing either, really. They are less precise than normalized numbers… if flushing them makes a difference I think it is a bad algorithm smell.
I don’t know if any bugs contribute to this but this in the intel case but it has been very common historically for subnormal performance to be lower on many processors, and things like the Alpha required you to handle them in software if the COU fired a trap.
It probably means Apple spent the silicon to handle subnormals at full speed in hardware, rather than triggering a slow microcode handler for such numbers.
I don't know who is down voting you. AFAIK IEEE 754:2008 does require support for subnormals. You can optionally have modes that flush them to zero, but you must support subnormals.
I haven't done any work on this stuff since 2019, so my memory may be hazy.
If you don't _need_ subnormals MXCSR.DAZ/FTZ (which you can get gcc to set via -mdaz-ftz) will let you ignore all of this.
IIRC intel’s compilers enable FTZ/DAZ, at least at higher optimization levels.
GCC does with the infamous -ffast-math as well.
IIRC that has been since split out of the flag and need to be asked for separately at link time.
If you use -ffast-math when linking an executable but not a shared library, both gcc and clang will link in crtfastmath.o which has the bit of code to set the DAZ/FTZ flags.
right, but the issue was that if an application was linked with a library that happened to link with a fast-math shared library, it would unknowingly bring along the crt code. Now the only way to get it is to use the fast-math flag when linking the final binary, which at least is an explicit request.
It used to be quite normal to add a low level random signal to inputs when writing DSP code so as to avoid dropping into subnormal territory. Careful analysis of the algorithm would identify any points where this was also necessary (e.g. feedback paths when running delays).
Obviously those lucky/unlucky enough to be writing 56k fixed precision code wouldn't have this concern, but other ones instead :)
I think flush to zero is probably the preferred strategy these days.
This has been the case since a zillion years, since the Core 2 Duo days at minimum.
The interesting part is that this seems to be Intel-specific.
That's what I mean though- in the Core 2 Duo vs K8 / Athlon days, Intel was much slower for denormals and subnormals than AMD. I'm not sure why but I thought this was common knowledge.
This is true on most hardware. Running ftz on H100s or B200 gives a free 10-20% boost for GEMM-epilogue workload
Apparently it only happens on P-cores, recent E-cores have a fast path for subnormals.
That seems weird. They did throw extra hardware at it to speed it up for the efficient cores, but not for the performance cores?
Probably different teams. Plus you can not underestimate the role that momentum plays in semiconductor engineering teams. If some respected person determined that subnormals are either Hard (tm) or not a Real Problem (tm), it will take a long time to correct this mistaken belief. I believe there have been some recent academic papers on FP implementation from the Intel E-Core team, which is a sign that they are a bit more with the times.
They are designed by different teams and have different ways of doing math. You could just as easily say “why are the efficiency cores wasting an extra cycle for each multiplication doing fixups of an uncommon case”.
Fabian Giesen explains in more detail here: https://mastodon.gamedev.place/@rygorous/117277063419144390
larger SIMD ALUs, which also have been around for longer?
I'm still trying to understand what a subnormal number is; IE, I'm looking for the TLDR so I know just enough to know if I'm using them and need to learn more.
Unfortunately, the Wikipedia article, while probably being accurate, doesn't give a clear and concise answer.
IE, is 0.0001 a subnormal? Or is it 0.000000000000000000001?
Floating point numbers are usually interpreted as
where sign, mantissa and exponent are fixed bit width integers. The 1. before the number is normally implicit because it would be a waste of a bit to encode it when you could just use a diferent exponent to represent such a number.However with this simple scheme the number zero and a relatively large gap around it cannot be represented (relatively large to the gap between the smallest and next smalles number that can be represented).
So there is a special case where for the smallest encodeable exponent the mantissa must also specify that 1. or 0. prefix. Because its a special case it needs special handling that clever silicon engineers might think is unimportant enough to handle in microcode instead of dedicated silicon.
x86 has a mode to assume that all such small numbers are actually equal to zero which can then be handle without microcode fallback. Technically its even a bit more complicated because x86 has two different float implementations and for at least SSE floats you can control the denormals-are-zero and flush-(denormals)-to-zero-(when writing) modes independently. GCC -ffast-math actual enables that mode for the entire main thread.
AFAIK ARM NEON always works in that mode so the Gravion and Apple benchmarks might be unfair here undless you compare with DAZ and FTZ enabled on Intel. No idea if the AMD benchmarks might have used different modes. Because the flags are global per thread you can easily have unrelated loaded libraries messing the benchmark up.
> AFAIK ARM NEON always works in that mode
This was only true for ARMv7 NEON (32-bit). ARMv8 / AArch64 NEON is IEEE compliant.
That depends on how you store it.
Each number can be written in infinitely many ways, for example 12, 1.2E1, and 0.012E3 all are “twelve”
In (binary) IEEE floats, the canonical way to write floats is
with 1 ≤ significant < 2. So, “twelve” gets stored as 1.5 × 2³ and not as, for example, 0.375 × 2⁵, 12 × 2⁰ or 96 × 2⁻³.Float operations normally return numbers satisfying that.
However, in IEEE, the exponent cannot be made arbitrary small. Because of that, some very small numbers cannot be represented that way.
In those cases the standard says operations can return numbers with the value closest to the correct value with a significant less than 1. Those number representations are called subnormals.
The 32-bit subnormals are all the non-zero 32-bit floating point values between but not including -0.000000000000000000000000000000000000011754943508222875079687365372222456778186655567720875215087517062784172594547271728515625 and +0.000000000000000000000000000000000000011754943508222875079687365372222456778186655567720875215087517062784172594547271728515625
Does that help you?
[Edited: correct decimal after noticing that my calculator defaulted to the wrong setting]
[And again because I think there's a bug in the last few digits, so debugging that's a fun activity for the weekend]
[And a third time because nope, those were correct and I can't type]
Very interesting, you are right. The IEEE 754 standard defines positive/negative zero as not subnormal numbers.
Mechanically speaking, the two zeros use the subnormal number format, so in that sense they are subnormal (but definitionally they aren't). Also, I guess FPUs treat zero differently from other subnormal numbers, which is why zero doesn't have a performance penalty.
Relevant articles to read: https://stackoverflow.com/questions/73890260/why-is-zero-not... , https://en.wikipedia.org/wiki/Sterbenz_lemma , https://en.wikipedia.org/wiki/Subnormal_number
Yes, that's a perfect explanation!
(And thankfully, for when I work with small numbers, they are still much larger than that.)
Binary floating-point numbers are scientific notation except the pieces are all in binary. In proper scientific notation, the only time the digit before the decimal point can be 0 is when the number itself is 0. Since the only other digit in binary notation is 1, there is no need to store the digit before the decimal point, since it's always 1... except now you can't store 0.
This problem is fixed by reserving one of the exponents for the representation of 0. Some of the formats (e.g. VAX floating point) that introduced this implicit-1-bit for the binary format said that every number with this special-0-exponent was a zero. But IEEE 754 introduced the concept of gradual underflow, and says instead that it is a bit string with the implicit digit before the decimal point as a 0 instead of 1.
Putting it differently and more succinctly: a subnormal number is a number that has fewer digits of precision than is normally implied by the format. Which numbers are subnormal numbers is entirely dependent on the floating-point format.
For 32-bit floats, subnormals are the numbers closer to 0 than 2**(-126) == 0.0000000000000000000000000000000000000117549. For 64-bit doubles, it's 2**(-1022), a number starting with 308 decimal zeroes.
What about 8-bit and 16-bit?
8-bit: There is no IEEE 754 standard format.
16-bit: https://en.wikipedia.org/wiki/Half-precision_floating-point_...
Usually IEEE floats have an implied 1 in the front. So for the standard represented numbers, there's some minimum number 1.bbbbbb.. * 2^-N. This allows 1bit more precision than is actually stored.
between any two numbers, there's basically the same epsilon difference, but from the smallest number to zero it's bigger.
A subnormal number breaks that convention, it just becomes 0.bbbbb... * 2^-N. As the numbers get smaller, the relative difference between the numbers gets larger. That also means their precision is smaller than the normal floats.
Are the results compared across architectures?
The article covers 5 CPUs.
I mean the values of the computation not the runtime. I don’t know enough about ARM to say if doubles simply punt denormals to 0 for example.
IEEE 754 defines bit exact results for a lot of FP operations, including denormals.
And yet floating point math in general is non-deterministic across different CPUs. IEEE 754 was not good enough, so it's a valid question.
...Is this running extra micro code to fix some hardware bug/unreliability? How can this happen? Doesn't look like a normal design decision.
Subnormal numbers have a different, basically fixed-point, representation. They exist in order to bridge the large (relatively speaking; indeed "infinite" in a sense) gap between the least positive normal number, zero, and the greatest negative normal number, caused by the usual significand-exponent representation.
Most "mundane" uses of floating point have no need for subnormal numbers, and results that underflow could just be flushed to zero. But they’re sometimes important in scientific computing to ensure sufficient smoothness around zero, avoiding precision issues.
I can think of one useful property of subnormal numbers off the top of my head. If subnormal processing is enabled, then for all finite values of `a` and `b`, `a != b` if and only if `a - b != 0`. But if subnormals are flushed to zero, then two tiny normal distinct values `a` and `b` would have a subnormal difference that is flushed to zero.
Isn't that just a scale issue that exists with or without subnormals? If a and b are closer to zero than the smallest representable number, a and b compare as the same. With subnormals your smallest possible number is smaller than without, but it's still the same issue.
Because subnormals are fixed point, ie. have a fixed exponent, the difference of any two distinct subnormal values is nonzero like with integers.
But that same statement applies to normal values too, right? With normal numbers you might get the oddity of a-b -> a even if b is nonzero, but you don't get the oddity of a-b -> 0 unless the same number is represented, IIUC. A and B might not be bit identical, but they represent the same number if the difference is 0.
One nice thing that subnormals get you is the property that if x-y == 0 then x == y. If you want to guard against division by 0, and your denominator is a difference of two terms, it’s nice to be able to check equality of those terms and know that if they are not equal, then their difference will not be 0.
More generally, subnormals are needed for Sterbenz Lemma to hold everywhere: https://en.wikipedia.org/wiki/Sterbenz_lemma
I don’t know how useful they are in scientific computing either, really. They are less precise than normalized numbers… if flushing them makes a difference I think it is a bad algorithm smell.
I don’t know if any bugs contribute to this but this in the intel case but it has been very common historically for subnormal performance to be lower on many processors, and things like the Alpha required you to handle them in software if the COU fired a trap.
Have a look at https://en.wikipedia.org/wiki/Subnormal_number for some context.
It's to satisfy IEEE 754 and it's been this way for decades.
Does that mean that the ARM processors in the writeup are not satisfying IEEE 754?
It probably means Apple spent the silicon to handle subnormals at full speed in hardware, rather than triggering a slow microcode handler for such numbers.
I don't know who is down voting you. AFAIK IEEE 754:2008 does require support for subnormals. You can optionally have modes that flush them to zero, but you must support subnormals.
I haven't done any work on this stuff since 2019, so my memory may be hazy.
[dead]
[flagged]