With bitmasks, if they just assumed bit 5 (whatever) as zero=whitespace, they would have saved billions of cpu cycles per device per day. With 32 bit unicode, that gives you 32 unique whitespace chars.
Should have got guys that grew up with the 8-bit micros to write up a spec.
Guys that grew up with the 8-bit micros did write the spec—the first design drafts of the standard started circulating in 1988, version 1.0 came out in 1991. Given IBM’s primary role in the whole thing, even parents of such guys could’ve been involved.
The problem was, they had to sell their invention to other guys that had grown up with 8-bit micros and in the meantime had gotten very attached to their specific 8-bit encodings, so they had to make a lot of compromises: first 128 characters match ASCII; first 256 characters match Latin-1; alphabets encoded in order; codepoint-to-codepoint transcoding from any (reasonable) legacy encoding; throw out enough stuff to not need more than 2^16 characters. They were also racing against time, because in the meanwhile ISO was cooking up 10646, the Universal Character Set, built around ISO 2022’s teletype-inspired architecture of charset switching, and that only got killed because software vendors essentially declared a boycott and pointed to Unicode instead.
The result came out rather awkward as well as fairly expensive, and a lot of the promises needed to be broken afterwards anyway. But they’d have no users had they had done things differently. (Meanwhile, the UCS as initially conceived lacked such advanced operations as string equality.)
I don't follow? If bit 5 == zero meant "this is whitespace", then white-space characters would take up fully half of the unicode space. There would be more than 2 billion of them, not 32. Seems a bit overkill...
That wouldn't work. As the "Nothing is set in stone" section points out, in Unicode 6.2 the MONGOLIAN VOWEL SEPARATOR went from space separator to a format character.
This code is very simple but I still don't really get it. Why is the hashing needed and how is it faster than a regex? Can someone smarter than me explain?
No branching (if statements), no heavy CPU operations, just a multiplication by a constant, a shift and a lookup in a small table that will get cached fast
You need a hashtable to check whether this character is part of a given set. And the perfect hashing-based implementation is the most efficient form of a hashtable since you only need a lookup and an equality check. I included a small visualisation in the article that illustrates how it works, so if you try clicking on the elements, it will show you how the calculations are made.
Whitespace depends on the context. For example, you could have characters that are not counted as whitespace in Unicode, but end up rendering as whitespace when displayed in a particular context. The only real way to check for this is to actually try to render it and see what happens. There are some other ways to approximate it, like checking the font that you use for rendering, or the bounds of the glyph, or other things like that, but the only way to really know for sure in all cases is to just render it and see what happens.
One example is a field that clips its contents to contain zalgo, but inadvertantly to the point where some glyphs can render completely out of bounds. Sometimes the clipping is so egregious that even descenders of normal letters like 'g' get cut off. (This is more common than you'd think.)
True, rendering out of bounds doesn't make something actually whitespace, it just means it could look like whitespace. Though, consider the case where someone is looking to detect if a character is whitespace, when what they actually want is to detect if it could look like whitespace. That's where context comes into play. Irrelevant if it's not about the looks, but potentially relevant if it is, hence why I figured I'd leave a quick comment about it here.
I don't think you can "just render it" to figure out it's whitespace?
As the section "What is whitespace, anyway?" points out, if you render an OGHAM SPACE MARK in a font with a stemline you'll get a visible horizontal line.
> For instance, ZERO WIDTH SPACE (U+200B) isn't a whitespace character at all. Despite the name, it's a format character that can indicate a word or line break opportunity without adding a visible gap.
that's what happens when you throw a lot of garbage in, mix it up and forget to clean (piece of trivia - this was a space char many years ago; another one - the Braille whitespace char is unitechnically not whitespace)
> So even the assumption that whitespace must look empty isn't always true.
indeed, another obvious fail of this classification ("naming is hard")
> not frozen in time, and is constantly evolving
unfortunatly the evolution is freezing, so such issues remain unfixed. Here is a few emojis as a consolation prize instead!
With font.color = white even an ordinary "W" could count as whitespace...
Something to remember if you see whitespace with an extraordinary width out there in the wild...
The \s character class in JavaScript regular expressions includes 25 characters. I don’t know if that perfectly matches the Unicode list of 25 characters, but it does make identifying white space simplistic.
White space characters are challenging to differentiate in human readable text, but JavaScript makes that perfectly clear to humans by splitting a string on every character, which is helpful for debugging.
The odd one out above is NEXT LINE (U+0085). Unicode explicitly includes it in White_Space, and Python accepts it too, but JavaScript doesn't. Instead, JavaScript recognises ZERO WIDTH NO-BREAK SPACE (U+FEFF), which Unicode doesn't include in the set.
I was here before the first character was carved into stone and I will be here when the last photon is emitted.
I'm also available via: @gmail.com
(please don't write code with `@whitespace` as I get a lot of GitHub notifications)
Ah, a whitespace supremacist!
I think horizontal white spacing should be distinguished, as that's what most care about with splitting etc.
Have a look at how GNU coreutils classifies each type of spacing (in a cross platform consistent way) at https://github.com/coreutils/coreutils/blob/master/src%2Fsys...
If you want to learn how that magic multiplier value was derived, look up how to derive Perfect Hash Functions for a known set of values.
I'm not sure which method was used for this specific set of values, but from googling it's seen advances in the past decade.
With bitmasks, if they just assumed bit 5 (whatever) as zero=whitespace, they would have saved billions of cpu cycles per device per day. With 32 bit unicode, that gives you 32 unique whitespace chars.
Should have got guys that grew up with the 8-bit micros to write up a spec.
Guys that grew up with the 8-bit micros did write the spec—the first design drafts of the standard started circulating in 1988, version 1.0 came out in 1991. Given IBM’s primary role in the whole thing, even parents of such guys could’ve been involved.
The problem was, they had to sell their invention to other guys that had grown up with 8-bit micros and in the meantime had gotten very attached to their specific 8-bit encodings, so they had to make a lot of compromises: first 128 characters match ASCII; first 256 characters match Latin-1; alphabets encoded in order; codepoint-to-codepoint transcoding from any (reasonable) legacy encoding; throw out enough stuff to not need more than 2^16 characters. They were also racing against time, because in the meanwhile ISO was cooking up 10646, the Universal Character Set, built around ISO 2022’s teletype-inspired architecture of charset switching, and that only got killed because software vendors essentially declared a boycott and pointed to Unicode instead.
The result came out rather awkward as well as fairly expensive, and a lot of the promises needed to be broken afterwards anyway. But they’d have no users had they had done things differently. (Meanwhile, the UCS as initially conceived lacked such advanced operations as string equality.)
To be fair, Unicode also isn't great for string equality
I don't follow? If bit 5 == zero meant "this is whitespace", then white-space characters would take up fully half of the unicode space. There would be more than 2 billion of them, not 32. Seems a bit overkill...
That wouldn't work. As the "Nothing is set in stone" section points out, in Unicode 6.2 the MONGOLIAN VOWEL SEPARATOR went from space separator to a format character.
This code is very simple but I still don't really get it. Why is the hashing needed and how is it faster than a regex? Can someone smarter than me explain?
No branching (if statements), no heavy CPU operations, just a multiplication by a constant, a shift and a lookup in a small table that will get cached fast
> Why is the hashing needed and how is it faster than a regex? Can someone smarter than me explain?
It's a perfect hash (i.e. no collisions) that hash to an entry in a 32-element array.
Very similar to how we usually do quick lookups of strings for `switch()` statements: https://github.com/lelanthran/libxtree/blob/878cb4fc11ffd617...
You need a hashtable to check whether this character is part of a given set. And the perfect hashing-based implementation is the most efficient form of a hashtable since you only need a lookup and an equality check. I included a small visualisation in the article that illustrates how it works, so if you try clicking on the elements, it will show you how the calculations are made.
You do not need a hashtable to check that, code that returns a TRUE if (opt1)|(opt2)|(opt3)|(op4)|(etc) matches or FALSE otherwise works just fine.
How do you think regular expressions are implemented...?
Not necessarily as hashmaps? They sure can be.
Whitespace depends on the context. For example, you could have characters that are not counted as whitespace in Unicode, but end up rendering as whitespace when displayed in a particular context. The only real way to check for this is to actually try to render it and see what happens. There are some other ways to approximate it, like checking the font that you use for rendering, or the bounds of the glyph, or other things like that, but the only way to really know for sure in all cases is to just render it and see what happens.
My tab renders as: " > ". I still consider it whitespace.
What turns into whitespace due to rendering? Just because my font lacks that symbol doesn't mean it is whitespace.
One example is a field that clips its contents to contain zalgo, but inadvertantly to the point where some glyphs can render completely out of bounds. Sometimes the clipping is so egregious that even descenders of normal letters like 'g' get cut off. (This is more common than you'd think.)
I get the desire to filter that out, not sure whether I'd classify it as whitespace though.
True, rendering out of bounds doesn't make something actually whitespace, it just means it could look like whitespace. Though, consider the case where someone is looking to detect if a character is whitespace, when what they actually want is to detect if it could look like whitespace. That's where context comes into play. Irrelevant if it's not about the looks, but potentially relevant if it is, hence why I figured I'd leave a quick comment about it here.
I don't think you can "just render it" to figure out it's whitespace?
As the section "What is whitespace, anyway?" points out, if you render an OGHAM SPACE MARK in a font with a stemline you'll get a visible horizontal line.
> For instance, ZERO WIDTH SPACE (U+200B) isn't a whitespace character at all. Despite the name, it's a format character that can indicate a word or line break opportunity without adding a visible gap.
that's what happens when you throw a lot of garbage in, mix it up and forget to clean (piece of trivia - this was a space char many years ago; another one - the Braille whitespace char is unitechnically not whitespace)
> So even the assumption that whitespace must look empty isn't always true.
indeed, another obvious fail of this classification ("naming is hard")
> not frozen in time, and is constantly evolving
unfortunatly the evolution is freezing, so such issues remain unfixed. Here is a few emojis as a consolation prize instead!
With font.color = white even an ordinary "W" could count as whitespace... Something to remember if you see whitespace with an extraordinary width out there in the wild...
The \s character class in JavaScript regular expressions includes 25 characters. I don’t know if that perfectly matches the Unicode list of 25 characters, but it does make identifying white space simplistic.
White space characters are challenging to differentiate in human readable text, but JavaScript makes that perfectly clear to humans by splitting a string on every character, which is helpful for debugging.
The article covers that: