At the first glance, the 100x difference is surprising. My first thought was that the Rust code [1] uses AVX instructions, which would've explained the difference if the regular wc was a naive C code counting characters in a loop; but from what I see, at least the regular coreutils wc [2] uses AVX too, so the reasons of the 100x difference are not immediately obvious to me. Have you got any good explanation why? I don't believe it's Rust vs C, it might be some optimization that does not exist in the coreutils C code?
I co-maintain GNU coreutils and wrote the Neon 'wc -l' code, for reference.
Based on my quick look at the source code, it looks like this program can handle ASCII whitespace. GNU coreutils uses iswspace, which is locale-aware. That limits how much we can optimize. We could probably add more optimization if the current locale is ASCII, e.g., LC_ALL=C on GNU/Linux.
At the first glance, the 100x difference is surprising. My first thought was that the Rust code [1] uses AVX instructions, which would've explained the difference if the regular wc was a naive C code counting characters in a loop; but from what I see, at least the regular coreutils wc [2] uses AVX too, so the reasons of the 100x difference are not immediately obvious to me. Have you got any good explanation why? I don't believe it's Rust vs C, it might be some optimization that does not exist in the coreutils C code?
[1]: https://github.com/CallMeAlphabet/fastwc/blob/main/src/main....
[2]: https://github.com/coreutils/coreutils/blob/master/src/wc.c
It's doing a tenth of the work, ignoring unicode. Locale Latin1, not much more. No more ws delims than ASCII, no word boundaries.
I co-maintain GNU coreutils and wrote the Neon 'wc -l' code, for reference.
Based on my quick look at the source code, it looks like this program can handle ASCII whitespace. GNU coreutils uses iswspace, which is locale-aware. That limits how much we can optimize. We could probably add more optimization if the current locale is ASCII, e.g., LC_ALL=C on GNU/Linux.