Now this is UX. Wonderful stuff.
Thank god for reader view because this makes me feel physically sick to look at.
I love the comparison of string length of the same UTF-8 string in four programming languages (only the last one is correct, by the way):
Python 3:
len(“🤦🏼♂️”)
5
JavaScript / Java / C#:
“🤦🏼♂️”.length
7
Rust:
println!(“{}”, “🤦🏼♂️”.len());
17
Swift:
print(“🤦🏼♂️”.count)
1
That depends on your definition of correct lmao. Rust explicitly counts utf-8 scalar values, because that’s the length of the raw bytes contained in the string. There are many times where that value is more useful than the grapheme count.
And rust also has the “🤦”.chars().count() which returns 1.
I would rather argue that rust should not have a simple len function for strings, but since str is only a byte slice it works that way.
Also also the len function clearly states:
This length is in bytes, not chars or graphemes. In other words, it might not be what a human considers the length of the string.
That Rust function returns the number of codepoints, not the number of graphemes, which is rarely useful. You need to use a facepalm emoji with skin color modifiers to see the difference.
The way to get a proper grapheme count in Rust is e.g. via this library: https://crates.io/crates/unicode-segmentation
Yeah, and as much as I understand the article saying there should be an easily accessible method for grapheme count, it’s also kind of mad to put something like this into a stdlib.
Its behaviour will break with each new Unicode standard. And you’d have to upgrade the whole stdlib to keep up-to-date with the newest Unicode standards.
It might make more sense to expose a standard library API for unicode data provided by (and updated with) the operating system. Something like the time zone database.
The way UTF-8 works is fixed though, isn’t it? A new Unicode standard should not change that, so as long as the string is UTF-8 encoded, you can determine the character count without needing to have the latest Unicode standard.
Plus in Rust, you can instead use .chars().count() as Rust’s char type is UTF-8 Unicode encoded, thus strings are as well.
turns out one should read the article before commenting
Unicode is thoroughly underrated.
UTF-8, doubly so. One of the amazing/clever things they did was to build off of ASCII as a subset by taking advantage of the extra bit to stay backwards compatible, which is a lesson we should all learn when evolving systems with users (your chances of success are much better if you extend than to rewrite).
On the other hand, having dealt with UTF-7 (a very “special” email encoding), it takes a certain kind of nerd to really appreciate the nuances of encodings.
I’ve recently come to appreciate the “refactor the code while you write it” and “keep possible future changes in mind” ideas more and more. I think it really increases the probability that the system can live on instead of becoming obsolete.
Unrelated, but what do you think (if anything) might end up being used by the last remaining reserved bit in IP packet header flags?
https://en.wikipedia.org/wiki/Evil_bit
https://en.wikipedia.org/wiki/Internet_Protocol_version_4#Header
Man, Unicode is one of those things that is both brilliant and absolutely absurd. There is so much complexity to language and making one system to rule them all ends up involving so many compromises. Unicode has metadata for each character and algorithms dealing with normalization and capitalization and sorting. With human language being as varied as it is, these algorithms can have really wacky results. Another good article on it is https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/
And if you want to RENDER text, oh boy. Look at this: https://faultlore.com/blah/text-hates-you/
Oh no, we’ve been hacked! Theres chinese character in the event log! Or was it just unicode?
The entire video is worth watching, the history of “Plain text” from the beginning of computing.
They believed 65,536 characters would be enough for all human languages.
Gotta love these kind of misjudgements. Obviously, they were pushing against pretty hard size restrictions back then, but at the same time, they did have the explicit goal of fitting in all languages and if you just look at the Asian languages, it should be pretty clear that it’s not a lot at all…
