Like if I type “I have two appl…” for example, often it will suggest “apple” singular instead of plural. Just a small example, but it is really bad at predicting which variant of a word should come after the previous
Phones don’t use LLM for predictive text. The algos are a lot less complex on phones.
I guess, the real question is: Could we be using (simplistic) LLMs on a phone for predictive text?
There’s some LLMs that can be run offline and which maybe wouldn’t use enormous amounts of battery. But I don’t know how good the quality of those is…
You can run an LLM on a phone (tried it myself once, with llama.cpp), but even on the simplest model I could find it was doing maybe one word every few seconds while using up 100% of the CPU. The quality is terrible, and your battery wouldn’t last an hour.
Does the AI processing have to be performed locally or constantly active?
The kind of local/offline LLMs that would work on your phone would not be very good quality. There’s been amazing progress in quantization of LLMs to get them working on weaker GPUs with lower VRAM and CPUs, so maybe it’ll occur, but I’m not an expert.
I also don’t foresee them linking it up to a cloud-based LLM as that’d be a shit load of queries and extremely expensive.
OpenAI is probably already handling a significant amount of queries, I think for daily use the LLM should simply initialize a word map based on user history and then update it semi-occasionally, like once a week or two. Most people don’t drastically change their vocabulary in the course of a few weeks
A pre trained model isn’t going to learn how you type the more you use it. Though with Microsoft owning SwiftKey, I imagine they will try it soon
The algorithms are the same. The models are different, being trained on a smaller data set.
No, the algorithms are not the same. Phones don’t use transformer models for text prediction, they use Markov chain-based approaches. Also, retraining of transformer models for individualized completion would be too expensive, whereas it’s basically free with Markov approaches. Where do you get these ideas?
AI is a vast field. LLMs and neural networks are a small part of it.
LLMs are very expensive to run and a lot more complex than the markov chains often used for predictive text.
Predictive text just chooses a likely word based on what’s typed. This may be as simple as looking for words that start with what you’ve typed.
LLMs vectorise words and understand the complex relationship between vectors using many data points. So it would spot the word “two” and realise that plurals are used with it.
Now guess how it feels to type German with predictive text. Most of our words can have more than a dozen different word endings depending on time and how the word is used. And that’s not taking into account that we use compound words, which word prediction pretty much cannot predict and often doesn’t even know. So spell check will mark a legal compound word as misspelled, because it doesn’t understand the concept of compound words and doesn’t know this specific word combination.
To show what I mean, the term “Danube steam boat captain’s hat” becomes “DonauDampfSchiffKapitänsMütze” (I added capital letters which shouldn’t be there to show where the next word in the compound word begins).
While this is an extreme example, it’s pretty common for compound words to consist of 4-5 words.
And for some reason, some cases seem to be missing completely on my Android default keyboard. “untersuchst”, just like a bunch of second person cases for slightly unusual words is non existent.
LLMs are orders of magnitude more sophisticated and expensive to run. But don’t worry, I’m sure not so far in the future we will see smaller LLMs being run on device to be used as autocorrect.
It would have to be pretty specific and small to work on a phone and I think a side effect would be everyone’s conversations start to sound a lot more homogeneous.
Even give years ago, Google had a keyboard that skimmed your emails and texts to start a bank of words you use to supplement it’s dictionary for autocorrect. Like if you are a chemist and send an email that includes the word “tetrahydrafuran” every couple month, it would be nice for your phone keyboard to just have it in the dictionary.
Can we have Scottish ones that know what a bawbag is, and when to put an “e” on the end of “shit”?
Thanks!
Think of it from the LLM’s perspective - in the general pool you have common English, you have less common variations such as this, and then you have whatever the heck people like Kid Rock are doing…
Bawitdaba, da bang, da dang diggy diggy
Diggy, said the boogie, said up jump the boogie
