“In 10 years, computers will be doing this a million times faster.” The head of Nvidia does not believe that there is a need to invest trillions of dollars in the production of chips for AI::Despite the fact that Nvidia is now almost the main beneficiary of the growing interest in AI, the head of the company, Jensen Huang, does not believe that
Despite the fact that Nvidia is now almost the main beneficiary of the growing interest in AI, the head of the company, Jensen Huang, does not believe that additional trillions of dollars need to be invested in the industry.
*Because of
You heard it, guys. There’s no need to create competition to Nvidia’s chips. It’s perfectly fine if all the profits go to Nvidia, says Nvidia’s CEO.
This isn’t necessarily about just hardware. Current ML architectures and inference engines are far from being at peak efficiency. Just last year we saw 20x speedups for llm inference on some hardware. “a million times” is obviously hyperpole though.
He doesn’t want a new competitor. He’s just spouting whatever will make the line move up. It has nothing to do with his opinion.
Honestly as someone who has watched the once-fanciful prefixes “giga” and “tera” enter common parlance, and saw kilobytes of RAM turn to gigabytes, it’s really hard for me to think what he’s saying is impossible.
Even if he is accurate, specialist hardware will outperform generic hardware at what it is specialized for.
I remember a story sometime in the 00s about PCs finally getting to the point where they were as fast as one of the WWII code breaking computers (or something like that). It wasn’t because we backtracked in computer speeds after WWII, but because even that ancient hardware was able to get good performance when it was purpose-built, but it couldn’t do anything else and likely would have required a lot of work to adjust to a different kind of cypher scheme, if it could be adapted at all.
So GP compute might be a million times faster in a decade, but specialist AI chips might be a million times faster than that.
A hardware neural net might be able to eliminate memory latency by giving each neuron fast resisters to handle all their memory needs. If it doesn’t need to change connections, each connection could be hard wired. A GPU wouldn’t have a chance at keeping up no matter how wide that memory bus gets or how many channels it gets split into. It might even use way less power (though with the elimination of memory latency, it could go fast enough to use way more, too).
Nobody is saying it won’t happen eventually. But a million times within the next decade, i.e. 4x better every year for 10 years?
This generation isn’t better than last generation by even close to that. Nevermind doing 4x for 10 years straight.
Sorry I have doubts, because that would require a factor 4x increase every year for 10 years! 4x^10 = 1,048,576x
Considering they historically have had problems achieving just twice the speed per year, it does not seem likely.
Why does that make a difference? Compute for AI is build on the progress for compute first for GPU then for data center. They are similar in nature.
Yes they have exceeded 2x for AI for a while, but that has been achieved through exploding die size and cost, but even that won’t make a million times faster in 10 years possible, because they can’t increase die sizes any further.
Building an ASIC for purpose built computation is significantly faster than generic gpu compute cores. Like when ASICs were built for bitcoin mining/sha256 and a little 5 watt usb device could outperform the best GPUs
Yes, but usually we keep those 2 kinds of optimizations separate, only combining chip design and production process. Because if the software is optimized, the hardware isn’t really doing the same thing.
So yes AI speed may increase more than just the hardware, but for the most sophisticated systems, the tasks will be more complex, which may again slow the software down.
So I think they will never be able to achieve it even when considering software optimizations too. Just the latest Tesla cars boast about 4 times higher resolution cameras, that will require 4 times the processing power to process image recognition, which then will be more accurate, but relatively slower.
We are not where we want to be, and the systems of the future will clearly be more complex, and on the software are more likely to be slower than faster.
even software that does the same thing gets slower example: Microsoft Office, Amazon, the web in general, etc.
