Over just a few months, ChatGPT went from correctly answering a simple math problem 98% of the time to just 2%, study finds. Researchers found wild fluctuations—called drift—in the technology’s abi…::ChatGPT went from answering a simple math correctly 98% of the time to just 2%, over the course of a few months.
This is what was addressed at the start of the comment, you can just roll back to a previous version. It’s heavily ingrained in CS to keep every single version of your software forever.
I don’t think it’s that easy. These are vLLMs that feed back on themselves to produce “better” results. These models don’t have single point release cycles. It’s a constantly evolving blob of memory and storage orchestrated across a vast number of disk arrays and cabinets of hardware.
[e]I am wrong the models are version controlled and do have releases.
That’s not how these LLMs work. There is a training phase which takes a large amount of compute power, and the training generates a model which is a set of weights and could easily be backed up and version-controlled. The model is then used for inference which is a less compute-intensive process and runs on much smaller hardware than the training phase.
The inference architecture does use feedback mechanisms but the feedback does not modify the model-weights that were generated at training time.
For simple language models sure but we’re talking about chatGPT here. OpenAI has some pretty bold claims…
100 trillion bites is 100 terrabytes and if you have any amount of actual data in those parameters then the size of the data could easily get into the petabyte range.
