It’s not just what they tell you. There are plenty of publicly accessible LLM models. Go and download them and open the files up. Surely if they are storing these things as complete data, you can easily find them by poking around the files instead of having to make then spit it out.
I’m aware of the availability of them, I’ve looked into building a private install of GPT4All. Even though we can look into those files directly, it doesn’t prove that the large “AI” systems run by the mega-corps are not storing copyrighted data. The only thing that could prove that is a complete audit of all the data storage that their “AI” systems have access to.
This will likely play out in the courts due to the numerous lawsuits in process from artists suing over their work being stolen. Legal discovery could compel that kind of data audit.