Most servers are not able to access the Google API.
Invidious report of the same(ish) problem: https://github.com/iv-org/invidious/issues/4045#issuecomment-1674373088
It’s important to know know that by this google still can’t run their own fingerprinting scripts or fingerprinting stylesheets, nor set cookies, so while not ideal, it’s not that bad.
I would suggest that (except when you watch something more private) until this gets somewhat more sorted, turning off video proxying is probably not a huge tradeoff. Also don’t forget that FreeTube and NewPipe don’t do proxying (FT can be set to do that, though)