Most servers are not able to access the Google API.
Invidious report of the same(ish) problem: https://github.com/iv-org/invidious/issues/4045#issuecomment-1674373088
That totally sounds possible to me! The other person responding seems to not have very good reading comprehension. Yeah, the only thing stopping this would be drm but apparently most yt videos don’t have drm, since if they did, those downloader apps couldn’t work.
It’s not a half bad idea honestly.