You’re right.
I like to peruse code and have read a lot of it from the sources that make it available. It’s not always the languages I know but even then I can get the idea of what most of it is doing. There are some code bases that are too big for any one person to fully comprehend. That said, I think the only way for one to be confident in open source is to read it yourself which is a problem for most as coding knowledge is not common combine with the size of some.
So it’s always going to be trusting trust for most people. The fact that it is open source and makes available the code for review limits malice to a much greater degree than proprietary ever will.