Yes, the “free” in free software specifically refers to the freedom to read and modify the source code.

That’s the specific freedom we are talking about when we say “free as in freedom”.

Also, under the philosophy, permissive licenses (like MIT or BSD) is still considered free since you can see and modify the source code. The only thing the GPL strive to ensure is that this freedom will be awarded all others who interact with the fruits of your labour.