It absolutely cannot be a privacy issue.
It takes the same amount of work as manually sharing your number. It cannot happen without deliberate action.
The feature does require confirmation.
It also requires accessing your contacts database, which is encrypted on iPhones…
Because it’s encrypted, it’s impossible to share contact details unless someone enters the device passcode (or else does a biometric unlock - which effectively stores your passcode temporarily in a secure location that is wiped whenever the device is powered off or left unused for several hours).