Many of the peoples who speak Semetic languages are in fact closely related, even if not all of the peoples are by basis of speaking the languages.
And yes, I agree with the second paragraph that the term is a poor one (for reasons I further laid out in my comment) and should be retired for two different terms regarding bias against Judaism and bias against ethnic Jews.