Substitute a race-based term for trans in your statement - that “normal” people don’t feel comfortable around black people because they just want to avoid them - but it’s not racism.
You might not feel comfortable around terms that have a specific meaning - like transphobia - but you should at least recognize that you’re redefining them to make yourself feel okay about them. What you’re describing is, in fact, transphobia. It is exactly analogous to feeling uncomfortable around black people but not wanting to label it racism.