I think the limit should be pretty high, but I’m fine with, as an example, people who spread abject hatred being rejected by most parts of society. I think not spreading hatred against your fellows is an integral part of the social contract.
What about someone who doesn’t think that transgender women are women? Should they be rejected by society for holding that view?