But I’m sure we’d be loosing good people from hexbear too

Not really, the good people will just make accounts elsewhere and start using those accounts to interact here and on the rest of the fediverse, most of the good ones already do that because hexbear is already very isolated by virtue of the fact they’re using whitelist federation and also the fact that they already have been defederated for the spam, trolling, and harassment their less savory members bring about (and more importantly the fact that it is unpunished by their instance’s moderators if not encouraged).