In the real world practice of small-scale egalitarian societies, these people either get killed, or the group packs up and goes somewhere else. That’s how humanity lived for the hundreds of thousands of years before we invented agriculture.

How we translate that into a contemporary agricultural context where private property and control of resources is a real force is beyond me, but I do think that we have to try.