What would it mean for there to be objective moral facts? It would mean that there are true moral statements that live outside of every cultural framework in some sort of transcendent manner, and to arrive at those moral facts would require an individual be able to step out of their subjective, culturally inculcated existence. And as well all know, subjectivity is not the sort of thing you can just step out of.
When you start talking about objective moral facts, you can start talking about societies that fail to adhere to those moral facts as being deficient. And from there it’s a quick hop skip and jump to genocide. Which is how this has played out historically.
I can see the the slippery slope consequences you describe coming into play, but I struggle to see how that negates some facts such as “Rape is morally wrong”