Google’s AI-driven Search Generative Experience have been generating results that are downright weird and evil, ie slavery’s positives.
Unfortunately, people who grow up in racist groups also tend to be racist. Slavery used to be considered normal and justified for various reasons. For many, killing someone who has a religion or belief different than you is ok. I am not advocating for moral relativism, just pointing out that a computer learns what is or is not moral in the same way that humans do, from other humans.
You make a good point. Though humans at least sometimes do some critical thinking between absorbing something and then acting it out.