Soap does destroy some bacteria, and a not insignificant portion. By destroying those fatty bonds the cellular membranes of many bacteria are destroyed, and many viruses denatured and rendered inert.

The removal is the primary action though, you are correct. Not all bacteria are destroyed by soap, which is why the leather, scrub, and scrub while rinsing steps are important to hand washing, since that mechanical action is what removes everything.

https://medicine.yale.edu/news-article/why-soap-works/