Rudeus is a really, really flawed character, and it definitely means that the anime and LN won’t be for everyone, but if you can accept it, there’s a lot there to enjoy. I agree that it makes it difficult to recommend to people.
I’ve read up through volume 19 of the LN, and though I love the series, I’d absolutely prefer if Rudy’s libido were slightly less of a focal point.