What’s going to tell if the client has been modified? The client, which has been modified…?
The server. Like how some games handle the hit detection client side (counter-strike) and others do it server side (battlefield). If everything was handled server side, the server should be able to detect modified clients and not permit them to even connect. This is basically how Blue Sentinel for Dark Souls 3 works. It’s a 3rd party anti-cheat that can detect modified clients and block them from connecting to you or vice versa if the client data doesn’t match up. This way you can only connect to vanilla clients if you’re playing vanilla or only to clients also running the mod you’re running.
And who tells the server that the client hasn’t been modified…?
But then you started to being in external solutions, which of course themselves could be modified, and you’re starting to answer your own question about why it’s pretty hard.
And who tells the server that the client hasn’t been modified
The server itself checks against what is allowed and what isn’t. It knows what the clients should be doing, and if they do something else it’s flagged. External hacks still, afaik, hook into the client and change the code as it is executed but still before it’s sent to the server, so you could still be checking against what the client is actually doing.
The external solution I mentioned in Blue Sentinel only exists because such a thing was not built into the game itself by FromSoftware, but there is no reason why it couldn’t be.