Swimmers look tired after the competition but they don't look like they are covered in sweat like wrestlers.
Add in many think "I could do that" with a bit of practice in swimming.(basketball, etc - and even boxing) but not in wrestling. The thought that 'I would get my head torn off and my shoulders dislocated" is there. It isn't in many other sports.
Then you have the rules. Not all that easy to understand for those with no experience or exposure to the sport.