This is true of mot Olympic sports. Even sports that are popular during Olympics years like swimming and diving are mostly irrelevant during non-Olympic years.
I don't know if there are pockets somewhere in the world where sports like Canoeing, Fencing, Shooting, Sailing, Water Polo etc. have large following and the athletes are celebrities, but looking at the list of Olympic sports I would guess the majority of them are niche sports similar or even less popular than wreslting.