Winning worlds is actually more difficult than winning the Olympics (at least at Olympic weight classes). The Olympics is thought of as having more prestige simply because it has a much wider audience.
Aside from more medals, what JB has on Smith is that he won a number of golds in era of only six weight classes (Smith had ten) and after the fall of the USSR.
Of course, winning six straight has an undeniable appeal.
At the end of the day, GOAT talk is fun but doesn't mean much in reality. Wrestling a chain reaction, a constantly evolving endeavor. If someone inspires you by their work ethic and by their grit and by their performances, that's really all the sport can offer fans.