The physical requirements go beyond a basketball throw, a shuttle run, and some pull ups. There are also medical requirements, not to mention the congressional nomination.
Then there's the whole SAT/ACT thing, which some Ivies no longer require. (I also know a NQ for an Ivy, a current wrestler, whom was not going to able to get into West Point)