The only drawback of smaller rims, and it is a small drawback if it is a problem at all, is that there is more "squish" available from weight on the tire, such as bumps, landing after a small jump, etc.
Larger wheels ONLY benefit brake rotor diameter, and are a general drawback in all other senses, including worse ride, more weight, and more cost. The more sidewall, the more traction both laterally and longitudinally. Dragsters with unlimited budgets are not running 24" wheels. NASCAR only switched to 18" wheels from 15's due to "keeping up with a more modern look for the cars."
I think people could carve corners, accelerate, and brake more efficiently with 18" or 17" wheels. The Dodge Demon somehow hasn't killed anyone by having 18" wheels and the attendant smaller-diameter brakes.