In Microsoft Unleashes the Kinect and Promptly Misses the Point, Tim Maly makes the point that "A key part of human manipulation of the world is the manual aspect." The more we abstract controls away from physical, tactile experiences the less precise they become. They are less "controls" than "suggestions", so to speak. This is a characteristic of virtual keyboards, gestural interfaces, touch screens where every touch is identical to every other touch, and systems like Kinect where there's no touching at all. They are in many ways the opposite of ergonomic. 

One of the threads in the history of computing is increasing abstraction. It's not just in the area of human interface, of course, and in many ways abstraction is incredibly powerful. In interaction, though, it's not an unalloyed good, and it can introduce imprecision — vagueness — into use cases.

This is just conjecture, but Maly mentions playing a musical instrument as an example where the physicality of the experience is central to the whole thing. It's not easy; it needs learning and takes a long time to master. It's the physical nature of it, though, that helps make it learnable. Introduce the vagueness of waving your hands in the air and it's less learnable. 

When something is learnable it's more likely to be satisfying, long term.