Gesture recognition is technology that uses sensors to read and interpret hand movements as commands. In the automotive industry, this capability allows drivers and passengers to interact with the vehicle — usually to control the infotainment system without touching any buttons or screens.
Aptiv developed the technology behind the first gesture recognition system for automotive applications, introduced in the BMW 7 Series in 2015, which can recognize hand gestures that control music/audio and incoming calls.
As the technology matures, gesture recognition will move beyond infotainment and will allow drivers to control other systems within the vehicle, such as heating and cooling, and to connect with smart home systems. For example, imagine being able to check your home security camera as you drive home by simply making a hand gesture. Gestures could also be coupled with telematics systems, allowing the vehicle to provide information about nearby landmarks if it recognizes that an occupant is pointing at it.
How it works
A gesture recognition system starts with a camera pointed at a specific three-dimensional zone within the vehicle, capturing frame-by-frame images of hand positions and motions. This camera is typically mounted in the roof module or other vantage point that is unlikely to be obstructed. The system illuminates the area with infrared LEDs or lasers for a clear image even when there is not much natural light.
Those images are analyzed in real time by computer vision and machine learning technologies, which translate the hand motions into commands, based on a predetermined library of signs.
Commands generated by the gesture recognition software become just another type of input, similar to turning a dial, pressing a button or touching a screen. Additionally, as the quantity and quality of cabin cameras improves, other passengers in the vehicle could eventually get in on the act.
What are the most common gestures?
Each OEM might develop its own variations, but here are some of the basic hand motions:
- A single finger spun clockwise to turn the radio volume up or zoom in on a map
- A single finger spun counterclockwise for the opposite action
- A pointing gesture, with either one or two fingers, to accept a call
- A swipe gesture to reject a call
- In the BMW, a gesture of two fingers, without motion, controls a variety of other functions, such as muting or unmuting the audio
A pinching motion between thumb and forefinger could also manipulate an image on the vehicle display. This would allow a driver to, for example, get a visual image of the vehicle in a tight parking spot from all angles.
The bigger picture
Benefits of gesture recognition include improved safety — since drivers do not have to take their attention off the road as much as they would with touch controls — and the simple convenience of being able to control vehicle functions with deliberate gestures rather than a potentially complex menu scheme.
Gesture recognition is an exciting technology, but it is only one aspect of a broader camera-based interior sensing platform that can interpret the voice commands, eye movements and body movements of the driver, while giving the vehicle the capability to recognize the driver, adjust settings to the driver’s preferences, monitor the driver’s attentiveness and mood, take over braking functions in an emergency, and switch back and forth between autonomous and human driving modes.
