Controlling A Robotic Arm By Gestures Using Kinect Sensor & Arduino

B.Avinash and J.Karthikeyan had developed a robotic arm that mimic their moves using a Kinect sensor with MATLAB Simulink and an Arduino. The arm was built based on servo motors that replicate the right arm shoulder, elbow and hand movements.

The Kinect sensor is a horizontal bar of motion sensing input devices which enable users to control and interact with their computers through a natural user interface using gestures and spoken commands.

The sensor consists of a RGB camera, depth sensor, and multi-array microphone running proprietary software. It provides full-body 3D motion capture, facial recognition, and voice recognition capabilities.

MATLAB Simulink is a graphical programming environment for modeling, simulating and analyzing multidomain dynamic systems. It supports simulation, automatic code generation, and continuous test and verification of embedded systems.

Simulink is developed by Mathworks, and it offers integration with MATLAB environment, enabling developers to incorporate MATLAB algorithms into models and export simulation results for further analysis. Simulink is widely used in automatic control and digital signal processing for multidomain simulation and Model-Based Design.

To build a similar gesture-controlled arm you need these components:

• Arduino / Genuino Mega 2560
• Microsoft Kinect for Xbox 360
• Microsoft Xbox Kinect Adapter
• 4 Servo Motor
• USB to Serial Breakout – FT232RL
• Gripper
• Solderless Breadboard Full Size
• Jumper wires
• A computer with these applications installed:
  • Arduino IDE
  • Microsoft Kinect for Windows SDK v1.6
  • Microsoft Windows SDK 7.1
  • Mathworks MATLAB

Thanks to Simulink support for Kinect, the computer collects data from the connected kinect device and translates them into servo angles in MATLAB. These angles are sent to the servos through the arduino via TTL device, resulting movement of the arm with a slight delay.

This project has been chosen in the week’s (29/10/2016) Pick of the Week during Matlab Simulink Hardware Challenge 2016, and it also had won the 4th place in “MATLAB International Simulink Hardware Challenge 2016“.

Arduino code, other files and resources are reachable at this instructable and this hackster.io page.

Please follow and like us: