Tag - Artificial intelligence

Scenario

The scenario enables interaction between an operator and a Robotic arm NED2​, using interfaces based on artificial intelligence. In particular, the operator can use gestures to designate the part to be picked up by the Robotic arm NED2, one gesture corresponding to a part from a collection arranged in the alpha zone. The Robotic arm NED2 places the gripped part in the operator’s hand, after identifying its position using the vision set camera, above the loading zone. This is a pick-and-place sequence, where the pick and place points are provided in real time by the operator’s gesture commands. Gesture and hand position recognition is performed by deep learning tools.

The objective of the scenario is to perform this operation by following the steps illustrated in the following algorithm:

Lab Contents

Chapter 1: Pick and Place

• Define points of interest
• Create movements for pick and place

Chapter 2: Gesture recognition

• Know how to use Teachable Machine to train a model
• Obtain predictions based on gestures
• Create a filter to validate a gesture

Chapter 3: Hand detection

• Detect the hand in the camera image
• Calibrate the camera
• Obtain the coordinates of the drop point in the middle of the hand

Chapter 4: Integration

• Integrate the subprograms into a complex and functional program

 

Prerequisite knowledge

Python: Basic syntax + simple data and control structures + loops + simple function calls

Reference frames and transformation: Understanding how Cartesian coordinate systems work and the principle of transformations

Required Equipment