Running with ROS

First of all you need to download the repo

git clone https://www.github.com/Tobic0/dynamic-hand-gesture-recognition

In order to run the code with ROS you need to uncomment some lines in the main.py code. In particular:

• line 36 -> import rospy
• line 37 -> from std_msgs.msg import String
• line 58 -> rospy.init_node('hand_gesture_recognizer')
• line 59 -> pub = rospy.Publisher('/hand_gesture', String, queue_size=10)
• line 138 -> pub.publish(actions[np.argmax(res)])

At this point in the dynamic-hand-gesture-recognition folder you need to perform the following commands:

mkdir src
source /opt/ros/noetic/setup.bash
catkin build
source devel/setup.bash

After that in one terminal window you need to start roscore, while in a second window you can run the main.py code, but only after you called the source devel/setup.bash first, by using:

python3 main.py

If you want to able to read from ros topic, you will need to open a new terminal window and type:

rostopic echo /hand_gesture

Landmark extraction process with MediaPipe

Raw data output generated by MediaPipe's hand landmark model

0	1	2	....	18	19	20
[0.6, 0.6, 0.0]	[0.68, 0.56, 0.009]	[0.74, 0.49, 0.007]	....	[0.53, 0.38, 0.01]	[0.54, 0.41, 0.02]	[0.54, 0.44, 0.02]

The x and y values are then multiplied, respectively, by the frame width and frame height in order to get pixel coordinates for each keypoint.

0	1	2	....	18	19	20
[386.9, 292, 0.0]	[438, 272, 0.009]	[473.7, 239, 0.007]	....	[342.7, 183.3, 0.01]	[347, 198.9, 0.02]	[350, 211.8, 0.02]

After that the landmark coordinates are transformed to relative position with respect to the wrist keypoint (keypoint 0), so that the hand landmarks are not relative to the current position of the hand in the captured frame.

0	1	2	....	18	19	20
[0.0, 0.0, 0.0]	[51.1, -20, 0.009]	[86.8, -53, 0.007]	....	[-44.2, -108.7, 0.01]	[-39.8, -93.1, 0.02]	[-36, -80.2, 0.02]

Then convert the list into a one-dimensional list.

One dimensional list
0.0, 0.0, 0.0, 51.1, -20, 0.009, 86.8, -53, 0.007, .... , -44.2, -108.7, 0.01, -39.8, -93.1, 0.02, -36, -80.2, 0.02

Get max & min values of landmarks list in order to apply min-max normalization method.

Normalzied one dimensional list
0.0, 0.0, 0.0, 0.36, -0.1, 0.0, 0.61, -0.3, 0.0, .... , -0.3, -0.7, 0.0, -0.2, -0.6, 0.0001, -0.2, -0.5, 0.0001