How to Get all the Co-ordinates of Hand using Mediapipe Hand Solutions ?

Image from https://mediapipe.dev/

Its been a while since Google Released their Mediapipe Solutions. They are Absolutely amazing. The Best Part about their Models is that they are extremely light-weight and Hence can even run on ARM Devices like Raspberry Pis and Even Mobile Phones.

But the Problems with their Solution is that even they are well documented, there is not much code available and Most of their work happens internally. So even After running the model on your Local Computer or on any devices like Raspberrypi or Mobile Phones, you can do much Customization. The problem becomes worst when you cannot access the coordinates of your Hand. I invested my Hours just to figure out how to get these Coordinates and Finally I figured that out By Actually Going through their Source Code with at the drawing_utils file.

So Today, we are going to install the Mediapipe Solutions and Learn how to Get all the Coordinates of your Hands using Mediapipe Hand Solutions API.

Lets Get Started ..

1. Installation

You can easily install the Mediapipe Solutions using pip install mediapipe and you should be good to go.

2. Runing the Provided Code for Hands Tracking.

Now lets run the code provided by the official Documentation of Mediapipe for Tracking Hand in realtime. You can read more about it here.

import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands


# For webcam input:
cap = cv2.VideoCapture(0)
with mp_hands.Hands(
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5) as hands:
  while cap.isOpened():
    success, image = cap.read()
    if not success:
      print("Ignoring empty camera frame.")
      # If loading a video, use 'break' instead of 'continue'.
      continue

    # Flip the image horizontally for a later selfie-view display, and convert
    # the BGR image to RGB.
    image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
    # To improve performance, optionally mark the image as not writeable to
    # pass by reference.
    image.flags.writeable = False
    results = hands.process(image)

    # Draw the hand annotations on the image.
    image.flags.writeable = True
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    if results.multi_hand_landmarks:
      for hand_landmarks in results.multi_hand_landmarks:
        mp_drawing.draw_landmarks(
            image, hand_landmarks, mp_hands.HAND_CONNECTIONS)
    cv2.imshow('MediaPipe Hands', image)
    if cv2.waitKey(5) & 0xFF == 27:
      break
cap.release()

Above is the Code Provided by mediapipe’s official Documentation for Tracking Hands in Realtime using your Webcam.

3. Getting All the Coordinates of Hand.

Now comes the Most interesting part about this Article. How to get all the 21 coordinates of your Hand and How to Use it for your own Purpose.

Heres is how to Get all the Coordinates of all your hand.

import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands


# For webcam input:
cap = cv2.VideoCapture(0)
with mp_hands.Hands(
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5) as hands:
  while cap.isOpened():
    success, image = cap.read()
    if not success:
      print("Ignoring empty camera frame.")
      # If loading a video, use 'break' instead of 'continue'.
      continue

    # Flip the image horizontally for a later selfie-view display, and convert
    # the BGR image to RGB.
    image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
    # To improve performance, optionally mark the image as not writeable to
    # pass by reference.
    image.flags.writeable = False
    results = hands.process(image)
    image_height, image_width, _ = image.shape
    # Draw the hand annotations on the image.
    image.flags.writeable = True
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    if results.multi_hand_landmarks:
      for hand_landmarks in results.multi_hand_landmarks:
        # Here is How to Get All the Coordinates
        for ids, landmrk in enumerate(hand_landmarks.landmark):
            # print(ids, landmrk)
            cx, cy = landmrk.x * image_width, landmrk.y*image_height
            print(cx, cy)
            # print (ids, cx, cy)
        mp_drawing.draw_landmarks(
            image, hand_landmarks, mp_hands.HAND_CONNECTIONS)
    cv2.imshow('MediaPipe Hands', image)
    if cv2.waitKey(5) & 0xFF == 27:
      break
cap.release()

First we have edited this line after image image_height, image_width, _ = image.shape to get the width and height of the image. The reason to get this is because mediapipe gives the coordinates in range of 0 to 1 So we manually need to convert them in pixels so that we can perform operations on that.

Later down after the loop of handlandmarks we have created a Loop that enumerates the handlandmarks.landmark which is actually all the landmarks of hands.

Further you can get each of the 21 landmarks by comparing the ids with the handlandmark chart provided by mediapipe here.

Image from https://google.github.io/mediapipe/solutions/hands#hand-landmark-model

If you encounter any difficulties Feel Free to Connect with me Abdul Rehman Kalsekar or on Twitter @ar_kalsekar or https://arkalsekar.in😊