1. Introduction
Human pose estimation (HPE) is a fundamental task in computer vision that involves identifying the positions of human body joints from images or videos. This process is particularly valuable for action recognition, where sequences of human movements are analyzed to determine the specific actions being performed. Essentially, action recognition can be considered a natural extension of pose estimation, where the dynamics of body movement are analyzed to infer human behavior.
The complexity of HPE is influenced by several factors. For example, a key distinction is made between single-person and multi-person pose estimation, which depends on the number of individuals present in a frame. Additionally, the choice of human body model—such as shape-based, cone-based, or mesh-based representations—can impact the estimation’s accuracy. Another critical factor is the method used for feature extraction, as it directly affects the precision of pose estimation.
Deep learning has significantly advanced the field of HPE, leading to numerous approaches for enhancing accuracy. These include holistic (top-down) methods that regress the positions of all joints simultaneously, part-based (bottom-up) methods that first detect individual body parts and then assemble them, and hybrid strategies that combine both global and local information. Furthermore, modern research explores motion features, pose estimation in video sequences, and challenges such as occlusion and foreshortening.
This paper reviews recent advancements in pose estimation and examines how skeletal data derived from images can be applied to action recognition. We begin by introducing widely-used datasets for evaluating pose estimation and action recognition techniques. Following this, we propose and assess a novel human pose estimation model that uses a ResNet50 backbone and custom convolutional neural network layers to predict 17 keypoints. We then discuss recent studies that leverage pose estimation for action recognition and highlight efforts to simultaneously estimate poses and recognize actions. Finally, the paper concludes by identifying ongoing challenges in the field and presenting a comparative evaluation of our proposed model against established, state-of-the-art solutions.
2. Literature Review
Human Pose Estimation (HPE) is a foundational area of computer vision focused on detecting and localizing keypoints that correspond to human body joints in images or videos. The field has broad applications, ranging from action recognition and motion analysis to augmented reality and healthcare monitoring. The complexity of this task arises from factors such as variations in body shape, multiple overlapping individuals, diverse backgrounds, and partial occlusions. Despite these challenges, advancements in deep learning have enabled HPE systems to achieve remarkable accuracy and reliability.
2.1. Evolution of Human Pose Estimation
Historically, early HPE methods relied on classical computer vision techniques, such as template matching and graphical models, which often struggled with complex backgrounds, occlusions, and variability in human poses. However, the advent of deep learning marked a paradigm shift in the field, as Convolutional Neural Networks (CNNs) became instrumental in learning hierarchical feature representations that significantly improved pose estimation accuracy.
Modern HPE techniques can be broadly categorized into two main types:
-
Single-Person Pose Estimation focuses on detecting keypoints for one individual in an image, typically assuming that the subject is centered and unobstructed.
-
Multi-Person Pose Estimation extends this approach to handle multiple individuals in a scene, requiring algorithms to detect, separate, and associate keypoints with the correct individuals.
2.2. Existing Technologies in Human Pose Estimation
Several state-of-the-art frameworks have emerged for human pose estimation, each offering distinct strengths and trade-offs:
-
MediaPipe: Developed by Google, MediaPipe is renowned for its real-time performance, utilizing a lightweight framework suitable for low-latency applications on edge devices. While its modular design simplifies integration into broader systems, its accuracy can degrade in scenarios involving significant occlusion or complex poses.
-
MoveNet: MoveNet offers a strong balance between speed and accuracy. It is designed to perform reliably across a wide range of scenarios, from simple to moderately complex poses. MoveNet is ideal for applications where computational efficiency and sufficient accuracy are prerequisites, though it may struggle with extreme or highly dynamic pose variations.
-
OpenPose: OpenPose excels at multi-person pose estimation and provides highly detailed joint localization. It employs a Part Affinity Field (PAF)-based approach to associate detected keypoints, which ensures accurate pose estimation even in crowded scenes. However, its significant computational requirements limit its applicability in real-time scenarios, particularly on resource-constrained devices.
2.3. Methodologies in Pose Estimation
This paper focuses on 2D pose estimation, which determines the positions of joints in the X and Y axes relative to a given image or video frame. This method provides a foundational understanding of human position in two-dimensional space and serves as a building block for higher-level tasks such as action recognition.
When estimating the pose of a single person in an image, two primary approaches can be employed: regression-based methods and detection-based methods.
2.3.1. Regression-Based Methods
Regression-based approaches aim to directly predict the coordinates of body joints from the input image using an end-to-end framework. The model learns a mapping from image features to joint positions, such as shoulders, elbows, or knees, to produce an immediate pose estimate.
-
Direct Regression: This method maps image features directly to joint coordinates. For a model predicting 17 keypoints, the output is typically a 17×2 matrix (or a flattened 34-element vector), representing the (x, y) coordinates of each joint.
-
Heatmap Regression: Widely used in 2D pose estimation, this method generates heatmaps for each joint, where pixel intensities represent the likelihood of a joint appearing at a given location. Heatmap regression is particularly effective for localizing keypoints on smaller body parts such as hands, face, and torso, and is more robust to noise and occlusion than direct regression.
2.3.2. Detection-Based Methods
Detection-based approaches also rely on heatmaps but treat pose estimation as a keypoint detection task. Each joint is represented by a separate heatmap generated using a 2D Gaussian distribution centered at the expected location. The model independently identifies each keypoint from these heatmaps and then aggregates them to construct the full pose. This method provides fine-grained spatial localization and is especially beneficial in complex or cluttered scenes.
3. Problem Statement
Despite significant advancements in Human Pose Estimation (HPE), achieving consistent accuracy and robustness across diverse real-world conditions remains a persistent challenge. Models frequently struggle in scenarios involving occlusions, complex poses, varying lighting, and multi-person interactions. These limitations hinder the deployment of HPE in practical applications such as healthcare monitoring, augmented reality, and human–robot interaction. A critical issue lies in the trade-off between performance and computational efficiency: real-time models tend to sacrifice accuracy, while high-precision models are often computationally expensive and unsuitable for edge deployment.
While frameworks like MediaPipe, MoveNet, and OpenPose represent the state-of-the-art in pose estimation, each exhibits distinct strengths and limitations. MediaPipe offers real-time performance on edge devices but performs poorly in complex or cluttered scenes. MoveNet strikes a balance between speed and accuracy but can underperform in edge cases, such as overlapping subjects. OpenPose delivers detailed multi-person pose estimation, yet its high computational demands restrict its use in real-time or resource-limited environments. Despite these differences, the lack of standardized comparative evaluations across these frameworks makes it difficult for researchers and practitioners to make informed decisions about which model best suits a given use case.
To address this gap, a comprehensive, comparative study is needed that evaluates these HPE models under a unified framework. This paper proposes such a study, introducing a novel deep learning–based methodology and benchmarking its performance against existing frameworks. The analysis focuses on key performance metrics such as keypoint detection accuracy and computational efficiency. This work aims to provide actionable insights into the trade-offs between current approaches, thereby guiding future model selection and development in the HPE domain.
4. Dataset Used
This study utilizes the Frames Labeled in Cinema (FLIC) dataset, a standard benchmark for evaluating human pose estimation algorithms. The dataset consists of 5,000 annotated frames extracted from various Hollywood films, each labeled with 2D keypoints corresponding to essential human body joints—including the head, shoulders, elbows, wrists, hips, knees, and ankles. These annotations provide a reliable foundation for supervised learning and evaluation tasks. The dataset includes a wide variety of body poses, clothing styles, background environments, and partial occlusions, offering the kind of real-world variability that is critical for training and testing pose estimation models.
The diversity of FLIC’s image collection ensures that the model is challenged to generalize effectively, enhancing its robustness and accuracy in unconstrained conditions. Furthermore, the dataset’s detailed ground-truth annotations are leveraged for both model training and quantitative evaluation. These annotations enable precise benchmarking of the model’s ability to detect and localize human joints, ensuring the results adhere to high standards of accuracy and reliability. While FLIC provides annotations for several joints, this study focuses on a subset of 17 keypoints aligned with the architectural constraints of the models under comparison.