Dr. Arnab Dey

Senior Researcher Scientist at Stealth Startup

🤖 Human Generation and Pose Estimation

Published:

🤖 Human Generation and Pose Estimation

An AI-powered toolkit that combines advanced image inpainting with human pose estimation capabilities. This project uses Stable Diffusion XL for generating realistic humans in images and MediaPipe for extracting detailed pose keypoints.

🚀 Key Features

Demo: Hugging Face Spaces

  • 🎨 AI Human Generation: Generate realistic humans in any image using Stable Diffusion XL inpainting
  • 🤸 Pose Estimation: Extract and visualize human pose keypoints using MediaPipe
  • 🖥️ Interactive Web Interface: User-friendly Gradio interface for easy experimentation
  • ⚡ GPU Acceleration: CUDA support for fast inference
  • 🐳 Docker Support: Containerized deployment with NVIDIA GPU support
  • ⚙️ Configurable: YAML-based configuration system for easy customization

🛠️ Technical Stack

AI/ML Frameworks:

  • PyTorch with CUDA support
  • Diffusers (Hugging Face) for Stable Diffusion XL
  • MediaPipe for pose estimation
  • Transformers for model handling

Web Interface:

  • Gradio for interactive web UI
  • PIL (Pillow) for image processing
  • NumPy for numerical computations

DevOps & Configuration:

  • Docker with NVIDIA GPU support
  • YACS for configuration management
  • YAML configuration files

🎯 Core Functionality

🎨 Inpainting Pipeline

from generate_human import Inpaint

# Initialize with configuration
inpaint = Inpaint(cfg)

# Generate human in image
result = inpaint.inpaint_image(
    input_image="path/to/image.jpg",
    prompt="A realistic human standing",
    bbox=[x1, y1, x2, y2],
    negative_prompt="multiple people"
)

🤸 Pose Estimation Pipeline

from generate_pose import HumanPose

# Initialize with configuration
pose = HumanPose(cfg)

# Extract keypoints
keypoints = pose.extract_keypoints(image)

# Visualize keypoints
visualized = pose.visualize_keypoints(image, keypoints)

🌟 Interactive Features

Inpainting Tab

  • Upload input images
  • Enter descriptive text prompts
  • Specify precise bounding box coordinates
  • Add negative prompts for quality control
  • Generate and save results

Pose Estimation Tab

  • Upload images containing humans
  • Automatic pose keypoint extraction
  • Color-coded visibility visualization:
    • 🔴 Red: Not visible (confidence < 0.3)
    • 🟠 Orange: Occluded (confidence 0.3-0.5)
    • 🟢 Green: Visible (confidence > 0.5)
  • Export keypoints data and visualizations

🐳 Deployment Options

Local Installation

# Clone and setup
git clone https://github.com/arnabdeypolimi/Human_inpainting_pose_estimation
cd hax
python -m venv venv
source venv/bin/activate

# Install dependencies
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install diffusers transformers mediapipe yacs gradio pillow

Docker Deployment

# Build and run with GPU support
docker build -t human-generation .
docker run --gpus all -p 7860:7860 human-generation

⚙️ Configuration System

YAML-based configuration for easy customization:

# Generation settings
prompt: 'A highly detailed, realistic human...'
bounding_box: [200, 200, 3000, 3000]
negative_prompt: "multiple people, group, crowd"

# Model settings
diffusers:
  model_name: 'diffusers/stable-diffusion-xl-1.0-inpainting-0.1'
  guidance_scale: 7.5
  num_inference_steps: 50

pose:
  model_complexity: 2

🧪 Quality Assurance

  • Comprehensive Test Suite: Unit tests covering all major functionality
  • Model Validation: Automated testing of model initialization and inference
  • Performance Benchmarks: Memory and speed optimization tests
  • Error Handling: Robust error management and user feedback

⚡ Performance Optimizations

  • GPU Acceleration: CUDA-optimized inference pipelines
  • Model Caching: Automatic model caching for faster subsequent runs
  • Memory Management: Efficient VRAM usage for large models
  • Configurable Quality: Adjustable inference steps for speed/quality trade-offs

📊 System Requirements

Minimum Requirements:

  • Python 3.8+
  • 8GB+ RAM
  • NVIDIA GPU with 4GB+ VRAM
  • CUDA 11.8+

Recommended:

  • 16GB+ RAM
  • NVIDIA GPU with 8GB+ VRAM
  • SSD storage for model caching

🌐 Links

This project demonstrates advanced expertise in computer vision, deep learning, and AI model deployment. It showcases the integration of state-of-the-art models (Stable Diffusion XL, MediaPipe) with modern MLOps practices including containerization, configuration management, and interactive web interfaces.