Gesture Detection & Machine Learning

Using Device Camera for Advanced Gaming UI/IX Development

Sistem lengkap untuk mendeteksi 64 macam gestur tubuh menggunakan kamera perangkat dan MediaPipe Pose

Camera not started

CPU: 0%

Memory: 0MB

FPS: 0

MQTT: Disconnected

Live Camera Feed

Not connected to broker.hivemq.com

MQTT messages will appear here...

Gesture Recognition

Waiting for detection...

Hex Command Output

// Hex commands will appear here

MQTT Command Output

// MQTT commands will appear here

Workshop Development

System Overview

Hardware Setup

Device Integration

Server Program

64 Gestures & Hex Codes

MQTT Integration

Performance Analysis

Troubleshooting

System Architecture

Arsitektur Sistem Terintegrasi:

[Device Camera] → [MediaPipe Pose] → [JavaScript Processing] → [Web Interface] → [User]
       ↓                      ↓              ↓              ↓
   Capture Image        Pose Detection  Gesture Analysis  Visualization
                        Keypoint Extraction ML Inference    Control
                                                            ↓
                                                      [MQTT Broker]
                                                            ↓
                                                    [External Devices]
                        

Komponen Utama:

Device Camera: Kamera depan/belakang perangkat untuk capture image
MediaPipe Pose: Library Google untuk pose estimation dan landmark detection
JavaScript Processing: Frontend processing dengan TensorFlow.js dan custom logic
Web Interface: Dashboard real-time untuk monitoring dan kontrol
Gesture Recognition: Custom algorithm untuk recognition 64 gestur
MQTT Integration: Pengiriman command ke broker MQTT untuk kontrol eksternal

Real-time Processing Pipeline

Image Acquisition: Device camera capture frame (640x480 @ 30fps)
Pose Estimation: MediaPipe extract 33 keypoints tubuh
Gesture Classification: Custom algorithm process pose data
Hex Command Generation: Mapping gestur ke kode operasional mesin
MQTT Transmission: Mengirim command ke broker MQTT
Real-time Feedback: Visual update ke dashboard

Device Requirements

Minimum Requirements

Camera: Front or rear camera

Browser: Chrome, Firefox, Edge, Safari

RAM: 2GB+

Processor: Dual-core 1.5GHz+

Recommended Requirements

Camera: 720p+ resolution

Browser: Latest Chrome

RAM: 4GB+

Processor: Quad-core 2.0GHz+

Setup Instructions

Camera Permission:

Click "Start Camera" button
Allow camera access when prompted
Position yourself in frame
System will automatically detect poses
                        

MQTT Connection:

Click "Connect MQTT" button
System connects to broker.hivemq.com
Detected gestures are sent to topic "PerintahGestureTubuh"
Format: "$PerintahGesture,nama_gesture,code_gesture"
                        

Device Integration

Camera Access Implementation:

async function startCamera() {
    try {
        const stream = await navigator.mediaDevices.getUserMedia({
            video: {
                width: { ideal: 640 },
                height: { ideal: 480 },
                facingMode: 'user'
            }
        });
        
        videoElement.srcObject = stream;
        return true;
    } catch (error) {
        console.error('Error accessing camera:', error);
        return false;
    }
}
                        

MediaPipe Pose Integration:

// Import MediaPipe Pose
const pose = new Pose({
    locateFile: (file) => {
        return `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`;
    }
});

pose.setOptions({
    modelComplexity: 1,
    smoothLandmarks: true,
    minDetectionConfidence: 0.5,
    minTrackingConfidence: 0.5
});

pose.onResults(onPoseResults);
                        

Gesture Processing Logic

Gesture Recognition Algorithm:

function analyzePose(landmarks) {
    // Extract key angles and positions
    const leftArmAngle = calculateArmAngle(landmarks[11], landmarks[13], landmarks[15]);
    const rightArmAngle = calculateArmAngle(landmarks[12], landmarks[14], landmarks[16]);
    
    // Compare with gesture database
    for (let gesture of GESTURE_DATA) {
        const confidence = calculateGestureConfidence(gesture, landmarks);
        if (confidence > 0.7) {
            return { gesture, confidence };
        }
    }
    
    return null;
}

function calculateArmAngle(shoulder, elbow, wrist) {
    // Calculate angle between three points
    const v1 = { x: elbow.x - shoulder.x, y: elbow.y - shoulder.y };
    const v2 = { x: wrist.x - elbow.x, y: wrist.y - elbow.y };
    
    const dotProduct = v1.x * v2.x + v1.y * v2.y;
    const magnitude1 = Math.sqrt(v1.x * v1.x + v1.y * v1.y);
    const magnitude2 = Math.sqrt(v2.x * v2.x + v2.y * v2.y);
    
    return Math.acos(dotProduct / (magnitude1 * magnitude2)) * (180 / Math.PI);
}
                        

64 Gestures & Hex Command Mapping

Complete Gesture to Hex Mapping Table:

ID	Gesture Name	Hex Code	Category	Machine Command

MQTT Integration

MQTT Configuration:

// MQTT Broker Configuration
const MQTT_CONFIG = {
    host: "broker.hivemq.com",
    port: 8000,  // WebSocket port
    clientId: "gesture_detector_" + Math.random().toString(16),
    topic: "PerintahGestureTubuh"
};

// Message Format
const MQTT_MESSAGE_FORMAT = "$PerintahGesture,{gesture_name},{hex_code}";

// Example: "$PerintahGesture,Arms Up,0x02"
                        

MQTT Implementation:

function connectMQTT() {
    client = new Paho.MQTT.Client(
        MQTT_CONFIG.host, 
        MQTT_CONFIG.port, 
        MQTT_CONFIG.clientId
    );
    
    client.onConnectionLost = onConnectionLost;
    client.onMessageArrived = onMessageArrived;
    
    client.connect({
        onSuccess: onConnect,
        onFailure: onFailure,
        useSSL: true
    });
}

function sendGestureCommand(gesture, confidence) {
    if (client && client.isConnected()) {
        const message = new Paho.MQTT.Message(
            `$PerintahGesture,${gesture.name},${gesture.hex}`
        );
        message.destinationName = MQTT_CONFIG.topic;
        client.send(message);
        
        addMqttMessage(`Sent: $PerintahGesture,${gesture.name},${gesture.hex}`);
    }
}
                        

Performance Analysis & Metrics

Real-time Performance Monitoring:

Current Performance Metrics

Total Gestures Detected: 0

Average Confidence: 0%

Average Processing Time: 0ms

Most Frequent Gesture: -

Troubleshooting Guide

Problem	Possible Cause	Solution
Camera tidak bisa diakses	Izin kamera tidak diberikan	Periksa pengaturan browser, berikan izin akses kamera
Pose tidak terdeteksi	Pencahayaan kurang optimal	Tambah lighting, hindari backlight
MQTT tidak terhubung	Koneksi internet bermasalah	Periksa koneksi internet, firewall, atau proxy
Gesture tidak terdeteksi	Posisi tubuh tidak jelas	Pastikan seluruh tubuh terlihat dalam frame
Confidence score rendah	Gerakan tidak sesuai dengan database	Lakukan gerakan dengan lebih jelas dan tepat
Processing time lambat	Spesifikasi perangkat rendah	Tutup aplikasi lain, gunakan browser terbaru