تطوير نظام إنترنت الأشياء للمرضى المصابين بالشلل باستخدام تقنيات تقييم حركة الحاجب والصوت

Developing An Internet of Things System for Paralyzed Patients Using Eyebrow Movement and Voice Assessment Technologies

Abstract

This thesis presents an advanced IoT-based system designed to control smart home devices, specifically developed to meet the needs of people with paralysis or disabilities. It addresses the critical role of IoT in facilitating independent living for individuals with physical limitations by enabling smart home applications that respond to various user inputs. Two types of paralyzed users are considered: those who are unable to speak, using eyebrow movements to control the smart home, and those who can speak, utilizing voice commands converted to text to control devices or send messages.

The system incorporates the YOLO model, which has been specially trained to detect landmarks of the face with high accuracy by processing 9,333 images from two image datasets, the Wider Face and Celebrity Face datasets. After applying augmentation and data cleaning it was turned into 62,938 images. Using the MediaPipe library, 102 facial landmarks were extracted, covering regions such as the eyebrows, eyes, upper and lower lips, and other facial landmarks. Each dataset image was labeled and used to train the YOLOv8n-pose model.

Key features of the system include the ability to control household appliances—such as lights, televisions, fans, and air conditioners—through simple eyebrow movements. For users who can speak, the Whisper library was employed to convert voice to text, enabling device control via voice commands. The Gemini API model serves two functions: detecting whether a user is awake or asleep and engaging in conversation with a custom AI assistant. The former assists speaking users, while the latter is designed for non-speaking users.

The model demonstrates exceptional performance metrics, achieving a facial detection accuracy of 98%, a pose estimation for 102 landmarks accuracy of 94%, and a voice command accuracy of 95%.