Gesture Recognition and Biometrics ElectroMyogram (GRABMyo)

Abstract

We present the Gesture Recognition and Biometrics ElectroMyogram (GRABMyo) dataset, an open-access dataset of electromyogram (EMG) recordings collected from the wrist and forearm muscles while performing hand gestures. Data were collected from 43 healthy participants (age range: 24-35 years) on three different days while performing 16 hand and finger gestures in identical experimental sessions on each day. The GRABMyo dataset can be used for research on: 1) EMG-based biometrics for personal identification and verification, and 2) EMG-based gesture recognition for neurorehabilitation and home applications. The large sample size will provide sufficient power for establishing results, specifically for applications such as biometrics. Further, it will be useful for subject-independent applications such as generalized classification models for gesture recognition. The multiday recordings will provide results that will be consistent over a longer duration, crucial for the reliability of EMG-based wearable devices.

Background

Recently novel biometric traits based on surface electromyogram (EMG) have been shown to be more resilient to spoofing than the conventional biometric traits such as fingerprints and facial recognition [1]. Additionally, the gesture recognition property of EMG allows individuals to set customized gestures as passwords for enhanced security [2]. Some of the studies have reported a high biometric authentication and identification performance (>95%) [3-9]. While most of the studies had less than 25 subjects, only three studies had a larger number (>40) of subjects [6, 8, 9]. Further, these large sample studies were limited to only one day of data collection. Currently, there exist a few multi-day EMG datasets of which two datasets with large sample-size (>40) involved two days of data collection [10, 11]. Only three studies involved a higher number (>2) of data collection days, however, the number of subjects was less (#N<11) [12, 13]. In a multi-session scenario spread across days, non-stationary factors including electrode shifts, sweat, and dry skin, and physical conditions (rest vs. exercising) will affect the accuracy and consistency of EMG signals. Therefore, the multi-day performance with a large number of subjects is crucial for validating the effectiveness of EMG as a biometric trait.

To address the above limitations in the currently available datasets, we provide an open-access Gesture Recognition and Biometrics ElectroMyogram (GRABMyo) Dataset. The presented dataset is the largest EMG dataset in terms of the total number of recordings (43 subjects x 3 days = 129 recordings). The present data collection uses a 2048 Hz biosignal amplifier. The two other large-sample (>40) multiday datasets were limited to two days of data collection with a sampling frequency of 1000 Hz [11] and 5012 Hz [10]. In addition to biometrics research, the dataset can also provide resources for gesture recognition research such as user-independent prosthetic control and robustness in a multiday scenario. The cross-day data can be used to assess robustness towards factors such as skin conditions, electrode shifts, and physical conditions. The two electrode positions: forearm and wrist will facilitate the research and development of industry-grade wearable wristband and bracelet for biometric authentication and gesture recognition applications

Methods

We recruited 43 healthy participants (23 M, 20 F) for the study. The average age was 26.35 ± 2.89 and the average forearm length (measured from the styloid process on the wrist to the olecranon on the elbow) was 25.15 ± 1.74 cm. The experiments were conducted following the Declaration of Helsinki and the research protocol was approved by the Office of Research Ethics of the University of Waterloo (ORE# 31346).

Prior to the experiment, the participant’s forearm length is measured as the distance between the olecranon process and the ulnar styloid process. The forearm circumference is measured at a distance one-third of the forearm length from the olecranon process. The wrist circumference is measured at 2 cm away from the ulnar styloid process. After taking these measurements, the electrodes are placed on the forearm and wrist. For the forearm electrode placement, sixteen monopolar sEMG electrodes (AM-N00S/E, Ambu, Denmark) were placed in the form of two rings, each consisting of eight electrodes equally spaced around the forearm, forming eight bipolar pairs.

The center-to-center distance between the two rings was maintained at 2 cm. For the wrist electrode setup, twelve monopolar sEMG electrodes of the same type as the forearm rings were placed in the form of two rings, each consisting of six electrodes equally spaced around the wrist and forming six bipolar pairs. The center-to-center distance between the two rings was maintained at 2 cm, similar to the forearm-setup. Therefore, a total of 28 monopolar sEMG electrodes were used for each session. To maintain consistency of the positions of the electrodes across all participants, the first electrode in each ring (total rings = 4) was anatomically positioned on the centerline of the elbow crease.

Following the setup, the participant is seated comfortably on the chair with both their upper limbs in a resting position. The participants performed 16 hand and wrist gestures with the help of visual instruction provided on the computer screen. the resting data was collected after each trial comprising of the 16 gestures (hence, a total of 17 gestures). Each gesture was repeated seven times (17 gestures x 7 trials = 119 contractions) with five seconds of recording in each trial. A ten-second rest period was provided between consecutive contractions to avoid muscle fatigue.

Data Description

For the GrabMyo Dataset, the three top-level folders represent the three sessions. Within each session folder, there are 43 sub-folders for each of the participants. The three folders for the three data collection sessions are: "Session1", "Session2", and "Session3". Each of the folders contains subfolders named "sessioni_subjectj" for each of the participants. Each subject subfolder contains data recording files named "sessioni_subjectj_gesturek_trialt.dat" and "sessioni_subjectj_gesturek_trialt.hea". For the files, i ∈ {1,2,3} for the respective session indexes, j ∈ {1,2,...,43} for the respective subject index, k ∈ {1,2,...,16} for the respective gesture indexes and t ∈ {1,2,...,7} for the respective trial indexes.

There are 32 signals/channels for the 5 secs of EMG recordings sampled at 2048 Hz (10240x32). Of these 32 channels, 16 channels are collected from the forearm and 12 channels are collected from the wrist in a bipolar configuration (8+8 for the forearm and 6+6 for the wrist in the form of 4 rings, as shown in the electrodelocation.pdf file), while 4 channels remain unused.

The forearm channels are listed as {F1-F8} (corresponding to ring1 with column numbers {1,2,..8}) and {F9-F16} (corresponding to ring2 with column numbers {9,10...16}). The wrist channels are numbered {W1-W6} (corresponding to ring3 with column numbers {18,19,...23}) and {W7-W12} (corresponding to ring4 with column numbers {26,27...31}). The unused channels are listed as {U1-U4} (corresponding to column numbers {17,24,25,32}) are provided to distinguish the rings of electrode setup. The sample codes provided along with data have demonstrated the channel selection for both the forearm and wrist configuration (see grabmyo_convert_wfdb_to_mat.py). Additional codes have been provided for feature extraction using the frequency division technique (FDT, see grabmyo_feature_extraction.m) [21].

Usage Notes

The data may be used for the following investigations:

1. Improving biometric authentication. A multicode EMG-based biometric framework was introduced for a secure biometric system and different fusion strategies were discussed [2]. A standard biometric authentication system consists of four modules: 1) sensor module which collects the biometric data, 2) feature extractor for generating feature vectors utilized as biometric entries, 3) matcher module that compares with genuine user’s template to generate a score, and 4) decision module to grant access or rejection based on its score threshold. The multiday performance analysis of multicode EMG-based biometrics needs to be investigated.
2. Biometric identification. Another major biometric application is the identification mode where the system predicts the identity of the presenting user by finding the closest match. As per the definition, the identification is a more error-prone application than the authentication as the system makes N comparisons, where N is the number of users enrolled in the database. Therefore, the factors affecting system such as multiple days and sample size of the database needs to be investigated for real-life applications [7].
3. Subject independent gesture recognition. Extensive research on EMG has been performed on gesture recognition with application in rehabilitation using prosthetic and orthotic devices, home application for assisting daily activities, virtual environment control and sign language recognition [14, 15]. However, with increase in class labels, there is exists a training burden for setting up machine learning models [16]. Recent studies have suggested deep learning techniques for cross-user calibration free which trains generalized models using the population data, and hence reduce the training burden [17, 18]. The presented large-sample dataset can provide adequate resources for such calibration free models.
4. Electrode shift invariant techniques. One of the significant factors affecting the cross-day performance is the shift in the electrode positions. It is impossible to fix the location of armband electrodes on the forearm and wrist for daily-wear use. These variations affect the performance of both the EMG-based biometric and gesture recognition applications. Some techniques such as classification model adaptation [19] and feature space transformation using transfer learning [20] have been suggested to address the electrode shift variations. These techniques could be further investigated to potentially improve biometrics and gesture recognition performance.

Release Notes

Version 1.1.0. The project was updated as follows:

1. Deleted code biometric_fileread.m and rdwfdb.m (conversion errors, please do not use from previous versions. Recommend using Python codes added to the update.
2. Uploaded grabmyo_convert_wfdb_to_mat.py (conversion code, instructions provided in the readme.txt) 
3. Uploaded grabmyo_visualize.py
4. Renamed biometric_feature_extraction.m to grabmyo_feature_extraction.m
5. Clarification: The correct gesture order is provided in MotionSequence.txt. The published paper has presented 'Wrist Flexion' first and then 'Wrist Extension'. In the motion sequence, the order is 'Wrist Extension' and then 'Wrist Flexion'.
6. Updated Usage notes and readme.txt

Version 1.0.2. The project was updated as follows:

1. Updated older citation article. New one is: Pradhan, A., He, J. & Jiang, N. Multi-day dataset of forearm and wrist electromyogram for hand gesture recognition and biometrics. Sci Data 9, 733 (2022). https://doi.org/10.1038/s41597-022-01836-y

Version 1.0.0. Initial release

Ethics

The experiments were conducted following the Declaration of Helsinki and the research protocol was approved by the Office of Research Ethics of the University of Waterloo (ORE# 31346). The authors declare no ethics concerns.

Conflicts of Interest

The authors have no conflicts of interest

References

1. G. Dahia, L. Jesus, and M. Pamplona Segundo, "Continuous authentication using biometrics: An advanced review," Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 10, no. 4, p. e1365, 2020
2. A. Pradhan, J. He, and N. Jiang, "Score, Rank, and Decision-Level Fusion Strategies of Multicode Electromyogram-based Verification and Identification Biometrics," IEEE Journal of Biomedical and Health Informatics, 2021.
3. J. He and N. Jiang, "Biometric From Surface Electromyogram (sEMG): Feasibility of User Verification and Identification Based on Gesture Recognition," Frontiers in Bioengineering and Biotechnology, vol. 8, p. 58, 2020
4. X. Jiang et al., "Enhancing IoT Security via Cancelable HD-sEMG-based Biometric Authentication Password, Encoded by Gesture," IEEE Internet of Things Journal, 2021.
5. L. Lu, J. Mao, W. Wang, G. Ding, and Z. Zhang, "A study of personal recognition method based on EMG signal," IEEE Transactions on Biomedical Circuits and Systems, vol. 14, no. 4, pp. 681-691, 2020.
6. S. Said, A. S. Karar, T. Beyrouthy, S. Alkork, and A. Nait-ali, "Biometrics Verification Modality Using Multi-Channel sEMG Wearable Bracelet," Applied Sciences, vol. 10, no. 19, p. 6960, 2020
7. A. Pradhan, J. He, and N. Jiang, "Performance Optimization of Surface Electromyography based Biometric Sensing System for both Verification and Identification," IEEE Sensors Journal, 2021.
8. B. Fan, X. Su, J. Niu, and P. Hui, "EmgAuth: Unlocking Smartphones with EMG Signals," arXiv preprint arXiv:2103.12542, 2021
9. S. Shin, M. Kang, J. Jung, and Y. T. Kim, "Development of Miniaturized Wearable Wristband Type Surface EMG Measurement System for Biometric Authentication," Electronics, vol. 10, no. 8, p. 923, 2021.
10. P. Kaczmarek, T. Mańkowski, and J. Tomczyński, "putEMG—a surface electromyography hand gesture recognition dataset," Sensors, vol. 19, no. 16, p. 3548, 2019
11. A. Turgunov, K. Zohirov, and B. Muhtorov, "A new dataset for the detection of hand movements based on the SEMG signal," in 2020 IEEE 14th International Conference on Application of Information and Communication Technologies (AICT), 2020, pp. 1-4: IEEE
12. F. Giordaniello et al., "Megane pro: myo-electricity, visual and gaze tracking data acquisitions to improve hand prosthetics," in 2017 International Conference on Rehabilitation Robotics (ICORR), 2017, pp. 1148-1153: IEEE
13. Y. Fang, X. Zhang, D. Zhou, and H. Liu, "Improve inter-day hand gesture recognition via convolutional neural network based feature fusion," International Journal of Humanoid Robotics, 2020
14. A. Dwivedi, Y. Kwon, and M. Liarokapis, "EMG-Based Decoding of Manipulation Motions in Virtual Reality: Towards Immersive Interfaces," in 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2020, pp. 3296-3303: IEEE.
15. J. Wu, L. Sun, and R. Jafari, "A wearable system for recognizing American sign language in real-time using IMU and surface EMG sensors," IEEE Journal of Biomedical and Health Informatics, vol. 20, no. 5, pp. 1281-1290, 2016.
16. G. Marano, C. Brambilla, R. M. Mira, A. Scano, H. Müller, and M. Atzori, "Questioning Domain Adaptation in Myoelectric Hand Prostheses Control: An Inter-and Intra-Subject Study," Sensors, vol. 21, no. 22, p. 7500, 2021.
17. U. Côté-Allard et al., "Unsupervised domain adversarial self-calibration for electromyography-based gesture recognition," IEEE Access, vol. 8, pp. 177941-177955, 2020
18. U. Côté-Allard, E. Campbell, A. Phinyomark, F. Laviolette, B. Gosselin, and E. Scheme, "Interpreting deep learning features for myoelectric control: A comparison with handcrafted features," Frontiers in Bioengineering and Biotechnology, vol. 8, p. 158, 2020.
19. M. M.-C. Vidovic, H.-J. Hwang, S. Amsüss, J. M. Hahne, D. Farina, and K.-R. Müller, "Improving the robustness of myoelectric pattern recognition for upper limb prostheses by covariate shift adaptation," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 24, no. 9, pp. 961-970, 2015.
20. C. Prahm et al., "Counteracting electrode shifts in upper-limb prosthesis control via transfer learning," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 27, no. 5, pp. 956-962, 2019
21. Pradhan, A., Jiang, N., Chester, V., & Kuruganti, U. (2020). Linear regression with frequency division technique for robust simultaneous and proportional myoelectric control during medium and high contraction-level variation. Biomedical Signal Processing and Control, 61, 101984.