ScientISST MOVE: Annotated Wearable Multimodal Biosignals recorded during Everyday Life Activities in Naturalistic Environments

Abstract

Existing datasets containing physiological data are mostly acquired at rest or in controlled scenarios. As a result, algorithms developed using such data may not perform as well as with biosignals acquired in dynamic and uncontrolled environments. ScientiSST MOVE is a multimodal dataset recording natural everyday activities, including lifting a chair, greeting, gesticulating, walking and running. Data was collected using three wearable devices, namely: a chestband, an armband, and the Empatica E4 wristband. This setup enabled recording of multi-channel Electrodermal Activity (EDA), Photoplethysmography (PPG) and Electrocardiography (ECG). Additionally, recordings were also made for bicep Electromyography (EMG), wrist temperature and chest and wrist actigraphy. A total of 17 healthy volunteers participated in the experimental data acquisition sessions, resulting in an average of 37 useful minutes of synchronised data from all sensors. ScientISST MOVE has been primarily designed to study the effect of daily activities on physiological data acquisition. Having been acquired with multiple wearable devices, some of which measuring the same modalities, it can also be useful in signal quality comparison studies.

Background

The human physiology responds dynamically to factors such as age, sex, and health conditions [1-3]. It is also influenced by daily activities, such as sitting or walking [4], performing physical efforts [5] or highly cognitive tasks [6]. The human body behaves differently when at rest or at stress [7, 8], when at different temperatures [3, 9], and it is also influenced by circadian rhythms [10, 11].

Recorded biosignals provide insights into these physiological changes [12, 13]. Key biosignal modalities include ECG, PPG, SpO2, EMG, EDA, and motion sensors like ACC and GYR. Wearable devices have gained popularity for biosignal acquisition [14-16], since they offer portability, ease of use, and continuous monitoring, making it feasible to capture daily-life physiological phenomena [17].

Wearable biosignal datasets are crucial for developing and testing denoising, feature extraction, and event detection algorithms that can perform well in real-world conditions [18]. Our contribution, the ScientISST MOVE dataset, includes 7 biosignal modalities, 3 sensor-equipped devices, and over 6 everyday activities, allowing different projects in signal denoising and wearable comparison, studying the physiological differences between daily activities, or that aim at getting a full picture of the body state on daily environments.

Methods

We collected the electrocardiogram (ECG), the electrodermal activity (EDA), the photoplethysmogram (PPG), the electromyogram (EMG), the skin temperature (TEMP), the chest acceleration (C-ACC), and the wrist acceleration (W-ACC), of healthy volunteers. The following subsection describes the hardware setup.

Hardware

In total, 10 sensors were used, distributed over three devices, each identified as Sx, where x is a number from 1 to 10. The following Table summarises the modalities and placement of each sensor:

Two of the devices were crafted out of ScientISST Core boards [19], one to be worn on the chest and another to be worn on the left forearm, hence herein they will be referred to as ScientISST-Chest and ScientISST-Forearm, respectively. In these two, all sensors were synchronously sampling at 500 Hz. S1 is connected to three gel electrodes, two of them were placed contralaterally on the chest - equivalent to Lead I - and the ground on the left iliac crest, all of them secured in place with surgical tape. S2 is connected to the dry contacts of a Polar chest band with no ground reference. S1 electrodes were placed right below the dry contacts, so that they capture a double-view of the same biosignal - potentially, one channel more resistant to noise (S1) and the other more prone to noise (S2). The same arrangement was prepared for the pairs S3-S4 and S5-S6. S5 was placed on the left index finger, using a gold-standard optical interface, also secured in place with surgical tape, whereas S6 recorded with the Empatica E4 wrist optical interface. The third device - the Empatica E4 wristband [20] - acquired EDA at 4 Hz, PPG at 64 Hz, temperature at 4 Hz and ACC at 32 Hz sampling frequency.

Acquisition Protocol

Each session with a volunteer was divided in five stages:

1. Briefing and Questionnaire: Demographic and clinical information was collected from the volunteers.
2. Wearing and Adjusting the Hardware Setup: All the devices and electrodes were placed on the volunteers' body. The researcher operating the experiment would check if every sensor was properly acquiring and sending data to the dedicated devices. Two mobile phones were used for mobility convenience. On these, both the ScientISST Sense Web App and the Empatica E4 Connect App were installed, to deal with the synchronous acquisition and data storage. Before proceeding, the initial timepoint of the session was marked on all three devices, in order to later synchronise the biosignals.
3. Acquisition of Biosignals: The protocol was thought out to include a wide range of activities humans execute on a daily basis. During each session, upon request of the accompanying researcher, subjects would press the Empatica E4 event button to mark the onset and offset of each activity. The volunteers were asked to execute the following main activities:
   1. Lift: To repeatedly lift a chair;
   2. Greetings: To repeatedly handshake and to wave with the left hand;
   3. Gesticulate: To gesticulate with both hands while talking;
   4. Jumps: To repeatedly jump;
   5. Walk-Before: To walk outside before running;
   6. Run: To run outside;
   7. Walk-After: To walk outside after running.
4. Uploading and De-Identifying the Data: All files from the ScientISST and Empatica E4 devices were retrieved to a computer and opened on Python with LTBio [21]. According to the initial timepoints of each device, LTBio synchronises all biosignals. Also with LTBio, regarding the subject, the following associations were made to the biosignals - age, gender, any reported medical conditions, surgical procedures, and current medication - information which was gathered in Stage 1. Moreover, a subject code was attributed to each volunteer. Subject codes were sequences of 4 alphanumeric figures randomly generated. Also, the start date and time of the biosignals was shifted to midnight of the first of January of 2000. Finally, the biosignals were saved in the .biosignal format, the serialisation format offered by LTBio, and the original files were deleted. No name, birthdate, session date, or any other unique identifier, according to the HIPAA Safe Harbor De-Identification guidelines, was present in the biosignal files.
5. Annotating the Biosignals: Later on the same day of each session, after the volunteer had left, the research operating the experiment annotated the biosignals with LTBio's Events. Every event was reviewed and annotated with a standard set of labels (e.g. "run", "lift", etc.), which is described in Data Description. Additionally, more associations were made to the biosignal, namely, location of the electrodes and name of the channels, which was an automated equal procedure for all subjects.

Data Description

Participants

Seventeen volunteers were selected to participate in the acquisition trial. The cohort comprehends 10 (59%) male and 7 (41%) female Caucasian subjects, with a median age of 24 years old. In the file subjects_info.csv ​​, find each subject details in a tabular form. For each subject (row), you can access in each column their age at the time of acquisition, their gender, and their relevant clinical history.

All subjects completed their session without quitting and, to the present day, none has requested the deletion of any of their biosignals or associated data. However, for different reasons, not all activities were performed in all sessions. The following Table shows which activities were performed in each session and the useful recorded duration of each. By "useful" it should be understood "after discarding the periods in which no activity was being executed".

Additionally, as part of technical difficulties, natural to experimental data gathering studies, the ScientISST-Forearm acquisition of session ME93 and a portion of session H39D were considered invalid and, therefore, excluded from the dataset.

EDF Files Structure

The biosignals are provided in EDF+C files, which can be opened in Python or MATLAB, for instance. These files are grouped by subject, each subject having its own directory. The following tree structure is found subject/x.edf, where subject is the  subject's code, and x is any of the following {scientisst_chest, scientisst_forearm, empatica }.

Each file:

• Includes the signals from the device after which the file is named. For instance, scientisst_chest.edf files contain S1, S2 and S7. The channels can be identified by the names { ecg-gel, ecg-dry, eda-gel, eda-dry, ppg-index, ppg-wrist, acc-chest-AXIS, acc-wrist-AXIS, emg, temp }, respectfully from S1 to S10, where AXIS can take the values { x, y, z }.
• Includes the activities onsets and durations, in the EDF annotations channel, with the labels { baseline, lift, greetings, jumps, walk_before, run, walk_after }.
• Includes, in the header, the subject's code and gender.

In some sessions, sub-activities were annotated, such as if the subject was going downstairs (walk_before_downstairs), or when running if the subject sprinted (sprint), or when lifting the chair if the activity was repeated (lift-1, lift-2).

Usage Notes

We provide a Jupyter notebook to get started with the dataset in our GitHub repository [22], which were developed and tested on Python 3.10.4. There, this notebook (edf.ipynb) lets you get a grasp on how to open the files and index activity periods of interest. At the end, it also presents some examples on how to preprocess (e.g. filter, normalize, etc.) the signals. If using EDF files, the MNE package [23] is a recommended choice for Python users, as it also offers post-processing methods.

Ethics

This trial and respective acquisition protocol were unanimously approved by the Ethics Committee of Instituto Superior Técnico (Lisbon, Portugal), under the process of internal reference number 22/2022. There were no adverse events to declare in the course of the trial. Moreover, volunteers were informed they could take breaks between the activities of the third stage, or even during the activities if they were feeling any discomfort. Volunteers also had the right to drop out of the trial at any stage of the protocol, or even after the session had ended, in which case all their biosignals and data would be immediately and permanently deleted.

Volunteers were informed that all biosignals and data acquired from them were stored unlinked from their name, address, date of session, or any other piece of information that could be linked to their identity. They were informed that, on the other hand, their age, sex, and relevant clinical history would be linked to their data, for research purposes. By signing the consent form, volunteers also gave permission to anonymously share their biosignals and data in a public dataset for research purposes.

Acknowledgements

This work was partially funded by the IST research grants BL88/2022 and BL16/2023, under the scope of project 1018P.06071.1.01.01 "CardioLeather", by the IT research grant BI16/2021, under the project PCIF/SSO/0163/2019 ”SafeFire”, and by the Fundação para a Ciência e Tecnologia (FCT) / Ministério da Ciência, Tecnologia e Ensino Superior (MCTES) research grants 2021.08297.BD and 2022.12369.BD, through national funds and when applicable co-funded by EU funds. The authors also thank to the participants that volunteered for this trial and José Gouveia for the hardware support.

Conflicts of Interest

The authors declare that they have no known competing financial interests or personal relationships which have or could be perceived to have influenced the acquisition trials. ScientISST boards and the LTBio software were provided free-of-charge by ScientISST, a non-profitable educational organisation. H.P.S. was involved in the development of the ScientISST boards and J.A.S. and M.A. in the development of LTBio. The authors have no relation with Empatica, from which the Empatica E4 device was bought.

References

1. Hunter, S. K., Critchlow, A. & Enoka, R. M. Influence of aging on sex differences in muscle fatigability. J. Appl. Physiol. 97, 1723–1732, 10.1152/japplphysiol.00460.2004 (2004).
2. Macfarlane, P. W. The influence of age and sex on the electrocardiogram. In Kerkhof, P. L. M. & Miller, V. M. (eds.) Sex-Specific Analysis of Cardiovascular Function, Advances in Experimental Medicine and Biology, 93–106, 10.1007/978-3-319-77932-4_6 (Springer International Publishing, Cham, 2018).
3. Cramer, M. N., Gagnon, D., Laitano, O. & Crandall, C. G. Human temperature regulation under heat stress in health, disease, and injury. Physiol. Rev. 102, 1907–1989, 10.1152/physrev.00047.2021 (2022).
4. Cook, G., Burton, L., Hoogenboom, B. J. & Voight, M. Functional movement screening: The use of fundamental movements as an assessment of function - Part 1. Int J Sports Phys Ther 9, 396–409 (2014).
5. Seshadri, D. R. et al. Wearable sensors for monitoring the physiological and biochemical profile of the athlete. NPJ Digit. Med. 2, 1–16, 10.1038/s41746-019-0150-9 (2019).
6. Posada-Quintero, H. F. & Bolkhovsky, J. B. Machine learning models for the identification of cognitive tasks using autonomic reactions from heart rate variability and electrodermal activity. Behav. Sci. 9, 45, 10.3390/bs9040045 (2019).
7. McArdle, W. D., Katch, F. I. & Katch, V. L. Exercise physiology: nutrition, energy, and human performance (Wolters Kluwer Health/Lippincott Williams & Wilkins, Philadelphia, 2015).
8. Ladakis, I. & Chouvarda, I. Overview of biosignal analysis methods for the assessment of stress. Emerg. Sci. J. 5, 233–244, 10.28991/esj-2021-01267 (2021).
9. Majumder, R., Mohamed Nazer, A. N., Panfilov, A. V., Bodenschatz, E. & Wang, Y. Electrophysiological characterization of human atria: The understated role of temperature. Front. Physiol. 12 (2021).
10. Black, N. et al. Circadian rhythm of cardiac electrophysiology, arrhythmogenesis, and the underlying mechanisms. Hear. Rhythm. 16, 298–307, 10.1016/j.hrthm.2018.08.026 (2019).
11. Miklošíková, M., Tomaszek, L. & Malcˇík, M. Circadian rhythm and skin conductivity, their measurement and use of obtained data to plan daily activities. In Proc. of the 20th Int’l Conf. on Emerging eLearning Technologies and Applications (ICETA), 422–427, 10.1109/ICETA57911.2022.9974888 (2022).
12. Penzel, T., Kesper, K. & Becker, H. F. Biosignal monitoring and recording. In Zielinski, K., Duplaga, M. & Ingram, D. (eds.) Information Technology Solutions for Healthcare, Health Informatics, 288–301, 10.1007/1-84628-141-5_13 (Springer, London, 2006).
13. Mukhopadhyay, S. C. Wearable sensors for human activity monitoring: A review. IEEE Sensors J. 15, 1321–1330, 10.1109/JSEN.2014.2370945 (2015).
14. Prieto-Avalos, G. et al. Wearable devices for physical monitoring of heart: A review. Biosensors 12, 292, 10.3390/ bios12050292 (2022).
15. Ates, H. C. et al. End-to-end design of wearable sensors. Nat Rev Mater 7, 887–907, 10.1038/s41578-022-00460-x (2022).
16. Kim, M. et al. Emerging bio-interfacing wearable devices for signal monitoring: Overview of the mechanisms and diverse sensor designs to target distinct physiological bio-parameters. Adv. Sens. Res. n/a, 2200049, 10.1002/adsr.202200049 (2023).
17. Stuart, T., Hanna, J. & Gutruf, P. Wearable devices for continuous monitoring of biosignals: Challenges and opportunities. APL Bioeng 6, 021502, 10.1063/5.0086935 (2022).
18. Yoon, D., Jang, J.-H., Choi, B. J., Kim, T. Y. & Han, C. H. Discovering hidden information in biosignals from patients using artificial intelligence. Korean J Anesth. 73, 275–284, 10.4097/kja.19475 (2020).
19. ScientISST Sense: https://scientisst.com/sense [Accessed 20/09/2023]
20. Empatica E4: https://empatica.com/research/e4 [Accessed 20/09/2023]
21. LTBio: https://pypi.org/project/LongTermBiosignals/ [Accessed 20/09/2023]
22. ScientISST MOVE Repository: https://github.com/jomy-kk/ScientISST-MOVE/tree/main/usage [Accessed 20/09/2023]
23. MNE Library: https://mne.tools [Accessed 20/09/2023]