Detecting and Quantifying Apnea Based on the ECG: The PhysioNet/Computing in Cardiology Challenge 2000

Introduction

Obstructive sleep apnea (intermittent cessation of breathing) is a common problem with major health implications, ranging from excessive daytime drowsiness to serious cardiac arrhythmias. Obstructive sleep apnea is associated with increased risks of high blood pressure, myocardial infarction, and stroke, and with increased mortality rates. Standard methods for detecting and quantifying sleep apnea are based on respiration monitoring, which often disturbs or interferes with sleep and is generally expensive. A number of studies during the past 15 years have hinted at the possibility of detecting sleep apnea using features of the electrocardiogram. Such approaches are minimally intrusive, inexpensive, and may be particularly well-suited for screening. The major obstacle to use of such methods is that careful quantitative comparisons of their accuracy against that of conventional techniques for apnea detection have not been published.

We therefore offer a challenge to the biomedical research community: demonstrate the efficacy of ECG-based methods for apnea detection using a large, well-characterized, and representative set of data. The goal of the contest is to stimulate effort and advance the state of the art in this clinically significant problem, and to foster both friendly competition and wide-ranging collaborations. We will award prizes of US$500 to the most successful entrant in each of two events.¹

Data for development and evaluation

Data for this contest have kindly been provided by Dr. Thomas Penzel of Philipps-University, Marburg, Germany, and are available here.

The data to be used in the contest are divided into a learning set and a test set of equal size. Each set consists of 35 recordings, containing a single ECG signal digitized at 100 Hz with 12-bit resolution, continuously for approximately 8 hours (individual recordings vary in length from slightly less than 7 hours to nearly 10 hours). Each recording includes a set of reference annotations, one for each minute of the recording, that indicate the presence or absence of apnea during that minute. These reference annotations were made by human experts on the basis of simultaneously recorded respiration signals. Note that the reference annotations for the test set will not be made available until the conclusion of the contest. Eight of the recordings in the learning set include three respiration signals (oronasal airflow measured using nasal thermistors, and chest and abdominal respiratory effort measured using inductive plethysmography) each digitized at 20 Hz, and an oxygen saturation signal digitzed at 1 Hz. These additional signals can be used as reference material to understand how the apnea annotations were made, and to study the relationships between the respiration and ECG signals.

The database does not contain episodes of pure central apnea or of Cheyne-Stokes respiration; all apneas in these recordings are either obstructive or mixed. Minutes containing hypopneas (defined as intermittent drops in respiratory flow below 50%, accompanied by drops in oxygen saturation of at least 4%, and followed by compensating hyperventilation) are also scored as minutes containing apnea. Additional information about the recordings was posted here after the conclusion of the competition, including (for all recordings) age, gender, height, weight, AI (apnea index), HI (hypopnea index), and AHI (apnea-hypopnea index). The subjects of these recordings are men and women between 27 and 63 years of age, with weights between 53 and 135 kg (BMI between 20.3 and 42.1); AHI ranges from 0 to 93.5 in these recordings.

Sleep apnea definitions

Several definitions for clinically significant sleep apnea have been in clinical use since 1978, when Guilleminault defined "sleep apnea syndrome" as more than 30 apneas per night. In 1981, Lavie proposed a more selective criterion of 100 apneas per night. Later criteria were based on an "apnea index" (the number of apneas per hour, or the number of minutes containing apnea per hour). Most clinicians regard an apnea index below 5 as normal, and an apnea index of 10 or more as pathologic. In 1988, He et al. found increased mortality in untreated patients with apnea indices of 20 or more, and such patients are now recognized as in need of treatment. Criteria used in current practice rely not only on an apnea index, but also on symptoms and cardiovascular sequelae.²

Data classes

For the purposes of this challenge, based on these varied criteria, we have defined three classes of recordings:

• Class A (Apnea): These meet all criteria. Recordings in class A contain at least one hour with an apnea index of 10 or more, and at least 100 minutes with apnea during the recording. The learning and test sets each contain 20 class A recordings.
• Class B (Borderline): These meet some but not all of the criteria. Recordings in class B contain at least one hour with an apnea index of 5 or more, and between 5 and 99 minutes with apnea during the recording. The learning and test sets each contain 5 class B recordings.
• Class C (Control): These meet none of the criteria, and may be considered normal. Recordings in class C contain fewer than 5 minutes with apnea during the recording. The learning and test sets each contain 10 class C recordings.

Events and scoring

Each entrant may compete in one or both of the following events:

1. Apnea screening

In this event, your task is to design software that can classify the 35 test set recordings into class A (apnea) and class C (control or normal) groups, using the ECG signal to determine if significant sleep apnea is present. Your classifications for the 5 class B (borderline) recordings will not influence your score in this event (but you must classify them into either class A or class C, since you will not know which records belong to class B until the correct classifications of the 35 test set records are disclosed after the end of the contest). Your score for this event is simply the number of correct classifications; thus the maximum score possible is 30.

An example may help to clarify the scoring: A contestant submits her results, classifying 22 recordings in class A and 13 in class C (for a total of 35). Out of the 22 recordings that her software has identified as class A, 16 of them are actually class A, 3 are class B and 3 are class C. Out of the 13 recordings that her software identified as class C, 7 have been correctly identified, and other 6 include 4 class A and 2 class B. The score in this case is 23 (16 correct class A identifications, plus 7 in class C). Class B cases do not contribute to the final score; rather, they provide a buffer zone between the cut of classes A and C.

We have chosen to exclude the class B recordings from the calculation of the scores because the utility of a screening test depends primarily on the accuracy with which it classifies the unambiguous cases, both positive and negative (classes A and C respectively in this instance). If you wish to attempt to classify recordings into all three groups, you may submit a second set of classifications, and we will calculate your score in the same way (but the maximum possible score in this case will be 35). The highest scores obtained in this way will be published, but will not be the basis for an award.

2. Quantitative assessment of apnea

In this event, your software must generate a minute-by-minute annotation file for each recording, in the same format as those provided with the learning set, using the ECG signal to determine when sleep apnea occurs. Your annotations will be compared with a set of reference annotations to determine your score. Each annotation that matches a reference annotation earns one point; thus the highest possible score for this event will be approximately 16800 (480 annotations in each of 35 records). It is important to understand that scores approaching the maximum are very unlikely, since apnea assessment can be very difficult even for human experts. Nevertheless, the scores can be expected to provide a reasonable ranking of the ability of the respective algorithms to mimic the decisions made by human experts.

Obtaining scores

A form that will permit you to submit your classifications and/or annotations for scoring is now available. You will receive a reference number and your score(s) by return e-mail. You may revise your submissions and try again if you wish, but attempts to exploit this service in order to discover the correct classifications are contrary to the spirit of the contest and will result in disqualification.

How to enter

To enter the competition, submit an abstract with a concise description of your approach to the problem to Computers in Cardiology 2000 no later than Wednesday, 3 May 2000. Your abstract must include your reference number and score(s); for this reason, do not wait until the last minute to submit your classifications and/or annotations for scoring. If your abstract is accepted, you will be expected to prepare a four-page paper for presentation during the conference and publication in the conference proceedings. We welcome and encourage contributions to PhysioNet of software developed during this competition.

Awards

The author(s) of the top-scoring eligible entry in each event will receive an award of US$500 in recognition of his or her achievement. In the event of a tie, the date of the author's abstract submission will be the tie-breaker. This rule favors early submission of abstracts, but permits authors to improve their results if they can after submitting their abstracts. Classifications or annotations received for scoring after noon GMT on Friday, 22 September 2000 will not be eligible for awards. Submissions from members and affiliates of our research groups at MIT, Boston University, Harvard Medical School, Beth Israel Deaconess Medical Center, McGill University, and Phillips-University are not eligible for awards, although all are welcome to participate.

Workshop/Panel discussion

All entrants are invited to describe their methods during a panel discussion at Computers in Cardiology in Boston on Sunday, 24 September 2000, when the awards will be given. Individual presentations of accepted papers will be scheduled for one or more sessions of the conference during the following days (25-27 September).

Acknowledgements

1. Funding for the awards has been contributed by the Margret and H.A. Rey Laboratory for Nonlinear Dynamics in Medicine at Boston's Beth Israel Deaconess Medical Center.

2. We thank Thomas Penzel for the discussion of diagnostic criteria for sleep apnea syndrome, as well as for making this event possible by his generous contribution of data.

Challenge Results

Event 1 (Apnea Screening)

In this event, the five recordings in class B are not counted. The score is the total number of correct classifications of the 20 class A (apnea) and 10 class C (control/no apnea) recordings, so that the maximum possible score is 30. Since four entrants achieved a perfect score, the date of the top-scoring entrant's submission is the tiebreaker.

The top scores in event 1 are:

Each entrant's best score is shown, along with the date when they achieved that score. Many entrants submitted multiple entries, and the 'Entries' shown indicate how many entries were submitted by each entrant up to and including the one that scored highest (later entries, and entries that did not receive scores because of formatting errors were not counted); this gives some sense of how much 'tuning' may have taken place. Entries noted as 'unofficial' came from two of the PhysioNet core research groups, and were therefore not eligible for awards, although they followed all of the rules of the competition.

Only one entry, from PK Stein, was submitted for an unofficial 'three-way score' (the total of correct classifications of all 35 records, including the 5 class B (borderline) records). This is a significantly more difficult task, since the amount of apnea in the class B records must be accurately determined in order to classify them correctly, and recognizing apnea in recordings that have only small amounts of apnea is more difficult than in recordings with frequent apneas. The single entry received a score of 33/35, an excellent result; the only errors occurred when a class A record was put in class B, and a class B record was put in class A.

Event 2 (Apnea Quantification)

In this event, each minute of each of the 35 recordings in the test set must be classified as containing apnea (A) or not (N). The maximum possible score is 17268 (the total number of minutes in the 35 recordings for which reference classifications are available).

The top scores in event 2 are:

As in event 1, each entrant's best score is shown above, along with the date it was achieved and the number of entries submitted (excluding any entries submitted after the one that received the best score, and any that were not scored because of formatting errors). Notably, four of the top five finishers in this event also achieved perfect scores in event 1. The classification accuracy achieved by the top finishers is comparable to the roughly 90% concurrence of human experts in classification of the original polysomnograms with reference to the full set of signals, including nasal airflow, respiratory effort, and oxygen saturation.

Papers

These papers were presented at Computers in Cardiology 2000.

Stimulating Rapid Research Advances Via Focused Competition: The Computers In Cardiology Challenge 2000
GB Moody, RG Mark, AL Goldberger, T Penzel
Computers in Cardiology 2000;27:207-210.

The Apnea-ECG Database
T Penzel, GB Moody, RG Mark, AL Goldberger, JH Peter
Computers in Cardiology 2000;27:255-258.

Detection of Sleep Apnoea from Frequency Analysis of Heart-Rate Variability
MJ Drinnan, J Allen, P Langley, A Murray
Computers in Cardiology 2000;27:259-262.

Detection of Sleep Apnea in Single Channel ECGs from the PhysioNet Data Base
M Schrader, C Zywietz, V von Einem, B Widiger, G Joseph
Computers in Cardiology 2000;27:263-256.

Screening for Obstructive Sleep Apnoea Based on the Electrocardiogram: The Computers in Cardiology Challenge
B Raymond, RM Cayton, RA Bates, MJ Chappell
Computers in Cardiology 2000;27:267-270.

Detecting OSAHS from Patterns Seen on Heart-Rate Tachograms
PK Stein, PP Domitrovich
Computers in Cardiology 2000;27:271-274.

Bayesian Hierarchical Model with Wavelet Transform Coefficients of the ECG in Obstructive Sleep Apnea Screening
F Ng, I García, P Gomis, A La Cruz, G Passariello, F Mora
Computers in Cardiology 2000;27:275-278.

Detection of Obstructive Sleep Apnea through Auditory Display of Heart-Rate Variability
M Ballora, B Pennycook, PCh Ivanov, A Goldberger, L Glass
Computers in Cardiology 2000;27:739-740.

Recognition and Quantification of Sleep Apnea by Analysis of Heart-Rate Variability Parameters
C Maier, M Bauch, H Dickhaus
Computers in Cardiology 2000;27:741-744.

Automatic Classification of Sleep Apnea Epochs using the Electrocardiogram
P de Chazal, C Heneghan, E Sheridan, R Reilly, P Nolan, M O'Malley
Computers in Cardiology 2000;27:745-748.

Obstructive Sleep Apnea Classification Based on Spectrogram Patterns in the Electrocardiogram
J McNames, A Fraser
Computers in Cardiology 2000;27:749-752.

Detection of Obstructive Sleep Apnea from Cardiac Interbeat Interval Time Series
JE Mietus, C-K Peng, PCh Ivanov, AL Goldberger
Computers in Cardiology 2000;27:753-756.

Obstructive Sleep Apnea Detection Based on Electrocardiogram Analysis
Z Shinar, A Baharav, S Akselrod
Computers in Cardiology 2000;27:255-258.