Predicting Paroxysmal Atrial Fibrillation/Flutter: The PhysioNet/Computing in Cardiology Challenge 2001

Introduction

Following the success of the first Computers in Cardiology Challenge, we are pleased to offer a new challenge from PhysioNet and Computers in Cardiology 2001. The challenge is to develop a fully automated method to predict the onset of paroxysmal atrial fibrillation/flutter (PAF), based on the ECG prior to the event. The goal of the contest is to stimulate effort and advance the state of the art in this clinically significant problem, and to foster both friendly competition and wide-ranging collaborations.

Data for development and evaluation

PhysioNet provides free access to a set of data to be used for development and evaluation of algorithms. The PAF Prediction Challenge Database consists of 100 pairs of half-hour ECG recordings. Each pair of recordings is obtained from a single 24-hour ECG. Subjects in group A experienced PAF; for these subjects, one recording ends just before the onset of PAF, and the other recording is distant in time from any PAF (there is no PAF within 45 minutes before or after the excerpt). Subjects in group N do not have PAF; in these, the times of the recordings have been chosen at random.

The database is divided into a learning set and a test set of equal size, each containing approximately equal numbers of subjects from groups A and N. The classifications of the recordings in the learning set are provided; those for the test set will be revealed after the conclusion of the challenge.

Events and scoring

We will award prizes of US$500 to the most successful entrant in each of two events:

Event 1: PAF screening

Event 1 is intended to determine if subjects in group A can be distinguished from those in group N. (In other words, can individuals at risk of PAF be identified within a larger population, based on their ECGs?) The number of correctly classified subjects (0 to 50) is the event 1 score.

Event 2: PAF prediction

Event 2 is intended to determine if subjects in group A have distinctive and detectable changes in their ECGs immediately before PAF. (In other words, is the imminent onset of PAF predictable in an individual known to be at risk of PAF?) A successful method for doing so should be able to determine which record of each pair of group A records immediately precedes PAF. If the identities of the group A records were known, it would be sufficient to classify these records only; since the goal of event 1 is to identify group A, however, we have not provided this information! Entrants in event 2 of the challenge must therefore classify exactly one of each pair of records in the test set as `A' (defined as ``immediately preceding PAF, if the patient belongs to group A''), and the other as `N' (defined as ``not immediately preceding PAF''). One point is awarded for each correctly classified record pair, so that the event 2 scores range from n to 50 (the lower bound is n, the number of subjects in group N, because the group N subjects are always considered correctly classified).

If a tie occurs in either event, the date of the submission is the tiebreaker.

How to enter

To enter the competition:

1. Develop an algorithm for classifying the test set recordings. The algorithm must perform this task unaided (manual and semiautomated methods are not eligible).
2. Submit your classifications to PhysioNet for scoring.
3. Submit an abstract with a concise description of your approach and results to Computers in Cardiology 2001. (Deadline: 1 May 2001). Please submit your abstract using the category ECG: Arrhythmia.

If your abstract is accepted, you will be expected to prepare a four-page paper for presentation during the conference and publication in the conference proceedings. We welcome and encourage contributions to PhysioNet of software developed during this competition.

If you wish to improve your score, you may revise your entry and submit it again for scoring. The number of submissions is limited (you will be allowed six entries, which may be all in one event, or divided between the two events as you wish). If you wish to submit additional entries, the autoscorer will enforce a waiting period, which is 24 hours for the seventh entry and doubles for every subsequent entry.

If you have submitted an abstract to Computers in Cardiology 2001 on or before 1 May 2001, you are eligible for awards based on any scores you receive before the challenge deadline of noon GMT on Friday, 21 September 2001.

Links

• PAF Prediction Challenge Database. The entire database (both learning and test sets) is available for downloading here.
• Predicting Onset of Atrial Fibrillation. A brief overview of the challenge problem and its clinical significance, including a bibliography with links to a collection of useful references (your suggestions for additions to this collection are welcome).
• Obtaining CinC Challenge 2001 Scores. How to prepare and submit your results to the autoscorer. (Although the competition has ended, you may still submit your results for unofficial scoring.)
• Computers in Cardiology 2001 This is the conference web site, which includes an on-line form for composing and submitting an abstract.

Results

The top scorers in the 2001 challenge were announced during the 25 September plenary session of Computers in Cardiology in Rotterdam. The top score and the award in event 1 was obtained by Günther Schreier and colleagues of the Austrian Research Centers Seibersdorf (Graz, Austria). In event 2, the top score was obtained by Wei Zong and colleagues at the Harvard-MIT Division of Health Sciences and Technology (Cambridge, Massachusetts, USA); because this entry came from one of the PhysioNet core research groups, however, it was unofficial, and the award in event 2 was given to the team with the highest official score, who were once again Günther Schreier and colleagues. We congratulate and thank all of the participants in this challenge.

The immediate goal of the Computers in Cardiology Challenge 2001 was to develop a fully automated method to predict the onset of paroxysmal atrial fibrillation/flutter (PAF), based on the ECG prior to the event. Entrants developed and tested methods using a database created for this challenge.

The test set of the PAF Prediction Challenge Database consists of 50 pairs of half-hour ECG recordings. Each pair of recordings is obtained from a single 24-hour ECG. Subjects in group A experienced PAF; for these subjects, one recording ends just before the onset of PAF, and the other recording is distant in time from any PAF (there is no PAF within 45 minutes before or after the excerpt). Subjects in group N do not have PAF; in these, the times of the recordings have been chosen at random. Entrants were told that between 20 and 30 of the 50 subjects belong to each group; the exact size of each group (28 in group A, 22 in group N) was disclosed only after the conclusion of the challenge in September.

Brief descriptions of the methods used can be viewed by following the links in the tables below to abstracts submitted by many of the entrants for presentation at Computers in Cardiology 2001; please note that these abstracts were written no later than May 2001, and do not mention results achieved since then. For details on the CinC Challenge 2001, follow the links at the bottom of this page.

Event 1 (PAF Screening)

Event 1 was intended to determine if subjects in group A can be distinguished from those in group N. (In other words, can individuals at risk of PAF be identified within a larger population, based on their ECGs?) The number of correctly classified subjects (0 to 50) is the event 1 score. The best methods were able to achieve roughly 80% classification accuracy on the test set.

The top scores in event 1 are:

Each entrant's best score is shown, along with the date when they achieved that score. Many entrants submitted multiple entries, and the 'Entries' shown indicate how many entries were submitted by each entrant up to and including the one that scored highest (later entries, and entries that did not receive scores because of formatting errors were not counted); this gives some sense of how much 'tuning' may have taken place. The entry noted as 'unofficial' came from one of the PhysioNet core research groups, and was therefore not eligible for awards, although the entrant followed all of the rules of the competition.

Event 2 (PAF Prediction)

Event 2 was intended to determine if subjects in group A have distinctive and detectable changes in their ECGs immediately before PAF. (In other words, is the imminent onset of PAF predictable in an individual known to be at risk of PAF?) A successful method for doing so should be able to determine which record of each pair of group A records immediately precedes PAF. If the identities of the group A records were known, it would be sufficient to classify these records only; since the goal of event 1 was to identify group A, however, we did not provide this information! Entrants in event 2 of the challenge therefore were required to classify exactly one of each pair of records in the test set as `A' (defined as ``immediately preceding PAF, if the patient belongs to group A''), and the other as `N' (defined as ``not immediately preceding PAF''). One point was awarded for each correctly classified record pair, so that the raw event 2 scores range from 22 to 50 (the lower bound is 22, the number of subjects in group N, because the group N subjects are always considered correctly classified). In the table below, the scores have been adjusted by subtraction of 22 from the raw scores, so that the adjusted scores can range between 0 and 28.

The top scores in event 2 are:

As in event 1, each entrant's best score is shown above, along with the date it was achieved and the number of entries submitted (excluding any entries submitted after the one that received the best score, and any that were not scored because of formatting errors).

Papers

These papers were presented at Computers in Cardiology 2001.

Predicting the Onset of Paroxysmal Atrial Fibrillation: The Computers in Cardiology Challenge 2001
GB Moody, AL Goldberger, S McClennen, SP Swiryn

Automated Assessment of Atrial Fibrillation
P de Chazal, C Heneghan

Can Paroxysmal Atrial Fibrillation be Predicted?
P Langley, D di Bernardo, J Allen, E Bowers, FE Smith, S Vecchietti, A Murray

A Methodology for Predicting Paroxysmal Atrial Fibrillation Based on ECG Arrhythmia Feature Analysis
W Zong, R Mukkamala, RG Mark

Screening and Prediction of Paroxysmal Atrial Fibrillation by Analysis of Heart Rate Variability Parameters
C Maier, M Bauch, H Dickhaus

An Automatic ECG Processing Algorithm to Identify Patients Prone to Paroxysmal Atrial Fibrillation
G Schreier, P Kastner, W Marko

Prediction of Paroxysmal Atrial Fibrillation by Footprint Analysis
ACC Yang, HW Yin

A Two-Stage Solution Algorithm for Paroxysmal Atrial Fibrillation Prediction
KS Lynn, HD Chiang

Some Important R-R Interval Based Paroxysmal Atrial Fibrillation Predictors
G Krstacic, D Gamberger, T Smuc, A Krstacic