Evaluation of ECG Analyzers

bxb -r record -a reference-annotator test-annotator [ options ... ]
rxr -r record -a reference-annotator test-annotator [ options ... ]
mxm -r record -a reference-annotator test-annotator [ options ... ]
epic -r record -a reference-annotator test-annotator [ options ... ]
sumstats file
plotstm file
ecgeval
nst [ options ... ]

The motivation for developing the MIT and AHA databases was to provide material for evaluating the accuracy of arrhythmia detectors, particularly with respect to ventricular arrhythmias. Between 1984 and 1987, the Association for the Advancement of Medical Instrumentation (AAMI) sponsored the development of a recommended practice (designated ECAR) for using the databases for this purpose. The aim of ECAR was to specify the evaluation methodology in sufficient detail to permit reproducible testing, and to encourage informed comparisons of the performance of ventricular arrhythmia detectors in the analysis of these standard test recordings. More recently, the AAMI has developed, and ANSI has adopted as American National Standards, a standard (ANSI/AAMI EC38:1998) for ambulatory electrocardiographs, and a companion standard (ANSI/AAMI EC57:1998) for testing and reporting performance results of cardiac rhythm and ST segment measurement algorithms. EC38 and EC57 specify standard protocols for evaluating the automated analysis algorithms that are included in many such devices. These protocols include those developed for the earlier recommended practice, and extend them to evaluation of supraventricular arrhythmia and ischemia detection. EC38 and EC57 specify the use of ‘bxb’, ‘rxr’, ‘mxm’, and ‘epic’ to perform evaluations, and further specifies the use of the MIT DB (as well as two other databases formerly available on the MIT-BIH Arrhythmia Database CDROM and now freely available on PhysioNet), the AHA DB, and (for devices that perform analysis of the ST segment) the ESC DB. If you are interested in this subject, obtain copies of the American National Standards for Ambulatory Electrocardiographs (ANSI/AAMI EC38:1998) and for Testing and Reporting Performance Results of Cardiac Rhythm and ST Segment Measurements Algorithms (ANSI/AAMI EC57:1998; see section Sources).

To evaluate an arrhythmia detector using this software, obtain for each WFDB record to be used in the test an annotation file containing the detector’s analysis of each beat. These are referred to as the ‘test’ annotation files (or the ‘algorithm’ annotation files, in EC38 and EC57). The placement of the beat annotations must match those in the reference annotations within 150 msec; thus it is not necessary to place annotations precisely at the PQ junction (as in the AHA DB reference annotations) or on the major local extremum (as in the MIT DB reference annotations). If the detector is capable of shut-down (i.e., if it inhibits its QRS detection function during periods that it judges are unreadable), the test annotation files should include a NOISE annotation with subtyp = -1 at the beginning of each period of shut-down, and a NOISE annotation with any other subtyp at the end of each such period. (If the record ends while the detector is shut down, the annotation file should include a final ‘end of shut-down’ annotation as above to permit correct shut-down accounting.) If the detector is capable of ventricular fibrillation detection, the test annotation files should also include VFON and VFOFF annotations; it is not necessary to mark flutter waves (use FLWAV annotations to do so if desired). See the man page for ‘epic’, in the WFDB Applications Guide, for information on marking atrial fibrillation, ischemic ST episodes, and ST deviation measurements in test annotation files. Any annotations that appear in the first five minutes of an annotation file are treated as belonging to the detector’s learning period, and are not used in the evaluation. The evaluation software examines such annotations only to determine the detector’s state (normal, shut down, or in VF) at the beginning of the test period.

Program ‘bxb’ implements the beat-by-beat comparison algorithm described in EC38 (section 4.2.14.2.2) and EC57 (section 4.3.2). By default, the output is in a self-explanatory matrix format. The ‘-L’ option, which must be followed by two file names, specifies that the output of ‘bxb’ should be written in line format, for further processing by ‘sumstats’. The line-format output includes column headings only if the output file must be created from scratch. In this way, ‘bxb’ can be used repeatedly to build up a line-format tables for multiple records. Among the other options is ‘-o’, which causes ‘bxb’ to generate an output annotation file (with annotator name ‘bxb’) indicating agreements and discrepancies between the input annotators.

‘rxr’ can be used to performed the run-by-run comparison described in EC38 (section 5.2.14) and in EC57 (sections 4.4.3 and 4.4.4). ‘mxm’ compares heart rate, HRV, or other measurements, as described in EC38 (section 4.2.14.2.3). ‘epic’ evaluates VF and AF detection, and ST analysis, as described in EC38 (sections 5.2.14), and EC57 (sections 4.5 and 4.6). These programs also accept a ‘-L’ option to produce line-format output as for ‘bxb’.

‘sumstats’ derives the record-by-record, episode-by-episode, and aggregate performance statistics described in EC38 (section 4.2.14.3) and in EC57 (sections 3.5.2 and 3.5.3) from line-format output files produced by ‘bxb’, ‘rxr’, ‘mxm’, and ‘epic’. The input file must include the column headings so that ‘sumstats’ can recognize the file type. The output includes a copy of the input, with aggregate statistics appended at the end. ‘plotstm’ generates a PostScript scatter plot of ST measurement comparisons gathered by ‘epic’, as described in EC57 (section 4.6.2).

The easiest way to use these programs is to run ‘ecgeval’, which generates a script (batch) file to run ‘bxb’, ‘rxr’, etc., for each record in a database. See Evaluating ECG Analyzers (in the WFDB Applications Guide) for details.

By adding noise to annotated ECG records, the noise tolerance of an arrhythmia detector can be measured. This idea was described by the author, along with W.K. Muldrow and R.G. Mark, in “A noise stress test for arrhythmia detectors”, Computers in Cardiology 11:381-384 (1984). Program ‘nst’ adds calibrated amounts of noise to ECGs (or other signals), generating an output record in WFDB format. ‘nst’ was used to generate the graded series of noisy ECG records in the MIT-BIH Noise Stress Test Database (see http://physionet.org/physiobank/database/nstdb/). These records are among those specified as standard test material by EC38 (section 4.2.14.2) and EC57 (section 3.2).