Participate in EXACT

Online submission of algorithm's results is still under development. However, we are open for submission of results. Please contact us by email at exact09challenge@gmail.com for details on how to get your algorithm evaluated.

Data

The data and EXACT reference standard are described in detail in
P. Lo, B. van Ginneken, J.M. Reinhardt, Y Tarunashree, P.A. de Jong, B Irving, C Fetita, M Ortner, R Pinho, J. Sijbers, M Feuerstein, A Fabijanska, C Bauer, R Beichel, C.S. Mendoza, R. Wiemker, J. Lee, A.P. Reeves, S. Born, O Weinheimer, E.M. van Rikxoort, J Tschirren, K Mori, B Odry, D.P. Naidich, I.J. Hartmann, E.A. Hoffman, M. Prokop, J.H. Pedersen and M. de Bruijne, "Extraction of Airways from CT (EXACT’09)", in IEEE Transactions on Medical Imaging, vol. 31, pp. 2093-2107, 2012 [link][PDF]

The images are volumetric chest CT scans acquired at different sites using several different scanners, scanning protocols, and reconstruction parameters. The dataset ranges from clinical dose to ultra low dose scans, from healthy volunteers to patients with severe lung disease, and from full inspiration to full expiration.

All data are stored as anonymized DICOM slices, in directories named after the patient name stored in the DICOM header. The images are divided into two sets: a training set and a testing set. If your method requires training or parameter tuning, please do this on the training set only.

Rules

The collection of the data, organization of the event, evaluation of segmentations, and maintenance of this website requires a large effort. We are committed to maintaining this site as a public repository of benchmark results on our test data set in the spirit of cooperative scientific progress. In return, we ask everyone who uses this site to respect the rules below.

The content of this site may not be copied or redistributed without explicit permission from the organizers. This also applies to all images contained on the site and any data that can be derived from them.

The following rules apply to those who register a team and download data:

  • The original data sets and associated segmentation data downloaded here, or any data derived from these data sets, must not be given nor redistributed under any circumstances to persons not belonging to the registered team.

  • In any publications (journal publications, conference papers, technical reports, presentations at conferences and meetings, etc.) that use the data from this website, you must cite the following paper:

    P. Lo, B. van Ginneken, J.M. Reinhardt, Y Tarunashree, P.A. de Jong, B Irving, C Fetita, M Ortner, R Pinho, J. Sijbers, M Feuerstein, A Fabijanska, C Bauer, R Beichel, C.S. Mendoza, R. Wiemker, J. Lee, A.P. Reeves, S. Born, O Weinheimer, E.M. van Rikxoort, J Tschirren, K Mori, B Odry, D.P. Naidich, I.J. Hartmann, E.A. Hoffman, M. Prokop, J.H. Pedersen and M. de Bruijne, "Extraction of Airways from CT (EXACT’09)", in IEEE Transactions on Medical Imaging, vol. 31, pp. 2093-2107, 2012
    [download from publisher] [link]

Submission of results

A pdf document describing the algorithm, including any parameter settings that has been used to generate the results, must be submitted along with the results. Any user interaction should be clearly explained.

Results for both the training and the testing set are required. The segmentations submitted for the testing set will be evaluated against the references standard, while those of the training set may be made publicly available on this website.

Airway extraction results should be submitted as binary segmentation images in ".mhd" format, with char as pixel type. This format is ITK compatible and a full documentation is available here. The format consists of a header file, which stores information about the dimension of the file, and a raw image file, where the voxels are stored consecutively with index running first over x, then y, then z.

Please note that the segmentation should be stored in an "upside down" manner, such that the first slice corresponds to the lower part of the body.

The filename of the segmentation should be the same as the patient name (tag "0010,0010") stored in the DICOM header, which is also the folder name containing the DICOM slices. The results should consist of two compressed archive files, one for results on the training set and another one for results on the testing set. The allowed compression formats is ".zip". Below is a list of names that should be used and the corresponding size of the raw image file:

Training setTesting set
FilenameSize (in bytes)
CASE01153878528
CASE02153878528
CASE03147062784
CASE04131334144
CASE0595944704
CASE06131334144
CASE07109051904
CASE08107741184
CASE09107741184
CASE1080478208
CASE1192274688
CASE12100663296
CASE1376808192
CASE14100139008
CASE15133169152
CASE16108003328
CASE17115605504
CASE18110886912
CASE1963700992
CASE2083623936
FilenameSize (in bytes)
CASE21140509184
CASE22140509184
CASE23176947200
CASE24131334144
CASE25139198464
CASE26131334144
CASE2792012544
CASE2899876864
CASE2998566144
CASE3083361792
CASE3177856768
CASE32104071168
CASE33102236160
CASE34102498304
CASE35124780544
CASE36110886912
CASE37130547712
CASE38119275520
CASE3969992448
CASE4084934656


Airway tree segmentations should be a single 6-connected structure, containing at least 10% of the lower part of the trachea. If a submitted segmentation consists of several unconnected components, only the component that contains the trachea will the evaluated. The trachea is only needed as a starting point and is excluded from the measurements.

Algorithms can be submitted in two categories:
  • Fully automated methods
    As the name implies, the methods submitted under this category should not require any manual initialization, user interaction, or manual parameter tuning on images of the testing set.

  • Semi-automated methods
    Methods submitted under this category can use any form of user interaction. Documentation of methods under this category must state clearly what kind of interactions are required, and an estimate of the number of interactions and the time needed to complete a segmentation should be provided.


Evaluation of submitted results

Submitted results are compared against the reference standard constructed from the results of 15 algorithms which were originally submitted to the EXACT'09 challenge workshop. The reference standard is described in detail in
P. Lo, B. van Ginneken, J.M. Reinhardt, Y Tarunashree, P.A. de Jong, B Irving, C Fetita, M Ortner, R Pinho, J. Sijbers, M Feuerstein, A Fabijanska, C Bauer, R Beichel, C.S. Mendoza, R. Wiemker, J. Lee, A.P. Reeves, S. Born, O Weinheimer, E.M. van Rikxoort, J Tschirren, K Mori, B Odry, D.P. Naidich, I.J. Hartmann, E.A. Hoffman, M. Prokop, J.H. Pedersen and M. de Bruijne, "Extraction of Airways from CT (EXACT’09)", in IEEE Transactions on Medical Imaging, vol. 31, pp. 2093-2107, 2012 [link][PDF]

The following measurements are computed:
  1. Branch count
    The number of branches that are detected correctly. A branch is considered detected as long as the length of the centerlines is more than 1 mm.

  2. Branch detected
    The fraction of branches that are detected, with respect to the branches present in the reference standard.

  3. Tree length
    The sum of the length of the centerlines of all correctly detected branches.

  4. Tree length detected
    The fraction of tree length in the reference standard that is detected correctly.

  5. Leakage count
    The number of unconnected groups of "correct" regions that are neighboring with a "wrong" region. Indicates how easy/difficult it is to manually separate leakages from the correctly detected branches.

  6. Leakage volume
    The volume of regions that are wrongly detected.

  7. False positive rate
    The fraction of the volume of regions that are detected wrongly over the volume of all detected regions.

Note: The trachea is excluded for computing the sensitivity measures (tree and branch length and % detected). For the leakage measures, both trachea and main bronchi are excluded.