Participate in EXACT
Online submission of algorithm's results is still under development.
However, we are open for submission of results. Please contact us by email at exact09challenge@gmail.com for details on how to get your algorithm evaluated.
Data
The data and EXACT reference standard are described in detail in
P. Lo, B. van Ginneken, J.M. Reinhardt, Y Tarunashree, P.A. de Jong, B Irving, C Fetita, M Ortner, R Pinho, J. Sijbers, M Feuerstein, A Fabijanska, C Bauer, R Beichel, C.S. Mendoza, R. Wiemker, J. Lee, A.P. Reeves, S. Born, O Weinheimer, E.M. van Rikxoort, J Tschirren, K Mori, B Odry, D.P. Naidich, I.J. Hartmann, E.A. Hoffman, M. Prokop, J.H. Pedersen and M. de Bruijne, "Extraction of Airways from CT (EXACT’09)", in IEEE Transactions on Medical Imaging, vol. 31, pp. 2093-2107, 2012
[link][PDF]
The images are volumetric chest CT scans acquired at different sites using several different scanners, scanning protocols, and reconstruction parameters. The dataset ranges from clinical dose to ultra low dose scans, from healthy volunteers to patients with severe lung disease, and from full inspiration to full expiration.
All data are stored as anonymized DICOM slices, in directories named after the patient name stored in the DICOM header. The images are divided into two sets: a training set and a testing set. If your method requires training or parameter tuning, please do this on the training set only.
The collection of the data, organization of the event, evaluation of segmentations, and maintenance of this website requires a large effort. We are
committed to maintaining this site as a public repository of benchmark results on our test data set in the spirit of cooperative scientific progress. In return, we ask everyone who uses this site to respect the rules below.
The content of this site may not be copied or redistributed without explicit permission from the organizers. This also applies to all images contained on the site and any data that can be derived from them.
The following rules apply to those who register a team and download data:
- The original data sets and associated segmentation data downloaded here, or any data derived from these data sets, must not be given nor redistributed under any circumstances to persons not belonging to the registered team.
- In any publications (journal publications, conference papers, technical reports, presentations at conferences and meetings, etc.) that use the data from this website, you must cite the following paper:
P. Lo, B. van Ginneken, J.M. Reinhardt, Y Tarunashree, P.A. de Jong, B Irving, C Fetita, M Ortner, R Pinho, J. Sijbers, M Feuerstein, A Fabijanska, C Bauer, R Beichel, C.S. Mendoza, R. Wiemker, J. Lee, A.P. Reeves, S. Born, O Weinheimer, E.M. van Rikxoort, J Tschirren, K Mori, B Odry, D.P. Naidich, I.J. Hartmann, E.A. Hoffman, M. Prokop, J.H. Pedersen and M. de Bruijne, "Extraction of Airways from CT (EXACT’09)", in IEEE Transactions on Medical Imaging, vol. 31, pp. 2093-2107, 2012
[download from publisher]
[link]
Submission of results
A pdf document describing the algorithm, including any parameter settings that has been used to generate the results, must be submitted along with the results. Any user interaction should be clearly explained.
Results for both the training and the testing set are required. The segmentations submitted for the testing set will be evaluated against the references standard, while those of the training set may be made publicly available on this website.
Airway extraction results should be submitted as binary segmentation images in ".mhd" format, with char as pixel type. This format is ITK compatible and a full documentation is available here. The format consists of a header file, which stores information about the dimension of the file, and a raw image file, where the voxels are stored consecutively with index running first over x, then y, then z.
Please note that the segmentation should be stored in an "upside down" manner, such that the first slice corresponds to the lower part of the body.
The filename of the segmentation should be the same as the patient name (tag "0010,0010") stored in the DICOM header, which is also the folder name containing the DICOM slices. The results should consist of two compressed archive files, one for results on the training set and another one for results on the testing set. The allowed compression formats is ".zip". Below is a list of names that should be used and the corresponding size of the raw image file:
Training set | | Testing set
|
Filename | Size (in bytes)
| CASE01 | 153878528
| CASE02 | 153878528
| CASE03 | 147062784
| CASE04 | 131334144
| CASE05 | 95944704
| CASE06 | 131334144
| CASE07 | 109051904
| CASE08 | 107741184
| CASE09 | 107741184
| CASE10 | 80478208
| CASE11 | 92274688
| CASE12 | 100663296
| CASE13 | 76808192
| CASE14 | 100139008
| CASE15 | 133169152
| CASE16 | 108003328
| CASE17 | 115605504
| CASE18 | 110886912
| CASE19 | 63700992
| CASE20 | 83623936
|
|
|
Filename | Size (in bytes)
| CASE21 | 140509184
| CASE22 | 140509184
| CASE23 | 176947200
| CASE24 | 131334144
| CASE25 | 139198464
| CASE26 | 131334144
| CASE27 | 92012544
| CASE28 | 99876864
| CASE29 | 98566144
| CASE30 | 83361792
| CASE31 | 77856768
| CASE32 | 104071168
| CASE33 | 102236160
| CASE34 | 102498304
| CASE35 | 124780544
| CASE36 | 110886912
| CASE37 | 130547712
| CASE38 | 119275520
| CASE39 | 69992448
| CASE40 | 84934656
|
|
Airway tree segmentations should be a single 6-connected structure, containing at least 10% of the lower part of the trachea. If a submitted segmentation consists of several unconnected components, only the component that contains the trachea will the evaluated. The trachea is only needed as a starting point and is excluded from the measurements.
Algorithms can be submitted in two categories:
- Fully automated methods
As the name implies, the methods submitted under this category should not require any manual initialization, user interaction, or manual parameter tuning on images of the testing set.
- Semi-automated methods
Methods submitted under this category can use any form of user interaction. Documentation of methods under this category must state clearly what kind of interactions are required, and an estimate of the number of interactions and the time needed to complete a segmentation should be provided.
Evaluation of submitted results
Submitted results are compared against the reference standard constructed from the results of 15 algorithms which were originally submitted to the EXACT'09 challenge workshop. The reference standard is described in detail in
P. Lo, B. van Ginneken, J.M. Reinhardt, Y Tarunashree, P.A. de Jong, B Irving, C Fetita, M Ortner, R Pinho, J. Sijbers, M Feuerstein, A Fabijanska, C Bauer, R Beichel, C.S. Mendoza, R. Wiemker, J. Lee, A.P. Reeves, S. Born, O Weinheimer, E.M. van Rikxoort, J Tschirren, K Mori, B Odry, D.P. Naidich, I.J. Hartmann, E.A. Hoffman, M. Prokop, J.H. Pedersen and M. de Bruijne, "Extraction of Airways from CT (EXACT’09)", in IEEE Transactions on Medical Imaging, vol. 31, pp. 2093-2107, 2012
[link][PDF]
The following measurements are computed:
- Branch count
The number of branches that are detected correctly. A branch is considered detected as long as the length of the centerlines is more than 1 mm.
- Branch detected
The fraction of branches that are detected, with respect to the branches present in the reference standard.
- Tree length
The sum of the length of the centerlines of all correctly detected branches.
- Tree length detected
The fraction of tree length in the reference standard that is detected correctly.
- Leakage count
The number of unconnected groups of "correct" regions that are neighboring with a "wrong" region. Indicates how easy/difficult it is to manually separate leakages from the correctly detected branches.
- Leakage volume
The volume of regions that are wrongly detected.
- False positive rate
The fraction of the volume of regions that are detected wrongly over the volume of all detected regions.
Note: The trachea is excluded for computing the sensitivity measures (tree and branch length and % detected). For the leakage measures, both trachea and main bronchi are excluded.
|