|
Participate in EXACT09
To participate in the comparative study of airway extraction algorithms, you need to
- Register a team
- Login with your account
- Download the data
- If necessary, tune your algorithms on the training set
- Submit your results on both the training and the testing set. To help us evaluate all results in time, please submit your results as early as possible!
- Check and confirm the submitted results. Submitted results will only be processed after confirmation.
- Submit a paper describing the algorithm
- Register for the 2nd International Workshop on Pulmonary Image Analysis, attend the workshop, and present your algorithm.
Data
The images used in this challenge are volumetric chest CT scans acquired at different sites using several different scanners, scanning protocols, and reconstruction parameters. The dataset ranges from clinical dose to ultra low dose scans, from healthy volunteers to patients with severe lung disease, and from full inspiration to full expiration.
All data are stored as anonymized DICOM slices, in directories named after the patient name stored in the DICOM header. The images are divided into two sets: a training set and a testing set. Each team must submit results for both sets. If your method requires training or parameter tuning, please do this only on the training set.
The segmentations submitted for the training set will be made publicly available after the workshop.
Submission of results
All participants must submit a paper describing the algorithm, including any parameter settings, that has been used to generate the results. Any user interaction should be clearly explained. For guidelines on the format of the paper, please follow the instructions at the Pulmonary Image Analysis Workshop website.
Airway extraction results should be submitted as binary segmentation images in ".mhd" format, with char as pixel type. This format is ITK compatible and a full documentation is available here. The format consists of a header file, which stores information about the dimension of the file, and a raw image file, where the voxels are stored consecutively with index running first over x, then y, then z.
Please note that the segmentation should be stored in an "upside down" manner, such that the first slice corresponds to the lower part of the body.
The filename of the segmentation should be the same as the patient name (tag "0010,0010") stored in the DICOM header, or the folder name containing the DICOM slices, which are also named according to the patient name. The results should consist of two compressed archive files, one for results on the training set and another one for results on the testing set. The allowed compression formats is ".zip". Below is a list of names that should be used and the corresponding size of the raw image file:
| Training set | | Testing set
|
| Filename | Size (in bytes)
| | CASE01 | 153878528
| | CASE02 | 153878528
| | CASE03 | 147062784
| | CASE04 | 131334144
| | CASE05 | 95944704
| | CASE06 | 131334144
| | CASE07 | 109051904
| | CASE08 | 107741184
| | CASE09 | 107741184
| | CASE10 | 80478208
| | CASE11 | 92274688
| | CASE12 | 100663296
| | CASE13 | 76808192
| | CASE14 | 100139008
| | CASE15 | 133169152
| | CASE16 | 108003328
| | CASE17 | 115605504
| | CASE18 | 110886912
| | CASE19 | 63700992
| | CASE20 | 83623936
|
|
|
| Filename | Size (in bytes)
| | CASE21 | 140509184
| | CASE22 | 140509184
| | CASE23 | 176947200
| | CASE24 | 131334144
| | CASE25 | 139198464
| | CASE26 | 131334144
| | CASE27 | 92012544
| | CASE28 | 99876864
| | CASE29 | 98566144
| | CASE30 | 83361792
| | CASE31 | 77856768
| | CASE32 | 104071168
| | CASE33 | 102236160
| | CASE34 | 102498304
| | CASE35 | 124780544
| | CASE36 | 110886912
| | CASE37 | 130547712
| | CASE38 | 119275520
| | CASE39 | 69992448
| | CASE40 | 84934656
|
|
Airway tree segmentations should be a single 6-connected structure, containing at least 10% of the lower part of the trachea. If a submitted segmentation consists of several unconnected components, only the component that contains the trachea will the evaluated. The trachea is only needed as a starting point for extracting the measurements and is excluded from the actual measurements.
Algorithms can be submitted in two categories, which will be evaluated separately:
- Fully automated methods
As the name implies, the methods submitted under this category should not require any user interactions or manual parameter tuning on images of the testing set.
- Semi-automated methods
Methods submitted under this category can use any form of user interaction. Documentation of methods under this category must state clearly what kind of interactions are required, and an estimate of the number of interactions and the time needed to complete a segmentation should be provided. Note that the amount of interaction will not be used in the ranking of methods.
Note: Due to the time required for manual evaluation, we allow only results from a single method from each team of authors. If you would really like to enter the competition with two methods that are radically different, consult the organizers.
Evaluation of submitted results
All submitted segmentations are centrally evaluated by a team of trained observers.
The objective is to compare performance of different algorithms. For this purpose, a ground truth is constructed from all submitted segmentations and all submissions are subsequently evaluated with respect to this ground truth.
Evaluation of each individual segmentation is performed in the following steps:
- Airway segments or branches are extracted from submitted airway tree segmentations using a fast marching based algorithm (similar to Schlathölter, T., Lorenz, C., Carlsen, I. C., Renisch, S., Deschamps, T., 2002., "Simultaneous segmentation and tree reconstruction of the airways for virtual bronchoscopy." Vol. 4684. SPIE, pp. 103–113).
- Airway segments are evaluated visually on a set of extracted slices from both a reoriented view and a reformatted view with straightened airway centerlines.
- Each segment is scored as "correct" or "wrong", by at least two observers. The criterion used is whether the extracted airway segment indeed belongs to the airway tree; the exact airway shape and dimensions are not taken into account.
The ground truth is then defined as the union of all valid airway segments from all submitted segmentations.
The following measurements are computed and used for comparing the submitted results:
- Branch count
The number of branches that are detected correctly. A branch is considered detected as long as the length of the centerlines is more than 1 mm.
- Branch detected
The fraction of branches that are detected, with respect to the branches present in the ground truth.
- Tree length
The sum of the length of the centerlines of all correctly detected branches.
- Tree length detected
The fraction of tree length in the ground truth that is detected correctly.
- Leakage count
The number of unconnected groups of "correct" regions that are neighboring with a "wrong" region. Indicates how easy/difficult it is to manually separate leakages from the correctly detected branches.
- Leakage volume
The volume of regions that are wrongly detected.
- False positive rate
The fraction of the volume of regions that are detected wrongly over the volume of all detected regions.
Note: Trachea is excluded from the branch length and branch count related measurements. For the lekage based measures, both trachea and main bronchi are excluded.
A table similar to the one shown below will be provided. Participants are expected to include this table in the result section of their paper, which will be published at the 2nd International Workshop on Pulmonary Image Analysis. By default, only the LaTeX code for the table will be provided. For non LaTeX users, please contact the organizers at exact09@diku.dk
|
| |
|