The EMPIRE10 Challenge

This page provides in depth information about the EMPIRE10 challenge. It assumes you understand the basic concept already. For an overview description please see the home page.

If you are interested in participating please read all details carefully. If anything is not clear or you have additional questions please email Keelin.Murphy@radboudumc.nl .

Goal of the EMPIRE10 challenge

The main goal of the EMPIRE10 challenge is to evaluate the state of the art in chest CT registration. Although performance details of many algorithms are published, and indeed many algorithms are available for download, comparing their behaviour is not a trivial task. Authors typically use different datasets and different evaluation measures when publishing their performance results, making legitimate comparisons between their methods virtually impossible. It is possible to obtain several publicly available algorithms and compare them on a particular dataset, but many methods require in-depth knowledge and understanding in order to configure them in an optimal fashion. Such a comparison may therefore be flawed by incorrect configuration of one or more of the algorithms involved. EMPIRE10 therefore provides a unique opportunity to make legitimate comparisons between registration algorithms.

A summary article based on the initial phase of the challenge  is available: Murphy et al., "Evaluation of registration methods on thoracic CT: the EMPIRE10 challenge.", IEEE Trans Med Imaging. 2011 Nov;30(11):1901-20. Please cite this article if you reference EMPIRE10. Teams which participated in the initial phase of the challenge, including registering the workshop data are included as co-authors on this article. The article describes the challenge, a brief overview of each participating algorithm and a summary of the results. The challenge remains open, and teams are encouraged to submit improved results. Teams which did not participate in the initial phase are always welcome to submit new results also. In this way the EMPIRE10 website will continue to reflect the state of the art in registration of pulmonary CT images.

Rules

The EMPIRE10 challenge is organised in the spirit of cooperative scientific progress. We therefore ask anybody using this website to respect the rules below.

We do not claim any ownership or rights to the algorithms.

The following rules apply to those who register a team and download the data:

• The downloaded data sets or any data derived from these data sets, may not be given or redistributed under any circumstances to persons not belonging to the registered team.
• Data downloaded from this site may only be used for the purpose of preparing an entry to be submitted on this site. The data may not be used for other purposes in scientific studies and may not be used to train or develop other algorithms, including but not limited to algorithms used in commercial products.
• If the results of algorithms in this challenge are to be used in scientific publications (journal publications, conference papers, technical reports, presentations at conferences and meetings) you must make an appropriate citation. Murphy et al., "Evaluation of registration methods on thoracic CT: the EMPIRE10 challenge.", IEEE Trans Med Imaging. 2011 Nov;30(11):1901-20.
• Evaluation of registration results uploaded to this website will be made publicly available on this site (see the Results Section), and by submitting results, you grant us permission to publish our evaluation. Participating teams maintain full ownership and rights to their method.
• Teams must notify the organisers of EMPIRE10 about any publication that is (partly) based on the results data published on this site, in order for us to maintain a list of publications associated with the challenge.

Data

The data to be downloaded consists of 30 pairs of chest CT scans. Each pair of scans is taken from a single subject, in other words no inter-subject registrations are included. The scans come from a variety of sources and are provided by several different institutes. Scans may be taken at various phases in the breathing cycle (full inspiration, full expiration, phase from 4D breathing data). Subjects may exhibit lung disease or appear healthy. Data from a variety of scanners is included and a variety of voxel sizes occur. In this way we have a included broad range of data, encompassing many of the problems faced by researchers developing registration algorithms for this application.

In addition to the CT data, binary lung masks are provided for each scan. These may be used by participants in registering the scans if they wish to do so. The lung masks are automatically generated using an algorithm by van Rikxoort et al. [1]. They have been visually checked and manually corrected where necessary.

The data is available on the download page. It is split into several files to prevent the file size becoming problematic for some users. The downloaded data files are in .zip format and may be unzipped using any suitable program.

All data (both scans and lung masks) is in Meta format. This format stores an image as an ASCII readable header file with extension .mhd and a separate binary file for the image data with extension .raw. This format is ITK compatible; documentation is available here. An application that can read the data is SNAP. If you want to write your own code to read the data, note that in the header file you can find the dimensions of the scan, the voxel spacing and the pixel type (short for the CT scans and unsigned char for the binary lung masks). In the raw file the values for each voxel are stored consecutively with index running first over x, then y, then z.

All scans have origin at (0,0,0), center of rotation at (0,0,0) and anatomical orientation in the RPI frame

Please note that the scans provided are cropped from original images such that regions outside the lung volume are excluded where possible (i.e. cropped using a lung bounding box). This was done to make the data to be downloaded smaller and because registration outside the lung volumes will not be considered. Information on the cropping coordinates may be found here.

The unzipped data is divided into two folders, "scans" and "lungMasks". Within these folders, each filename includes an ID for the subject in question, followed by either "_Fixed" or "_Moving". e.g. 01_Fixed, 01_Moving, 02_Fixed, 02_Moving ... etc. The filenames imply the registration that is to be carried out therefore it is important to understand them correctly. In this case 01_Fixed and 01_Moving are a pair of scans (from subject number 01) to be registered, with 01_Fixed being the fixed (or target) image, and 01_Moving being the moving (or source) image.

Performing Registration

Registration should be carried out using your software with parameter settings that you consider optimal. The use of lung masks or other masks which you may wish to derive is permitted. Please note that evaluation will be based on the registration of the lung volume only therefore it is not necessary to consider the alignment of other surrounding tissue and structures.

Please make careful note of the parameter settings and configurations that are used in your registration as you will be required to report these.

Three categories of algorithm are defined: [From 2019 onwards only Fully Automatic entries are allowed]

• Fully Automatic (Uses the same parameters (or determines parameters automatically) on all scan pairs. Requires no user interaction)
• Semi-Automatic (Parameter settings were changed manually for different scan pairs. The total number of parameter settings used must be reported.)
• Interactive (User interaction such as manual alignment, defining corresponding point pairs etc. was required for 1 or more of the scan pairs processed.)

  Please notify Keelin.Murphy@radboudumc.nl if you wish to submit a method which is not in the Automatic category

  Submitting Results

  Please read this section very carefully before you submit your results. The organisers reserve the right not to evaluate data which is submitted in a format other than that described here.

  For each scan pair that you register you must submit a deformation field which contains displacement information. A deformation field, in this case, consists of three 3-D images, defX, defY and defZ, each one with the same dimensions as the fixed image in the registration pair. The images should be of type floating point (MET_FLOAT), and in the same Meta format as the data which you downloaded from this site. The values contained in the deformation field images should be displacement distances (in mm) in the X, Y and Z directions respectively. For additional information about how we define a deformation field please see here. An example output set can be downloaded on the downloads page.

  Your deformation images for each registration pair should be placed in a folder with the ID of the subject. For example, after registering the scans 01_Fixed and 01_Moving, the resulting deformation images should be placed into a folder named "01". The deformation images MUST have the names "defX.mhd", "defY.mhd" and "defZ.mhd" (with the appropriate raw file accompanying each one).

  Your submission should consist of the folders 01-30 as described above.

  To submit please see the submit page. Don't forget to prepare a pdf document describing your method in addition. The description should be sufficient to give a good understanding of how the method works. A checklist of items we would suggest you include is given below:

  • Rigid/Affine/Elastic components to the registration algorithm
  • Similarity measure(s) used
  • Lung masks (or any other anatomical information) used
  • Model of transformation (e.g. B-Spline, Optical flow... )
  • Optimization method used
  • Other details of the implementation, eg. multi-resolution strategy, iterative procedure, stopping conditions
  • Any parameter settings or other configuration information which would be required for a third party to re-implement your registrations
  • Hardware used and average time taken to register one pair of scans.
  • Strong points and limiting factors of your algorithm. Does it perform optimally on certain types of data and not so well on others? Are there parameter settings which one might consider altering to improve the performance on some particular scan-pairs?

   

  Evaluation

  Details of the evaluation methods used and the scores/rankings produced can be found on the evaluation page. Results for evaluated algorithms will appear on the results page.

  How often can I submit results?

  It is possible to re-submit results as often as you wish, bearing in mind that submission and evaluation of the files may take some time. New submissions will overwrite previous ones.

  Currently we do not offer the possibility for teams to remove submitted results. If you believe there are good reasons to remove certain results that you have submitted, please contact Keelin.Murphy@radboudumc.nl .
   

  [1]"Automatic lung segmentation from thoracic CT scans using a hybrid approach with error detection." E.M. van Rikxoort, B. de Hoop, M.A. Viergever, M. Prokop, B. van Ginneken.
  Medical Physics 36(7) (2009)