Evaluation Metrics
The main goal is to be able to tell what is the mitotic count on each image. A segmented mitosis would be counted as correctly detected if its centroid is localised within a range of 8 μm from the centroid of a ground truth mitosis.
Centroid-based Measures
The centroid-based measures deal with the number of objects detected. As the main goal of the contest is to give the correct mitotic count, the centroid-based measures are the most important ones. These measures are defined as follows:
- D = number of mitosis detected (centroid within a range of 5μm from the centroid of a ground truth mitosis)
- TP = number of True Positives, that is the number of mitosis that are ground truth mitosis among the D mitosis detected
- FP = number of False Positives, that is the number of mitosis that are not ground truth mitosis among the D mitosis detected
- FN = number of False Negatives, that is the number of ground truth mitosis that have not been detected
- recall (sensitivity) = TP / (TP+FN)
- precision (positive predictive value) = TP / (TP+FP)
- F-measure = 2 * (precision * sensitivity) / (precision + sensitivity)
- localisation quality: mean and standard deviation of the distance between detected and ground truth centroids
Region-based Measures
The region-based measures deal with the surface of the objects. Only segmented mitosis considered as correctly detected (that is having their centroid localised within a range of 5μm of the centroid of a ground truth mitosis) are considered for region-based measures. These measures are defined as follows:
- overlap = |A(S)∩A(G)| / |A(S)∪A(G)|
- recall (sensitivity) = |A(S)∩A(G)| / |A(G)|
- specificity = |N −A(S)∪A(G)| / |N −A(G)|
- precision (positive predictive value) = |A(S)∩A(G)| / |A(S)|
- F-measure = 2 * (precision * recall) / (precision + recall)
where N is the total number of pixels in an image, A(S) is the area of the segmented mitosis, A(G) is the area of the ground truth mitosis, |A(S)| is the number of pixels of the area of segmented mitosis, |A(G)| is the number of pixels of the area of ground truth mitosis.
Ranking of Contestants
The contestants will be ranked according to the following 3 centroid-based measures:
- F-measure
- recall (sensitivity)
- precision (positive predictive value)
For each measure, the 3 best contestants are kept and given a medal (gold, silver or bronze). The contestants will be ranked according to the number of medals they collected.
If there are contestants with equal number and value of medals, the localisation quality (mean and standard deviation of the distance between detected and ground truth centroids) will make the difference between them.