Good opening week + some initial clarifications
We have had a good opening week of the TUPAC16 challenge. Around 40 university research groups, companies and individuals have already registered to participate. With the first registered participants we have also received the first feedback and questions regarding the challenge. Some of the answers I have provided are given below.
I think that it is very important to separate the entries which are using additional training data.
We agree. The methods that use additional training data will be ranked separately with a clear specification of the external dataset that they used.
We need to fit a predictor to a response. Which response do we use for fitting - mitosis counting or molecular proliferation score? The problem is that those 2 scores almost not correlated.
These are two separate tasks. You can chose to model either or both of the responses. I would not say that thy are not correlated at all, but the agreement is definitely not perfect. The two scores aim to measure the same "quantity", however the methods are different. The scores for Task 1 are based on mitosis counting, so we know that the information is in the images. The scores for Task 2 are based on molecular data, so there is an implicit hypothesis that the same information (or a part of it) is contained in the image data. This can be the mitotic figures, but also in other tissue morphology features.
What metric for fitting the predictor to the response will you use for evaluation? (AUC, Accuracy, Specificity, Sensitivity, Predictive Value of the Test, etc.)
For Task 1 you need to provide scores and we will use the quadratic weighted kappa to perform the evaluation. The responses for Task 2 are continuous and you will need to also provide continuous predictions. We will use the Spearman's correlation coefficient for evaluation of this task, so the scale and range of the prediction is not important (only that they have a monotonic relationship with the ground truth).
In which format should we submit the data? (do you need probability column to calculate ROC curves, or you just need predicted labels to calculate confusion table, or both).
The results should be submitted in a .csv file format with each row corresponding to one case. We will not have to submit probability values, just the predictions.
Do we need to provide scoring for whole slides , for TIFF images or both?
Youl need to provide scoring for whole slide images. The TIFF images with mitotic figures are provided only as an auxiliary dataset.