This study presents a ranking for classifers using a Bayesian perspective. This ranking framework is able to evaluate the performance of the models to be compared when they are inferred from different sets of data. It also takes into account the performance obtained on samples not used during the training of the classifiers. Besides, this ranking assigns a prior to each model based on a measure of similarity of the training data to a test case. An evaluation consisting of ranking brain tumour classifiers is presented. These multilayer perceptron classifiers are trained with 1H magnetic resonance spectroscopy (MRS) signals following a multiproject multicenter evaluation approach. We demonstrate that such a framework can be effectively applied to the real problem of selecting classifiers for brain tumour classification.