Machine learning has been increasingly utilized to optimize the use of brain imaging data in clinical classification and to build predictive models for patients with schizophrenia. Assessing generalizability is one of the most important steps in evaluating predictive models but is surprisingly seldom conducted in clinical studies.

To bridge this gap of knowledge, Dr. Raymond Chan from the Institute of Psychology of the Chinese Academy of Sciences, collaborating with the international researchers, conducted a study to specifically examine generalizability of machine learning for schizophrenia classification based on resting-state imaging data.

Both internal validation and external validation were used to assess within-site and between-site generalizability.

They recruited two independent samples to achieve this aim, including 51 patients with schizophrenia and 51 healthy controls as the main data set and 34 patients with schizophrenia and 27 healthy controls as the validated data set. The resting-state imaging data and T1 structural data were acquired for all participants.

They established the within-site generalizability in the main data set and achieved an accuracy of 0.73, trained a model in the main data set, investigated between-site generalization in the validated data set, and found a classification accuracy of 0.55 (not achieving significant level). Finally, they recognized the poor between-site generalization performance.

They also updated the unsupervised algorithm by using additional unlabeled data for assessing the between-site generalizability and found and accuracy of 0.70 (at the significant level of 0.05 by permutation test).

These findings highlight the need to incorporate efforts to facilitate application of machine learning across multiple data sets. The improvement in performance with transfer learning also highlights the importance of taking into account the properties of data when constructing predictive models across samples and sites. Machine learning classification result based on a single study should be interpreted cautiously.

This work was supported by a grant from the National Key Research and Development Programme, the National Natural Science Foundation of China, the Beijing Municipal Science & Technology Commission Grant, and the CAS key Laboratory of Mental Health, Institute of Psychology.

The paper is available online from Human Brain Mapping.