A research team, based at the Pacific Northwest National Laboratory (PNNL), published the results of an international survey designed to assess the relative importance climate scientists assign to variables when analyzing a climate model’s ability to simulate real-world climate.

The results, which have serious implications for studies using the models, were published in Advances in Atmospheric Sciences (AAS), published by Springer and hosted by the Institute of Atmospheric Physics (IAP) of the Chinese Academy of Sciences.

In assessing climate models, experts typically evaluate across a range of criteria to arrive at an overall evaluation of the model’s fidelity. They use their knowledge of the physical system and scientific goals to assess the relative importance of different aspects of models in the presence of trade-offs. (Image by AAS) 

"Climate modelers spend a lot of effort on calibrating certain model parameters to find a model version that does a credible job of simulating the Earth’s observed climate," said Susannah Burrows, first author of the paper and a scientist at the PNNL who specializes in Earth systems analysis and modeling.

However, Burrows noted, there is little systematic study on how experts prioritize such variables as cloud cover or sea ice when judging the performance of climate models.

"Different people might come to slightly different assessments of how 'good' a particular model is, depending to large extent on which aspects they assign the most importance to," Burrows said.

One model, for example, may better simulate sea ice while another model excels in cloud simulation. Each scientist must strike a balance between their competing priorities and goals-a difficult thing to capture systematically in data analysis tools.

"In other words, there isn’t a single, completely objective definition of what makes a 'good' climate model, and this fact is an obstacle to developing more systematic approaches and tools to assist in model evaluations and comparisons," Burrows said.

The researchers found, from a survey of 96 participants representing the climate modelling community, that experts took specific scientific objectives into consideration when rating variable importance. They found a high degree of consensus that certain variables are important in certain studies, such as rainfall and evaporation in the assessment of the Amazonian water cycle. That agreement falters on other variables, such as how important it is to accurately simulate surface winds when studying the water cycle in Asia.

Understanding these discrepancies and developing more systematic approaches to model assessment is important, according to Burrows, since each new version of a climate model must undergo significant evaluation, and calibration by multiple developers and users. The labor-intensive process can take more than a year.

The tuning, while designed to maintain a rigorous standard, requires experts to make trade-offs between competing priorities. A model may be calibrated at the expense of one scientific objective in order to achieve another.

Burrows is a member of an interdisciplinary research team at PNNL working to develop a more systematic solution to this assessment problem.

To help climate modelers understand these trade-offs more clearly and efficiently, the visualization researchers are building interactive, intuitive visual interfaces that allow modelers to summarize and explore complex information about different aspects of model performance.

The data scientists are working to characterize expert climate model assessment in greater detail, building on the findings from the initial survey. Eventually, the researchers aim to blend a combination of metrics with human expertise to assess how well-suited climate models are for specific science objectives, as well as to predict how frequently experts will agree or disagree with that assessment.

"We plan to combine the best of both worlds, using computing to reduce manual effort and allowing scientists to more efficiently apply their human insight and judgment where it is most needed," Burrows said.