What is the "best" model? The answer to this question lies in part in the eyes of the beholder, nevertheless a good model must blend rigorous theory with redeeming qualities such as parsimony and quality of fit. Model selection is used to make inferences, via weighted averaging, from a set of K candidate models, (Formula presented.), and help identify which model is most supported by the observed data, (Formula presented.). Here, we introduce a new and robust estimator of the model evidence, (Formula presented.), which acts as normalizing constant in the denominator of Bayes theorem and provides a single quantitative measure of relative support for each hypothesis that integrates model accuracy, uncertainty, and complexity. However, (Formula presented.) is analytically intractable for most practical modeling problems. Our method, coined GAussian Mixture importancE (GAME) sampling, uses bridge sampling of a mixture distribution fitted to samples of the posterior model parameter distribution derived from MCMC simulation. We benchmark the accuracy and reliability of GAME sampling by application to a diverse set of multivariate target distributions (up to 100 dimensions) with known values of (Formula presented.) and to hypothesis testing using numerical modeling of the rainfall-runoff transformation of the Leaf River watershed in Mississippi, USA. These case studies demonstrate that GAME sampling provides robust and unbiased estimates of the evidence at a relatively small computational cost outperforming commonly used estimators. The GAME sampler is implemented in the MATLAB package of DREAM and simplifies considerably scientific inquiry through hypothesis testing and model selection.

Volpi, E., Schoups, G., Firmani, G., Vrugt, J.A. (2017). Sworn testimony of the model evidence: Gaussian Mixture Importance (GAME) sampling. WATER RESOURCES RESEARCH, 53(7), 6133-6158 [10.1002/2016WR020167].

Sworn testimony of the model evidence: Gaussian Mixture Importance (GAME) sampling

Volpi, Elena
;
Firmani, Giovanni;
2017-01-01

Abstract

What is the "best" model? The answer to this question lies in part in the eyes of the beholder, nevertheless a good model must blend rigorous theory with redeeming qualities such as parsimony and quality of fit. Model selection is used to make inferences, via weighted averaging, from a set of K candidate models, (Formula presented.), and help identify which model is most supported by the observed data, (Formula presented.). Here, we introduce a new and robust estimator of the model evidence, (Formula presented.), which acts as normalizing constant in the denominator of Bayes theorem and provides a single quantitative measure of relative support for each hypothesis that integrates model accuracy, uncertainty, and complexity. However, (Formula presented.) is analytically intractable for most practical modeling problems. Our method, coined GAussian Mixture importancE (GAME) sampling, uses bridge sampling of a mixture distribution fitted to samples of the posterior model parameter distribution derived from MCMC simulation. We benchmark the accuracy and reliability of GAME sampling by application to a diverse set of multivariate target distributions (up to 100 dimensions) with known values of (Formula presented.) and to hypothesis testing using numerical modeling of the rainfall-runoff transformation of the Leaf River watershed in Mississippi, USA. These case studies demonstrate that GAME sampling provides robust and unbiased estimates of the evidence at a relatively small computational cost outperforming commonly used estimators. The GAME sampler is implemented in the MATLAB package of DREAM and simplifies considerably scientific inquiry through hypothesis testing and model selection.
2017
Volpi, E., Schoups, G., Firmani, G., Vrugt, J.A. (2017). Sworn testimony of the model evidence: Gaussian Mixture Importance (GAME) sampling. WATER RESOURCES RESEARCH, 53(7), 6133-6158 [10.1002/2016WR020167].
File in questo prodotto:
File Dimensione Formato  
Volpi_et_al-2017-Water_Resources_Research.pdf

accesso aperto

Descrizione: Articolo
Tipologia: Versione Editoriale (PDF)
Licenza: DRM non definito
Dimensione 1.75 MB
Formato Adobe PDF
1.75 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11590/327780
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 30
  • ???jsp.display-item.citation.isi??? 29
social impact