title: Ecosystem Model Skill Assessment. Yes We Can!
abstract: Accelerated changes to global ecosystems call for holistic and integrated analyses of past, present and future states under various pressures to adequately understand current and projected future system states. Ecosystem models can inform management of human activities in a complex and changing environment, but are these models reliable? Ensuring that models are reliable for addressing management questions requires evaluating their skill in representing real-world processes and dynamics. Skill has been evaluated for just a limited set of some biophysical models. A range of skill assessment methods have been reviewed but skill assessment of full marine ecosystem models has not yet been attempted.
  We assessed the skill of the Northeast U.S. (NEUS) Atlantis marine ecosystem model by comparing 10-year model forecasts with observed data. Model forecast performance was compared to that obtained from a 40-year hindcast. Multiple metrics (average absolute error, root mean squared error, modeling efficiency, and Spearman rank correlation), and a suite of time-series (species biomass, fisheries landings, and ecosystem indicators) were used to adequately measure model skill. Overall, the NEUS model performed above average and thus better than expected for the key species that had been the focus of the model tuning. Model forecast skill was comparable to the hindcast skill, showing that model performance does not degenerate in a 10-year forecast mode, an important characteristic for an end-to-end ecosystem model to be useful for strategic management purposes.
  We identify best-practice approaches for end-to-end ecosystem model skill assessment that would improve both operational use of other ecosystem models and future model development. We show that it is possible to not only assess the skill of a complicated marine ecosystem model, but that it is necessary do so to instill confidence in model results and encourage their use for strategic management. Our methods are applicable to any type of predictive model, and should be considered for use in fields outside ecology (e.g. economics, climate change, and risk assessment).
authors.affiliation: Institute of Marine Research, PB 1870 Nordnes, N-5817, Bergen, Norway.|NOAA Northeast Fisheries Science Center, 166 Water St., Woods Hole, Massachusetts, 02543-1026, United States of America.
authors.forename: Erik
authors.initials: E
authors.lastname: Olsen
authors.affiliation: School for Marine Science and Technology, University of Massachusetts Dartmouth, 200 Mill Road, Fairhaven, Massachusetts, 02719, United States of America.
authors.forename: Gavin
authors.initials: G
authors.lastname: Fay
authors.affiliation: NOAA Northeast Fisheries Science Center, 166 Water St., Woods Hole, Massachusetts, 02543-1026, United States of America.
authors.forename: Sarah
authors.initials: S
authors.lastname: Gaichas
authors.affiliation: NOAA Northeast Fisheries Science Center, 166 Water St., Woods Hole, Massachusetts, 02543-1026, United States of America.
authors.forename: Robert
authors.initials: R
authors.lastname: Gamble
authors.affiliation: NOAA Northeast Fisheries Science Center, 166 Water St., Woods Hole, Massachusetts, 02543-1026, United States of America.
authors.forename: Sean
authors.initials: S
authors.lastname: Lucey
authors.affiliation: NOAA, National Marine Fisheries Service, 166 Water Street, Woods Hole, Massachusetts, 02543, United States of America.
authors.forename: Jason S
authors.initials: JS
authors.lastname: Link
mesh_terms: D018533:Biomass; D057231:Climate Change; D017753:Ecosystem; D005398:Fisheries; D006801:Humans; D008962:Models, Theoretical; D014481:United States