2. Methods

(A) Creation of the Smoke and Emissions Model Intercomparison Project (SEMIP)

Initially, we will codify the Smoke and Emissions Model Intercomparison Project (SEMIP).  A detailed list of standards, instructions for inclusion, test-case scenarios, collected data sets, evaluation metrics, analysis procedures, and processes for modifying the project at a later date will be documented.  Before proceeding, this document will be presented to a science advisory board, then to the wider research community for comment and review, and finally to the JFSP governing board for approval.  SEMIP will be open source and open access.

Creation of such an intercomparison project is aided by the example of previous and ongoing model intercomparison projects in other fields (see Section I.3, Background).  The project team has experience with past model intercomparison projects (e.g., NARSTO, 1999; Wheeler and Roney, 2001; Wheeler, 2003) and model performance evaluation studies that follow EPA guidance (EPA, 2007).  In addition, advice will be sought from several sources with experience in these projects, including AMIP with which the authors have some ties.  SEMIP will be structured similarly to these previous highly successful projects.

Because not every model produces output at each step, the SEMIP framework will be designed so that comparisons can be carried out at a variety of process step levels (Figure 2).  This design also allows the inclusion of component models that have not been incorporated into larger modeling systems.

While the final list of test cases, variables, performance metrics, and observation data sets will be determined as part of the proposed project with approval from the science advisory panel and JFSP board, they are likely to be similar to those outlined below.

Test Cases

Test cases are the specific fire events, episodes, and seasonal summaries that will be modeled by each model in SEMIP.  Some models are expected to perform best under specific circumstances; therefore, the test cases will be chosen to cover a range of geographic locations, seasons, burn severities, and fuel types.  Test cases to be considered include

• a large wildland fire use (WFU) fire in complex terrain in Idaho,

• a large wildfire under high wind conditions in Southern California,
• an understory prescribed burn in the Southeast,
• widespread rangeland fires in the Great Plains, and
• the annual national wildfire smoke emission inventory for a specific year (2002 or 2005).

The specific test cases chosen will be based on the availability of observational data and scientific questions surrounding the case.  For each case, meteorological model output and fire information will be gathered and published as input data sets to standardize model runs.  In addition to the specific test cases that will be identified to initiate the SEMIP, the science advisory board will approve a protocol for adding new test cases and input data sets as they become available.  In this way, as more detailed comparisons lead to new questions, SEMIP can be altered to remain relevant.  

Variables and Performance Metrics

Identified test cases will be evaluated in two ways:  an inter-model comparison, and a model-observation comparison.  The variables that each SEMIP model will compute will be compared to observation data using specific defined performance metrics.  Each model output step will have a SEMIP defined list of variables.  For example, for smoke impacts, variables might include

• hourly pollutant concentration PM2.5 from smoke,
• daily 24-hr average and maximum concentration PM2.5,
• domain maximum concentration PM2.5, and 
• NAAQS threshold exceedances. 
  

At each step, performance metrics will include, as appropriate,

• mean, mean bias, and standard deviation,
• threshold hits, misses, and false alarms, and
• spatial pattern statistics (e.g., spatial correlation, principal component analysis).
  

In addition to these metrics, other evaluation techniques will be explored to ensure that potential compensating internal errors do not exist or are minimized.

Observations

An extensive observational database will be prepared for each episode with ground, satellite, and airborne data being gathered and archived in uniform databases.  Additionally, meteorological model output and fire information required to simulate the test cases will be archived. Observation data sets to be considered for model-observation validation include

• routine monitoring networks (AIRNow, AQS, IMPROVE, STN, PAMS, NAATS),
• satellite aerosol optical depth (MODIS, MISR, OMI, VIIRS, Glory APS, Gas/NWS),
• vertical profile smoke data (CALIPSO, ELF, MISR, ground LIDAR),
• routine met networks (MADIS, RAWS, ASOS) and model data (NAM, WRF/MM5, others),
• special studies data (Frank Church, Flint Hills, Yosemite, other JFSP funded, others)

Once approved, the test case data sets and observations will be collected and distributed to the research community for open access via a SEMIP data warehouse web site.  The data warehouse will also include information on the quality, representativeness, and uncertainties associated with the observational data.  Databases (both SQL based and NetCDF based) will be prepared that will contain any submitted model results, enabling SEMIP to accept model output data for evaluation even from proprietary models.

(B) Evaluation of models (Phase 1 of SEMIP)

Once SEMIP becomes active, a substantial evaluation process, related to the overall project, will be carried out and considered SEMIP Phase 1.  It is expected that even after completion of Phase 1, SEMIP will continue to evolve and provide insights as new models are developed or existing models modified.

Phase 1 evaluation will be as comprehensive as possible, with the goal of encompassing all publicly available models.  These include all the major smoke prediction systems and emissions inventory systems listed in Section I.3 (Background), and the component models listed in Figure 2.  Models not listed will still be included if the appropriate model results are submitted to the project. 

For all models identified, five test cases will be evaluated.  The evaluation portion of the proposed work will consist of four steps:

• expansion of the BlueSky framework to support the specific models,
• execution of the models for each test case,
• computation of model-observation and intercomparison statistics, and
• analysis and reporting.

Expansion of the BlueSky framework

Cross-comparison and evaluation of these many models is an ambitious undertaking and considering them in light of this project is only possible because of significant prior work.  The BlueSky smoke modeling framework was recently professionally rewritten via a grant from NASA to be completely modular and extensible, enabling new component models to be incorporated easily and quickly.  BlueSky is not a “smoke model” because many combinations of component models are possible and because the framework can be started and stopped at any point.  We make use of that flexibility here.  Much of the evaluation can utilize the BlueSky framework, facilitating cross-comparison.  Many models listed have been linked to the BlueSky framework or are in the process of being linked.  Other models will be added for this project; specific models to be incorporated in this way are FOFEM, FRP, FLAMBE, and HYSPLIT.  Some models (e.g. PB-PIEDMONT, WRF-Chem, VSMOKE, Daysmoke) will still need to be run separately.  All the models listed are known to be available through willing collaborators at the USFS, NOAA, NCAR, several universities, and the Monterey Naval Research Laboratory.  

Execution of the models

Each model to be tested will be executed for each applicable test case.  For model steps that are gridded (e.g. smoke impacts), runs will be carried out at several grid resolutions from 1 km to 36 km.  Models usually run at other resolutions will be asked to run at these resolutions or to provide data interpolated to one or more of these resolutions for comparison.  All model output collected will be published in the SEMIP data warehouse.

Computation of model-observation and intercomparison statistics

The applicable variables and performance metrics laid out in the approved SEMIP study plan will be computed for each test case and model.  Results will be made available on the SEMIP web site.  Because all available models will be analyzed the same way, direct comparison of metric differences between models should reveal details on real-world model performance and those models that perform best and under what conditions will be identified.  Only some variables will be evaluated initially; e.g., smoke impacts will be evaluated for fine particulate matter (PM2.5) emissions and ambient concentrations initially.  Other pollutants related to smoke (ozone, PM10, VOCs, CO, toxics) can be added in later phases.  

Analysis and reporting

The results of SEMIP Phase 1 are expected to produce many outcomes of value to both the scientific community and the model-user community, such as how well specific models predict smoke and emissions under different conditions, which portions of the model chain are the most uncertain, and where future validation and development efforts should be focused.  The SEMIP and evaluation results will be documented for submission to a variety of journals and presented at a variety of conferences and meetings, as detailed in the Deliverables and Science Application and Delivery sections.  All SEMIP data sets, including model output collections will be published online on the SEMIP web site.