Test Cases
Identifying the Initial Test Cases
How and why the initial test cases were selected.
SEMIP is built around a number of test cases for which intercomparison will be carried out.
The SEMIP intercomparison should ultimately reflect all the usage cases to which these models are applied. Ideally the intercomparison should weight its results to the typically usage of the models. Unfortunately, these goals must yield to practical considerations such as data availability.
Test case considerations
Ideally the SEMIP test cases should reflect a range of:
- Fire types (e.g., wildfire, prescribed fire)
- Fire size
- Combinations of fires (e.g., regional event, single fire)
- Fuel and vegetation types
- Geographic regions
- Climatic regimes
- Meteorological conditions
Other considerations also include terrain, season, and potential impact on population centers.
How the Initial Test Cases were selected
The initial test cases were selected through a three-step process, first a brainstorm session that listed all potential test cases, second this list was paired to a set of cases that represent a range unique problems and scenarios, and finally the list was again paired down to those test cases that have the comprehensive input and observation data available for the model runs and comparison analyses.
Based on the considerations above, we generated a list of potential test cases from several different perspectives. Each fire was then listed by region and fuel type. Some fires were identified by several different approaches. Only North American fires were considered due to the desire to focus the initial study on U.S. land management needs. Only fires in the last 20 years are considered due to data quality and availability issues.
The brainstormed list was done from the following perspectives:
- Regional
Identification of the most significant fires and fire data resources by region.
- Impacts
Identification of the fires with the largest impacts (landscape changes, visibility, and/or National Ambient Air Quality Standard (NAAQS) exceedences.
- Historic
The most historic fires subjectively selected based on popular media, policy impacts, and scientific importance.
- Data
The fires with the most observational data overall across all modeling steps.
Added to this list were prescribed burns, both collected throughout a region or individually.
Potential test cases included a wide array of fires and dates. The initial test cases were selected from this list based primarily on:
- data availability
- balance across regions, fire types, vegetation, etc...
Despite this, and unavoidably, we recognize that these initial test cases have serious limitations, in geographic location, time and space scales, and other ways. See:
Initial Test Cases
Descriptions of the 8 test cases can be found here.
The initial test cases represent:
- both fully comprehensive but not realistic cases (Fires Everywhere) and fairly comprehensive but realistic cases (National Emissions Inventory);
- both wildfires and prescribed burns;
- both national scale (National cases) and regional scale (California, Northwest, Southeast); and
- both collected fires across a region (California, Northwest, Southwest) and individual fires (Bugaboo, Tripod, Naches).
They also represent cases with emphases on different parts of the modeling chain. Some test cases only go through Emissions, while others go all the way through Dispersion. Some test cases are designed in part to compare the fire information systems, while others take the fire information as a given. See the individual test case pages for more information.
The Initial Test Cases were designed to try to maximize their applicability across the country and across typical model uses. Effort has been made to cross temporal and spatial scales, and to maximize the amount of observational data that can be utilized for evaluation and validation (characteristics of the test cases is listed below).
Unfortunately, despite best efforts, the test cases as selected also have significant limitations, imposed primarily because of the need to maximize data availability in selecting them.
Geographic limitations
The Initial Test Cases are limited primarily in geographic region.
There are currently no test cases for the inland Southwest, core Rocky Mountain region, Midwest, and Northeast. The Midwest and Northeast are historically limited in fire data, in part because of the smaller size and shorter duration of fires compared with the West, and the relatively less well developed prescribed fire reporting systems. The Southwest and Rockies have had a number of fires that would make good potential test cases, but these were eliminated in favor of other cases that have happened in the past couple of years (e.g. the California Wildfires case), which allow for more robust, satellite data and the use of known rapid response research data. Fortunately for SEMIP, if not for the Southwest and Rockies, large fire episodes in any given geographic area tend to be cyclic, and the next such episode is likely to generate even more data than the recent California fires.
We are actively seeking out fires with good satellite and ground measurement data that can fill these geographic gaps. If you have suggestions, please contact us.
Additional test cases
In part because any list of test cases is limited and incomplete, the test case list should be dynamic. Additional test cases should be added over time as the science and technology push the field towards further understanding and new science questions, and new input and observational data become available. Similarly, older test cases that no longer are needed or useful can be dropped. SEMIP is set up to handle this through a process where such additions and deletions are submitted to the JFSP Board for approval. Test cases are re-evaluated once a year.
Upcoming
Several JFSP projects underway have the promise of significantly improving our information in several key areas including plume rise. These projects may result in additional SEMIP test cases as their data gets collected and becomes available. SEMIP is working with the PIs in question to enable this transfer of information.
