Master's thesis: Confusion Matrix Test
Is there an answer to mapping old growth?
An examination of two projects conducted with remote sensing and GIS
Robert A. Norheim
The primary test that I performed was to generate a confusion matrix for each forest (Table 5-5). The two projects mapped different categories, as described in Chapter Three. Thus, each confusion matrix lists the three different "Ancient Forest" types that Morrison distinguished (old growth, other ancient forest, high elevation ancient forest) in columns and two different old growth types (low and high elevation) as mapped by Pacific Meridian Resources. There is some question as to whether to count Morrison's "other ancient forest" type as "old growth" in Pacific Meridian's typology. Table 5-6 displays the effect on the overall agreement rates of listing other ancient forest agreeing with low elevation old growth, low or high elevation old growth, or not old growth in Pacific Meridian's scheme. Table 5-7 is a simplification of Table 5-5, with all ancient forest and old growth categories combined into a single category. The maps correspond to the classification in Table 5-7.
The four detailed confusion matrices (Table 5-5) are all quite different, although some trends are evident. Morrison classified between 10 and 17 percent of each forest as low elevation old growth, between 8 and 16 percent as other ancient forest, and between 2 and 8 percent as high elevation old growth. Pacific Meridian Resources had much broader ranges, mapping between 22 and 46 percent of each forest as low elevation old growth and 2 to 14 percent as high elevation old growth. The fact that the ranges are narrow or broad is not necessarily a reflection of accuracy or consistency, however, as the proportion of old growth may indeed vary significantly across the different forests.
Misclassifications between low and high elevation old growth are generally quite small; there are some discrepancies in the DEMs used, or at least in how they were processed and used in the projects, but the differences do not appear to be significant.
Eight to twelve percent of each forest is classified by both studies as low elevation old growth. On the other hand, the amount of forest where both agree that there is no old growth or ancient forest ranges from 40 to 62 percent. Pacific Meridian classified 69 to 71 percent of Morrison's low elevation old growth also as old growth. There is a similar strong agreement (75 to 76 percent) on high elevation old growth on three of the four forests, although the amount of high elevation old growth on the fourth Forest (the Olympic) is quite small. However, given the much greater acreage that Pacific Meridian reported as old growth for each forest, it is surprising that they did not classify as old growth all of the forest that Morrison classified as true old growth. One might at least have expected a higher rate of agreement on forests where Pacific Meridian found significantly more old growth than The Wilderness Society did, but this is not the case.
One might expect that Pacific Meridian would classify less of Morrison's other ancient forest category as old growth than of Morrison's old growth category, given the looser definition Morrison applied to the "other" category. If both projects were truly working from the same definition, then Morrison's "other" category should not be classified as old growth by Pacific Meridian. On the other hand, given the larger amount of old growth that Pacific Meridian found on each forest, one would think that Morrison's "other" category would typically be classified by Pacific Meridian as old growth. Indeed, Pacific Meridian classified 68 to 70 percent of the "other ancient forest" as old growth on the Gifford Pinchot and Mt. Hood National Forests - very similar to the agreement rate on low elevation old growth. However, the equivalent rates on the Olympic and Mt. Baker-Snoqualmie National Forests are only 35 and 50 percent. This discrepancy is hard to interpret; Pacific Meridian's classification of the "other" stands should not vary across forests. However, given that Morrison actually found more ancient forest on the Olympic than Pacific Meridian found old growth, the 35 percent rate of agreement on "other" is less surprising.
The figures in Table 5-5 show that Pacific Meridian found significant amounts of old growth that Morrison did not find, not a surprising conclusion given the discrepancies between the two studies as summarized in Table 5-1. For low elevation old growth, about one quarter of the old growth that Pacific Meridian found was found to not be ancient forest by Morrison on the Olympic and Mt. Baker-Snoqualmie National Forests. On the Gifford Pinchot and Mt. Hood National Forests, where the overall acreage found by Pacific Meridian is significantly higher than The Wilderness Society's figures, this proportion rises to over one half. For high elevation old growth, the figures are more consistent across the four forests; Morrison agreed with only 46 to 62 percent of the acreage that Pacific Meridian found.
Overall agreement rates are shown in Table 5-6. Four different rates are given. The first row does not count Morrison's "other ancient forest" category at all, and is simply the agreement rate for the three categories that both projects mapped (old growth, high elevation old growth, not old growth). This can be used as a base of comparison for the other three rows. These rows show the agreement rates that are dependent on whether the classification of Morrison's "other ancient forest" is counted as low elevation old growth, old growth (high or low), or not old growth in the Pacific Meridian scheme. As with the confusion matrices, there is no pattern to the figures. On the Olympic National Forest, agreement improves if "other" is classified as not old growth. On the Mt. Baker-Snoqualmie National Forest, there is no appreciable difference in agreement. On the Gifford Pinchot National Forest, agreement gets somewhat worse, and on the Mt. Hood National Forest, agreement gets notably worse. The overall agreement rates are in fairly broad ranges from 63 to 77.5 or 57.4 to 80.9 percent. If Morrison's "other ancient forest" category is treated as old growth, the Olympic and Mt. Baker-Snoqualmie National Forests have very similar agreement rates; however, there is no similarity between other agreement rates.
Table 5-7 aggregates the confusion matrices in Table 5-5. All of The Wilderness Society's three ancient forest categories are summed in one column, and both high and low elevation old growth from Pacific Meridian Resources are summed in one row. This provides a more direct comparison of the two categories. The accompanying maps correspond to these aggregated tables.
The map of the Olympic National Forest necessarily looks the cleanest because of the higher rate of agreement. However, some observations can be made. Many of the areas that Morrison claims to be old growth but Pacific Meridian disagrees are concentrated in a few specific areas of the forest (east and north). The areas where Pacific Meridian found old growth but Morrison disagrees are found in large patches mostly in the south and southeast.
The map of the Mt. Baker-Snoqualmie National Forest also look relatively clean. Areas that Morrison claims to be old growth but Pacific Meridian disagrees are generally along the edges of areas that both agree are old growth. Areas that Pacific Meridian found to be old growth but Morrison disagrees are both adjacent to agreed old-growth areas but also are scattered in large blocks across most of the forest except in the "checkerboard" lands south of Snoqualmie Pass. Such areas seem to be prevalent at higher elevations. This pattern is especially evident in the northern part of the forest.
The map of the Gifford Pinchot National Forest has a lot of contrasting areas. Both projects are in agreement that the Mt. St. Helens blast zone is not old growth. Both studies also agree on large areas of old growth to the east of Mt. St. Helens and in the northeastern portion of the forest, as well as a somewhat smaller area in the southern part of the forest. The rest of the forest is characterized by disagreement, with Pacific Meridian finding large amounts of old growth in north central and northwestern portions of the forest, in the area northeast of Mt. St. Helens, and in most of the southern half of the forest, all of which Morrison did not identify. The areas where Morrison identified ancient forest that Pacific Meridian did not are also scattered around the forest, with a large concentration north of Mt. Adams.
The map of the Mt. Hood National Forest is much more difficult to interpret. More than 50% of the forest area are places where Pacific Meridian found old growth and Morrison did not, and these places are well distributed throughout the forest. Areas where both studies found old growth are also found throughout the forest, with concentrations in the northern, south central, and far southwestern portions of the forest. Areas where Morrison found old growth and Pacific Meridian did not are largely adjacent to areas that are agreed upon as old growth. Areas where both studies agreed there was no old growth (only 10% of the forest area) are found scattered through much of the forest, with a notable lack of such areas in the northern portion. One interesting pattern is that The Wilderness Society tended to find old growth that Pacific Meridian did not on the north sides of stands where they agree, and Pacific Meridian Resources found old growth that Morrison did not on the south side of stands where they agree. This pattern is especially noticeable in the far northern and south central areas of the forest, where the map shows thin red areas below and thin orange areas above green areas. This is presumably a positional error, perhaps a relict of the reprojection that I performed, and despite the fact that the townships do line up correctly.
There are no trends evident in the data. Pacific Meridian Resources did not classify all of Morrison's ancient forest as old growth, despite their finding much more old growth in general. They found it outside areas where Morrison found it. They did not even have a higher rate of agreement on Forests where they found significantly more old growth. Thus, while this comparison is interesting, it does not find any consistent bias in the datasets.
