Accuracy assessment

Revision as of 13:57, 2 February 2016

sorry:

This section is still under construction! This article was last modified on 02/2/2016. If you have comments please use the Discussion page or contribute to the article!

For decision making, based on a map, the map has to be accurate or, at least, the accuracy has to be known. Therefore, accuracy assessment gives information about the quality of a map created by remotely sensed data. Besides the quality, an accuracy assessment is realised to identify possible sources of errors and as an indicator used in comparisons (e.g. of algorithms).

Confusion Matrix

A confusion matrix compare the relationship between known reference data and the results of the classification, based on the classes.

There are several characteristics that can be obtained by a confusion matrix:

• Classification errors of omission (exclusion) and commission (inclusion)
  • Number of elements that are wrongly excluded or included to a certain category
• Overall accuracy
  • Indicates the quality of the map classification
  • Can be calculated by dividing the total number of correctly classified pixels by the total number of reference pixels
• Producer's accuracy
  • Indicates the quality of the classification of training set pixels
  • Can be calculated by dividing the correctly classified pixels in each category by the number of training set pixels of the corresponding category
• User's accuracy
  • Indicates the probability that prediction represent reality
  • Can be calculated by dividing the correctly classified pixels in each category by the total number of pixels that were classified in that category

It has to be recognized, that the confusion matrix is based on training data and, therefore, this procedure only indicates how well the statistics extracted from these areas can be used to cateogrize the same areas.

Implementation in QGIS

GRASS

The GRASS Toolbox provides a tool named r.kappa, which assess the accuracy with the use of two raster files. Besides the confusion matrix, different kappa values are calculated.

1. Open Toolbox --> GRASS --> Raster --> r.kappa.
2. Set the classified map as Raster layer containing classification result.
3. Set the training data as Raster layer containing reference classes.
4. Save the output file under Error matrix and kappa.

Orfeo Toolbox

The Orfeo Toolbox provides two opportunities for the data format of the reference data: raster and shape.

1. Open Toolbox --> Orfeo --> Learning --> ComputeConfusionMatrix (vector) or ComputeConfusionMatrix (raster) depending on your reference data.
2. Set the classified map as Input image.
3. Set the training data as Input reference data.
4. In case of the ComputeConfusionMatrix (vector) tool, it is also possible to adjust the Field name, which includes the class information of the reference data.
5. Save the output file under Matrix output.

Semi-automatic classification plugin

Also the Semi-automatic classification plugin provides the opportunity to use raster files or shape files as reference data.

1. Install the the Semi-automatic classification plugin under Plugins --> Manage and Install Plugins....
2. Open SCP --> Post processing --> Accuracy.
3. Press the first button Refresh list to load all possible raster files as basis for the classified map and select in the drop-down menue of Select the classification to assess the right raster file.
4. Press the second button Refresh list to load all possible raster or shape files as basis for the reference and select in the drop-down menue of Select the reference shapefile or raster the right raster or shape file.
5. Press the button Calculate error matrix.

Furthermore, there are additional plugins like Accuracy Assessment, Accur Assess and Validation Tool to calculate an error matrix.

Literature

Congalton, R. G. & Green, K. (2008). Assessing the accuracy of remotely sensed data: principles and practices. CRC press.

Lillesand, T., Kiefer, R. W. & Chipman, J. (2008). Remote sensing and image interpretation. John Wiley & Sons.