Synoptic Data QC

Introduction

The /timeseries, /latest and /nearesttime API services provide Synoptic QC (i.e. data checks) as additional attributes delivered alongside the data. QC is performed immediately when new data is received, and is available for “real time” data in addition to historical data.

Examples applying a suite of Synoptic data checks are shown below. In each figure, observations that fail a check are colored according to the flag indicated in the legend. API users can elect to return these flags alongside the data, or to automatically remove flagged data.

Air temperature observations during June, 2019 for a 30-mile radius centered in Houston, TX:

Wind gust observations during Feb, 2020 for a 30-mile radius centered in Seattle, WA:

There are many potential causes of erroneous data: sensor failure, poor sensor siting or calibration, data transmission errors, or incorrect conversion of the transmitted data arising from invalid metadata. The Synoptic data checks are intended to identify: (1) physically implausible values and (2) values that may not be representative of the conditions prevailing at that time (outliers). All data accessible via the Synoptic Weather API must be considered provisional. Official data archives for some networks are available directly from data providers.

Basic QC

Basic QC includes the range check (sl_range_check), rate of change check (sl_rate_check) and persistence check (sl_pers_check). Note that by default, only the range check is automatically applied to remove data from all API requests. In addition, the range check acts as a single entry point, where no further basic or advanced data checks are run after a failed range check. If a user is specifying individual qc checks using qc_checks= (see “Data Checks and Quality Control” in the docs specific to each service), we highly recommend specifying these three basic data checks. Additional optional basic qc checks include the wind speed vs wind gust check (sl_windspd_maxgust_check) and the wind speed vs wind direction check (sl_windspd_winddir_check). All basic QC checks can be enabled with qc_checks=basic.

Advanced QC

Advanced QC includes the Synoptic spatial value check (sl_spatial_value_check) and a suite of percentile checks (see summary below). These checks are ideal for identifying values which may be physically plausible for a variable, but are clear outliers in context with either neighboring station data or in context to the station’s historic data distribution. In most cases, it is best to specify the full suite of Synoptic basic and advanced qc checks with qc_checks=synopticlabs. If you want to enable only the advanced qc checks, you can do so with qc_checks=advanced.

Note that sl_percentile_loflag_check and sl_percentile_hiflag_check are not included in the synopticlabs suite, such that a user can elect to remove data for all synopticlabs checks without removing any valid observations. In addition, the legacy MesoWest data checks (mw_multvariate_lin_reg_check, mw_24h_wind_persistence_check and mw_uu2dvar_rejection) are not included in the synopticlabs check suite, and must be enabled individually or with qc_checks=mesowest.

Please contact us for additional information on the appropriate application of Synoptic data checks for your use case.

QC via the Weather API

Within the Time Series, Latest and Nearest Time services, users can add QC parameters to their queries to flag the potentially erroneous observations. By default, the Weather API doesn’t return data that has been flagged as non-plausible by the Synoptic Range Change.

If the qc parameter is omitted then the API will return data while assuming the following: qc=on, qc_remove_data=on, qc_flags=off and qc_checks=sl_range_check. Note that if the range check removes all values for all requested stations and variables, a response message of “No stations found for this request” will be returned. If the range check removes values only for certain variables, those variables will not be present in the OBSERVATIONS object.

• qc (on [default], off), Indicates the application behavior of the QC attributes on the data requested. If set to off then all data will be returned without data checks and quality control (not recommended). If set to on, a QC_SUMMARY object is returned, a QC_FLAGGED: [bool] key will be inside the STATION object, and individual data checks will be in the qc response key for each variable within the OBSERVATIONS object.

• qc_remove_data (on, off, mark), Indicates the response behavior for an observation that fails a user specified data check. (default on if qc parameter omitted, else off if qc=on)

  • off returns the data values even if a data check failure is present for that data.

  • on removes failed data values, returning null. If all values for all requested stations and variables are set to null, a response message of “No stations found for this request” will be returned. If the value for an individual variable is set to null, the variable will not be present in the OBSERVATIONS object.

  • mark replaces failed data with a value of false.

• qc_flags (on, off), Indicates whether the data checks are returned alongside any data that failed a requested check. If on then the data checks will be returned in the qc response key for each variable within the OBSERVATIONS block. (default off if qc parameter omitted, else on if qc=on)

• qc_checks ([flag name], [flag source], keyword), defines a list of applied data checks. (defaults to sl_range_check if qc parameter omitted, else synopticlabs if qc=on)

  • “flag name” allows targeting one or more specific data checks in a comma separated list (e.g. sl_range_check,sl_rate_check)

  • “flag source” allows targeting one or more data check providers (synopticlabs, mesowest or madis).

  • “keyword” can be one of the following: basic,advanced, all. Click here to read more about the basic and advanced groups of checks).all will apply all data checks in our system. Caution: This argument will enable all flag sources, some of which can aggressively flag data, and may remove significant amounts of data if qc_remove_data=on.

Some examples of modifying the default QC checks are:

• qc_checks=synopticlabs,ma_range_check, Applies the Synoptic QC suite and MADIS range check

• qc_checks=synopticlabs,madis, Applies the Synoptic and MADIS QC suites.

QC segments service

Nearly all users will rely on the /timeseries, /latest and /nearesttime API services to identify the values that fail relevant data checks. However, those services rely on code to access the archived data checks that are also available to users via the QC segment API service /qcsegments. For efficient storage and short access times, only the start and end times are stored for each period for which any values have failed a data check (referred to as a QC segment).

Data Checks Summary

See a full list of our QC Checks

Data Check Descriptions

Technical descriptions of the Synoptic and MesoWest data checks.

Range check

Is the value physically plausible?

API Name: sl_range_check

Physically implausible values are identified using this check. Only if a value passes this check will the remaining checks be applied.

There are situations where physically implausible values may not indicate sensor failure. For example, negative snow depth is considered physically implausible but may result from the height of the sensor above ground changing over the winter season as snow compresses the grass or other underlying obstructions. Further, the maximum relative humidity with respect to water is defined to be 100%, although many research-grade sensors commonly report values in excess of that limit.

Time rate of change check

Is the change between two consecutive values too large?

API Name: sl_rate_check

The rate change check compares the absolute difference between two consecutive observations to the maximum plausible rate of change expected within the time interval.

Persistence check

Do the values appear to be unchanging?

sl_pers_check

For parameters for which it is appropriate to do so, the persistence check compares the range of values (difference between the maximum and minimum) within a specified time period relative to the minimum plausible range expected for that parameter.

The following table lists the variable, time period evaluated, and the minimum plausible range expected for that variable..

Wind speed vs. wind gust check

Is the wind speed value greater than the wind gust value?

API Name: sl_windspd_maxgust_check

If the reported wind speed is greater than the wind gust, both the wind speed and gust values are flagged. Note: the conventions for reporting wind speed and gust differ among data providers. The wind gust reported may be a maximum wind speed within the time interval for that observation.

Wind speed vs. wind direction check

Is a wind direction value reported when the wind speed is zero?

API Name: sl_windspd_winddir_check

If a wind direction value is reported when the wind speed is zero, wind direction is flagged.

Percentile high outlier check

Is the value a high outlier compared to the percentile distribution of the station’s historical record?

API Name: sl_percentile_hioutlier_check

This flag is applied to air temperature values if:

value > (99.5th percentile + 8°C)

Synoptic Data completes yearly re-analysis of the historical record of air temperature data for every station in our database. A series of outlier identification algorithms are run to identify and remove obvious outliers in the historical record. Using this “cleaned” record, we then leverage the percentile distribution of the data to compare to current observations. To be used in this check, a station must have at least 3 years of continuous air temperature data. This check is intended to identify values that are very likely to be erroneous (significantly beyond any historically high or low temperatures) for the date and hour of the observation. We use extreme air temperature events to calibrate the 8°C threshold, with the intention that flagged data is likely unrealistic and can be removed from an API request.

Percentile low outlier check

Is the value a low outlier compared to the percentile distribution of the station’s historical record?

API Name: sl_percentile_looutlier_check

This flag is applied to air temperature values if:

value < (0.5th percentile - 8°C)

See technical details for the percentile outlier checks as described above for the percentile high outlier check.

Percentile high flag check

Is the value near or above historical high values compared to the percentile distribution of the station’s historical record?

API Name: sl_percentile_hiflag_check

This flag is applied to air temperature values if:

99.5th percentile < value < (99.5th percentile + 8°C)

The percentile high flag is intended to indicate air temperature data that may be near or above historical maximum temperatures for the date and hour of the observation. These flags should not be used to remove data from an API request, but instead may be used to indicate “interesting” or historically significant observations. This check is not included in the synopticlabs test suite.

Percentile low flag check

Is the value near or below historical low values compared to the percentile distribution of the station’s historical record?

API Name: sl_percentile_loflag_check

This flag is applied to air temperature values if:

(0.5th percentile - 8°C) < value < 0.5th percentile

The percentile low flag is intended to indicate air temperature data that may be near or below historical minimum temperatures for the date and hour of the observation. These flags should not be used to remove data from an API request, but instead may be used to indicate “interesting” or historically significant observations. This check is not included in the synopticlabs test suite.

Spatial percentile check

Is the percentile value significantly different than neighboring observations?

API Name: sl_spatial_percentile_check

This flag is applied to air temperature values if:

abs(percentile value - median(neighbor values)) >= 0.6

For example, if the current value is at the 10th percentile and the median of three neighboring stations is >= the 70th percentile, the value will be flagged. To run this check there must be at least two neighboring stations that meet the following requirements:

• Within 30 km distance and 200 meters elevation

• Greater than 3 years of historical data

• Within 2 hours of the current value

• Neighboring percentiles must agree with each other within <= 0.2

• Cannot have any open QC flag ranges

Spatial value check

Is the value significantly different than neighboring observations?

API Name: sl_spatial_value_check

This flag is applied to multiple variables according to thresholds defined in the following table. Value thresholds indicate the threshold for flagging the current value, as: abs(value - median( neighbor values) >= threshold. Neighbor thresholds indicate the threshold for agreement between neighbor values, as: (max(neighbor values) - min(neighbor values)) <= threshold. Thresholds are not provided for pressure or relative humidity, as these variables are flagged via derived altimeter and derived dew point temperature, respectively.

To run this check there must be at least two neighboring stations that meet the following requirements:

• Within 30 km distance and 200 meters elevation

• Closest-in-time values are within -1.5 hrs and +0.5 hrs of the current value

• Cannot have any open QC flag ranges

Linear regression check

API Name: mw_multvariate_lin_reg_check

A multivariate linear regression check courtesy of the University of Utah MesoWest program. This check identifies air temperature, moisture, and pressure observations that appear to be significantly different than surrounding observations. More information on this check can be obtained here.

24-hr wind persistence check

API Name: mw_24h_wind_persistence_check

Persistence check to identify unchanging wind speed values, courtesy of the University of Utah MesoWest program. This check is redundant to the sl_pers_check.

UU2DVAR rejection check

API Name: mw_uu2dvar_rejection

A spatial comparison check with surrounding observations and gridded surface meteorological fields courtesy of the University of Utah Variational Surface Analysis (UU2DVAR). Observations that have been flagged by this check indicate they were statistically rejected from the UU2DVAR algorithm. For more information on UU2DVAR, please consult this publication.