Skip to main content
Skip table of contents

Synoptic Data QC

Introduction

The /timeseries, /latest and /nearesttime API services provide Synoptic QC (i.e. data checks) as additional attributes delivered alongside the data. QC is performed immediately when new data is received, and is available for “real time” data in addition to historical data.

Examples applying a suite of Synoptic data checks are shown below. In each figure, observations that fail a check are colored according to the flag indicated in the legend. API users can elect to return these flags alongside the data, or to automatically remove flagged data.

Air temperature observations during June, 2019 for a 30-mile radius centered in Houston, TX:

Wind gust observations during Feb, 2020 for a 30-mile radius centered in Seattle, WA:

There are many potential causes of erroneous data: sensor failure, poor sensor siting or calibration, data transmission errors, or incorrect conversion of the transmitted data arising from invalid metadata. The Synoptic data checks are intended to identify: (1) physically implausible values and (2) values that may not be representative of the conditions prevailing at that time (outliers). All data accessible via the Synoptic Weather API must be considered provisional. Official data archives for some networks are available directly from data providers.

Basic QC

Basic QC includes the range check (sl_range_check), rate of change check (sl_rate_check) and persistence check (sl_pers_check). Note that by default, only the range check is automatically applied to remove data from all API requests. In addition, the range check acts as a single entry point, where no further basic or advanced data checks are run after a failed range check. If a user is specifying individual qc checks using qc_checks= (see “Data Checks and Quality Control” in the docs specific to each service), we highly recommend specifying these three basic data checks. Additional optional basic qc checks include the wind speed vs wind gust check (sl_windspd_maxgust_check) and the wind speed vs wind direction check (sl_windspd_winddir_check). All basic QC checks can be enabled with qc_checks=basic.

Advanced QC

Advanced QC includes the Synoptic spatial value check (sl_spatial_value_check) and a suite of percentile checks (see summary below). These checks are ideal for identifying values which may be physically plausible for a variable, but are clear outliers in context with either neighboring station data or in context to the station’s historic data distribution. In most cases, it is best to specify the full suite of Synoptic basic and advanced qc checks with qc_checks=synopticlabs. If you want to enable only the advanced qc checks, you can do so with qc_checks=advanced.

Note that sl_percentile_loflag_check and sl_percentile_hiflag_check are not included in the synopticlabs suite, such that a user can elect to remove data for all synopticlabs checks without removing any valid observations. In addition, the legacy MesoWest data checks (mw_multvariate_lin_reg_check, mw_24h_wind_persistence_check and mw_uu2dvar_rejection) are not included in the synopticlabs check suite, and must be enabled individually or with qc_checks=mesowest.

Please contact us for additional information on the appropriate application of Synoptic data checks for your use case.

QC via the Weather API

Within the Time Series, Latest and Nearest Time services, users can add QC parameters to their queries to flag the potentially erroneous observations. By default, the Weather API doesn’t return data that has been flagged as non-plausible by the Synoptic Range Change.

If the qc parameter is omitted then the API will return data while assuming the following: qc=on, qc_remove_data=on, qc_flags=off and qc_checks=sl_range_check. Note that if the range check removes all values for all requested stations and variables, a response message of “No stations found for this request” will be returned. If the range check removes values only for certain variables, those variables will not be present in the OBSERVATIONS object.

  • qc (on [default], off), Indicates the application behavior of the QC attributes on the data requested. If set to off then all data will be returned without data checks and quality control (not recommended). If set to on, a QC_SUMMARY object is returned, a QC_FLAGGED: [bool] key will be inside the STATION object, and individual data checks will be in the qc response key for each variable within the OBSERVATIONS object.

  • qc_remove_data (on, off, mark), Indicates the response behavior for an observation that fails a user specified data check. (default on if qc parameter omitted, else off if qc=on)

    • off returns the data values even if a data check failure is present for that data.

    • on removes failed data values, returning null. If all values for all requested stations and variables are set to null, a response message of “No stations found for this request” will be returned. If the value for an individual variable is set to null, the variable will not be present in the OBSERVATIONS object.

    • mark replaces failed data with a value of false.

  • qc_flags (on, off), Indicates whether the data checks are returned alongside any data that failed a requested check. If on then the data checks will be returned in the qc response key for each variable within the OBSERVATIONS block. (default off if qc parameter omitted, else on if qc=on)

  • qc_checks ([flag name], [flag source], keyword), defines a list of applied data checks. (defaults to sl_range_check if qc parameter omitted, else synopticlabs if qc=on)

    • “flag name” allows targeting one or more specific data checks in a comma separated list (e.g. sl_range_check,sl_rate_check)

    • “flag source” allows targeting one or more data check providers (synopticlabs, mesowest or madis).

    • “keyword” can be one of the following: basic,advanced, all. Click here to read more about the basic and advanced groups of checks).all will apply all data checks in our system. Caution: This argument will enable all flag sources, some of which can aggressively flag data, and may remove significant amounts of data if qc_remove_data=on.

Some examples of modifying the default QC checks are:

  • qc_checks=synopticlabs,ma_range_check, Applies the Synoptic QC suite and MADIS range check

  • qc_checks=synopticlabs,madis, Applies the Synoptic and MADIS QC suites.

QC segments service

Nearly all users will rely on the /timeseries, /latest and /nearesttime API services to identify the values that fail relevant data checks. However, those services rely on code to access the archived data checks that are also available to users via the QC segment API service /qcsegments. For efficient storage and short access times, only the start and end times are stored for each period for which any values have failed a data check (referred to as a QC segment).

Data Checks Summary

Data Check

API name

Description

Range check

sl_range_check

Identifies physically implausible values. For example, an air temperature of 150°F or -80°F would be flagged.

Time rate of change check

sl_rate_check

Identifies if the absolute difference in two consecutive values is greater than the maximum plausible rate of change expected for the observed time interval.

Persistence check

sl_pers_check

Identifies if a sequence of observations appear unchanging. For example, an air temperature sensor reporting the same value every 15 minutes for 24 hours would be flagged as persistent.

Wind speed vs. wind gust check

sl_windspd_maxgust_check

Flags both wind speed and gust if the wind speed is greater than the wind gust.

Wind speed vs. wind direction check

sl_windspd_winddir_check

Flags wind direction if wind speed is zero.

Percentile high outlier check

sl_percentile_hioutlier_check

Uses the percentile distribution from the station’s historical record to identify high outliers (air temperature only).

Percentile low outlier check

sl_percentile_looutlier_check

Uses the percentile distribution from the station’s historical record to identify low outliers (air temperature only).

Percentile high flag check

sl_percentile_hiflag_check

Uses the percentile distribution from the station’s historical record to identify “interesting” values that are near or above the historical high value for that date and time (air temperature only).

Percentile low flag check

sl_percentile_loflag_check

Uses the percentile distribution from the station’s historical record to identify “interesting” values that are near or below the historical low value for that date and time (air temperature only).

Spatial percentile check

sl_spatial_percentile_check

Compares the current observation’s percentile value (using the station’s historical record) to neighboring station percentile values within distance and elevation thresholds (air temperature only).

Spatial Value check

sl_spatial_value_check

Compares the current observation value to nearest-in-time values from neighboring stations within distance and elevation thresholds (air temperature, relative humidity, dew point temperature, wind speed, wind gust, pressure and altimeter).

Linear regression check

mw_multvariate_lin_reg_check

Spatial analysis check courtesy of the University of Utah MesoWest program.

24-hr wind persistence check

mw_24h_wind_persistence_check

Persistence check to identify unchanging wind speed values, courtesy of the University of Utah MesoWest program. This check is redundant to the sl_pers_check.

UU2DVAR rejection check

mw_uu2dvar_rejection

Spatial analysis check courtesy of the University of Utah Variational Surface Analysis (UU2DVAR).

See a full list of our QC Checks

Data Check Descriptions

Technical descriptions of the Synoptic and MesoWest data checks.

Range check

Is the value physically plausible?

API Name: sl_range_check

Physically implausible values are identified using this check. Only if a value passes this check will the remaining checks be applied.

There are situations where physically implausible values may not indicate sensor failure. For example, negative snow depth is considered physically implausible but may result from the height of the sensor above ground changing over the winter season as snow compresses the grass or other underlying obstructions. Further, the maximum relative humidity with respect to water is defined to be 100%, although many research-grade sensors commonly report values in excess of that limit.

Variable

Unit

Minimum

Maximum

Altimeter

Pascals

84998.31

108025.75

Pressure

Pascals

60000.0

108000.0

Temperature

Celsius

-59.44

57.22

Dew Point

Celsius

-59.44

35.0

Relative Humidity

%

0.00

100.00

Wind Speed

m/s

0.00

200.00

Wind Direction

Degrees

0.00

360.00

Wind Gust

m/s

0.00

200.00

Solar Radiation

W/m**2

0.00

1500.00

Soil Temperature

Celsius

-58.00

185.00

Sea_level pressure

Pascals

850.00

1080.00

Hours of sun

Hours

0.00

24.00

Water Temperature

Celsius

0.00

135.00

Weather conditions

code

0.00

512000.00

Cloud_layer_3 height/coverage

code

0.00

8009.00

Low_cloud symbol

code

0.00

9.00

Mid_cloud symbol

code

10.00

19.00

High_cloud symbol

code

20.00

29.00

Pressure Tendency

code

0.00

8999.00

Road sensor number

 

1.00

10.00

Road Temperature

Celsius

-40.00

150.00

Road_Freezing Temperature

Celsius

0.00

40.00

Road_Surface Conditions

code

1.00

100.00

Cloud_layer_1 height/coverage

code

0.00

18009.00

Cloud_layer_2 height/coverage

code

0.00

8009.00

Visibility

Statute miles

0.00

200.00

Sonic_Wind Direction

Degrees

0.00

360.00

Remarks

text

NULL

NULL

Raw observation

text

NULL

NULL

6 Hr High Temperature

Celsius

-75.00

135.00

6 Hr Low Temperature

Celsius

-75.00

135.00

Peak_Wind Speed

m/s

0.00

200.00

Fuel Temperature

Celsius

-40.00

140.00

Fuel Moisture

gm

0.00

100.00

Ceiling

Meters

0.00

40000.00

Sonic_Wind Speed

m/s

0.00

200.00

Pressure change

code

0.00

2000.00

IR_Soil Temperature

Celsius

-75.00

135.00

Temperature in_case

Celsius

-75.00

135.00

Soil Moisture

%

0.00

100.00

Battery voltage

volts

0.00

50.00

Data Insert Date/Time

minutes

0.00

16000000.00

Data Update Date/Time

minutes

0.00

16000000.00

Road Subsurface Temperature

Celsius

-40.00

150.00

Water Temperature

Celsius

0.00

135.00

Evapotranspiration

Millimeters

0.00

5.00

24 Hr High Temperature

Celsius

-75.00

135.00

24 Hr Low Temperature

Celsius

-75.00

135.00

Peak_Wind Direction

Degrees

0.00

360.00

Net Radiation

W/m**2

-500.00

1000.00

Soil Moisture tension

centibars

0.00

300.00

1500 m Pressure

Pascals

700.00

1000.00

Wet bulb temperature

Celsius

-75.00

135.00

Air_Temperature at_2_meters

Celsius

-75.00

135.00

Air_Temperature at_10_meters

Celsius

-75.00

135.00

Pressure

Pascals

600.00

1080.00

Temperature

Celsius

-75.00

135.00

Relative Humidity

%

0.00

100.00

Wind Speed

m/s

0.00

200.00

Wind Direction

Degrees

0.00

360.00

Wind Gust

m/s

0.00

250.00

Latitude

Degrees

-90.00

90.00

Longitude

Degrees

-180.00

180.00

Elevation

Meters

-300.00

30000.00

Platform True_Direction

Degrees

0.00

360.00

Primary_Swell Wave_Direction

Degrees

0.00

360.00

Primary_Swell Wave_Period

Seconds

0.00

99.00

Primary_Swell Wave_Height

Meters

0.00

33.63

Secondary_Swell Wave_Direction

Degrees

0.00

360.00

Secondary_Swell Wave_Period

Seconds

0.00

99.00

Secondary_Swell Wave_Height

Meters

0.00

33.63

Tide Indicator

code

0.00

10.00

Tide Departure

Meters

0.00

100.00

Surface Temperature

Celsius

-58.00

185.00

Platform True_Speed

m/s

0.00

125.00

Wave Period

Seconds

0.00

99.00

Wave Height

Meters

0.00

33.63

Net Shortwave Radiation

W/m**2

-500.00

1000.00

Net Longwave Radiation

W/m**2

-500.00

1000.00

Sonic Temperature

Celsius

-75.00

135.00

Vertical_Velocity

m/s

-2.00

2.00

Zonal_Wind Standard_Deviation

m/s

0.00

5.00

Meridional_Wind Standard_Deviation

m/s

0.00

5.00

Vertical_Wind Standard_Deviation

m/s

0.00

5.00

Temperature Standard_Deviation

Centigrade

0.00

5.00

Vertical Heat_Flux

m/s C

-2.00

2.00

Friction Velocity

m/s

0.00

5.00

SIGW/USTR

nondimensional

0.00

5.00

Sonic_Obs Total

nondimensional

0.00

5000.00

Sonic Warnings

nondimensional

0.00

5000.00

Moisture Standard_Deviation

g/m**3

0.00

5.00

Vertical Moisture_Flux

m/s g/m**3

-1.00

1.00

Dew Point

Celsius

-75.00

135.00

Virtual Temperature

Celsius

-75.00

135.00

Geopotential Height

Meters

-300.00

30000.00

Outgoing Shortwave Radiation

W/m**2

0.00

1000.00

Clear Sky Solar Radiation

W/m**2

0.00

1500.00

Grip 2 Level of Grip

 

0.00

1.00

Grip 1 Ice Friction Code

 

0.00

1.00

Photosynthetically Active Radiation

umol/m**2 s

0.00

2500.00

PM_2.5 Concentration

ug/m3

0.00

200.00

Flow Rate

liters/min

0.00

100.00

Internal Relative_Humidity

%

0.00

100.00

Air Flow Temperature

Celsius

-75.00

135.00

Ozone Concentration

ppb

0.00

500.00

Stream flow

ft3/s

NULL

NULL

Gage height

ft

NULL

NULL

Black Carbon Concentration

ug/m3

0.00

200.00

Soil Temperature 2

Celsius

-58.00

185.00

18_Inch Soil_Temperature

Celsius

-58.00

185.00

18_Inch Soil_Temperature2

Celsius

-58.00

185.00

20_Inch Soil_Temperature

Celsius

-58.00

185.00

Soil Temperature 3

Celsius

-58.00

185.00

Soil Temperature 4

Celsius

-58.00

185.00

Soil Moisture 2

%

0.00

100.00

Particulate Concentration

ug/m3

0.00

10000.00

Filter Percentage

%

0.00

100.00

Sensor Error Code

code

0.00

1000.00

Electric Conductivity

dS/m

0.00

10.00

Permittivity

 

0.00

100.00

Past weather

code

0.00

9.00

Surface Level

Millimeters

-100.00

100.00

Incoming Longwave Radiation

W/m**2

0.00

1500.00

Outgoing Longwave Radiation

W/m**2

0.00

1500.00

Time rate of change check

Is the change between two consecutive values too large?

API Name: sl_rate_check

The rate change check compares the absolute difference between two consecutive observations to the maximum plausible rate of change expected within the time interval.

Variable name

Unit

Rules

Pressure

Pascals

5 min: 925
15 min: 1000
30 min: 1500
60 min: 1500

Temperature

Celsius

1-3 min: 3
4 min: 4
5 min: 5
15 min 7.5
30 min: 15
60 min: 20

Persistence check

Do the values appear to be unchanging?

sl_pers_check

For parameters for which it is appropriate to do so, the persistence check compares the range of values (difference between the maximum and minimum) within a specified time period relative to the minimum plausible range expected for that parameter.

The following table lists the variable, time period evaluated, and the minimum plausible range expected for that variable..

Variable name

Unit

Evaluation period

Min number of obs

Required change

Altimeter

Pascals

24 hrs

24

5.0

Pressure

Pascals

24 hrs

24

5.0

Temperature

Celsius

24 hrs

24

0.1

Dew Point

Celsius

24 hrs

24

0.1

Relative Humidity

%

24 hrs

24

0.5

Wind Speed

m/s

24 hrs

24

0.25

Wind Direction

Degrees

24 hrs

24

2.5

Wind Gust

m/s

24 hrs

24

0.25

Solar Radiation

W/m**2

48 hrs

48

0.5

Soil Temperature

Celsius

48 hrs

48

0.05

Sea_level pressure

Pascals

24 hrs

24

5.0

Water Temperature

Celsius

48 hrs

48

0.05

Road Temperature

Celsius

24 hrs

24

0.1

Sonic_Wind Direction

Degrees

24 hrs

24

1.0

Peak_Wind Speed

m/s

24 hrs

24

0.25

Fuel Temperature

Celsius

72 hrs

72

0.1

Fuel Moisture

gm

24 hrs

24

0.1

Sonic_Wind Speed

m/s

24 hrs

24

0.1

IR_Soil Temperature

Celsius

24 hrs

24

0.1

Road Subsurface Temperature

Celsius

48 hrs

48

0.05

Water Temperature

Celsius

48 hrs

48

0.05

Peak_Wind Direction

Degrees

24 hrs

24

2.5

Net Radiation

W/m**2

48 hrs

48

0.5

Air_Temperature at_2_meters

Celsius

24 hrs

24

0.1

Air_Temperature at_10_meters

Celsius

24 hrs

24

0.1

Pressure

Pascals

24 hrs

24

5.0

Temperature

Celsius

24 hrs

24

0.1

Relative Humidity

%

24 hrs

24

0.5

Wind Speed

m/s

24 hrs

24

0.25

Wind Direction

Degrees

24 hrs

24

2.5

Wind Gust

m/s

24 hrs

24

2.5

Surface Temperature

Celsius

24 hrs

24

0.1

Net Shortwave Radiation

W/m**2

48 hrs

48

0.5

Net Longwave Radiation

W/m**2

48 hrs

48

0.5

Sonic Temperature

Celsius

24 hrs

24

0.1

Dew Point

Celsius

24 hrs

24

0.1

Virtual Temperature

Celsius

24 hrs

24

0.1

Outgoing Shortwave Radiation

W/m**2

48 hrs

48

0.5

Photosynthetically Active Radiation

umol/m**2 s

48 hrs

48

0.5

PM_2.5 Concentration

ug/m3

24 hrs

24

0.5

Ozone Concentration

ppb

24 hrs

24

0.5

Black Carbon Concentration

ug/m3

24 hrs

24

0.5

Particulate Concentration

ug/m3

24 hrs

24

0.5

Wind speed vs. wind gust check

Is the wind speed value greater than the wind gust value?

API Name: sl_windspd_maxgust_check

If the reported wind speed is greater than the wind gust, both the wind speed and gust values are flagged. Note: the conventions for reporting wind speed and gust differ among data providers. The wind gust reported may be a maximum wind speed within the time interval for that observation.

Wind speed vs. wind direction check

Is a wind direction value reported when the wind speed is zero?

API Name: sl_windspd_winddir_check

If a wind direction value is reported when the wind speed is zero, wind direction is flagged.

Percentile high outlier check

Is the value a high outlier compared to the percentile distribution of the station’s historical record?

API Name: sl_percentile_hioutlier_check

This flag is applied to air temperature values if:

value > (99.5th percentile + 8°C)

Synoptic Data completes yearly re-analysis of the historical record of air temperature data for every station in our database. A series of outlier identification algorithms are run to identify and remove obvious outliers in the historical record. Using this “cleaned” record, we then leverage the percentile distribution of the data to compare to current observations. To be used in this check, a station must have at least 3 years of continuous air temperature data. This check is intended to identify values that are very likely to be erroneous (significantly beyond any historically high or low temperatures) for the date and hour of the observation. We use extreme air temperature events to calibrate the 8°C threshold, with the intention that flagged data is likely unrealistic and can be removed from an API request.

Percentile low outlier check

Is the value a low outlier compared to the percentile distribution of the station’s historical record?

API Name: sl_percentile_looutlier_check

This flag is applied to air temperature values if:

value < (0.5th percentile - 8°C)

See technical details for the percentile outlier checks as described above for the percentile high outlier check.

Percentile high flag check

Is the value near or above historical high values compared to the percentile distribution of the station’s historical record?

API Name: sl_percentile_hiflag_check

This flag is applied to air temperature values if:

99.5th percentile < value < (99.5th percentile + 8°C)

The percentile high flag is intended to indicate air temperature data that may be near or above historical maximum temperatures for the date and hour of the observation. These flags should not be used to remove data from an API request, but instead may be used to indicate “interesting” or historically significant observations. This check is not included in the synopticlabs test suite.

Percentile low flag check

Is the value near or below historical low values compared to the percentile distribution of the station’s historical record?

API Name: sl_percentile_loflag_check

This flag is applied to air temperature values if:

(0.5th percentile - 8°C) < value < 0.5th percentile

The percentile low flag is intended to indicate air temperature data that may be near or below historical minimum temperatures for the date and hour of the observation. These flags should not be used to remove data from an API request, but instead may be used to indicate “interesting” or historically significant observations. This check is not included in the synopticlabs test suite.

Spatial percentile check

Is the percentile value significantly different than neighboring observations?

API Name: sl_spatial_percentile_check

This flag is applied to air temperature values if:

abs(percentile value - median(neighbor values)) >= 0.6

For example, if the current value is at the 10th percentile and the median of three neighboring stations is >= the 70th percentile, the value will be flagged. To run this check there must be at least two neighboring stations that meet the following requirements:

  • Within 30 km distance and 200 meters elevation

  • Greater than 3 years of historical data

  • Within 2 hours of the current value

  • Neighboring percentiles must agree with each other within <= 0.2

  • Cannot have any open QC flag ranges

Spatial value check

Is the value significantly different than neighboring observations?

API Name: sl_spatial_value_check

This flag is applied to multiple variables according to thresholds defined in the following table. Value thresholds indicate the threshold for flagging the current value, as: abs(value - median( neighbor values) >= threshold. Neighbor thresholds indicate the threshold for agreement between neighbor values, as: (max(neighbor values) - min(neighbor values)) <= threshold. Thresholds are not provided for pressure or relative humidity, as these variables are flagged via derived altimeter and derived dew point temperature, respectively.

Variable

Unit

Value threshold

Neighbors threshold

Altimeter

Pascals

3000

3000

Pressure

Pascals

Air Temperature

Celsius

10

15

Dew Point Temperature

Celsius

15

15

Relative Humidity

%

Wind Speed

m/s

20

15

Wind Gust

m/s

20

15

To run this check there must be at least two neighboring stations that meet the following requirements:

  • Within 30 km distance and 200 meters elevation

  • Closest-in-time values are within -1.5 hrs and +0.5 hrs of the current value

  • Cannot have any open QC flag ranges

Linear regression check

API Name: mw_multvariate_lin_reg_check

A multivariate linear regression check courtesy of the University of Utah MesoWest program. This check identifies air temperature, moisture, and pressure observations that appear to be significantly different than surrounding observations. More information on this check can be obtained here.

24-hr wind persistence check

API Name: mw_24h_wind_persistence_check

Persistence check to identify unchanging wind speed values, courtesy of the University of Utah MesoWest program. This check is redundant to the sl_pers_check.

UU2DVAR rejection check

API Name: mw_uu2dvar_rejection

A spatial comparison check with surrounding observations and gridded surface meteorological fields courtesy of the University of Utah Variational Surface Analysis (UU2DVAR). Observations that have been flagged by this check indicate they were statistically rejected from the UU2DVAR algorithm. For more information on UU2DVAR, please consult this publication.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.