Synoptic uses the term “latency” to refer to the period of time between when a sample is measured and when it is available to end users from Synoptic services. There are two contributors to latency:
-
Provider latency, the amount of time it takes the upstream events to make the data available to Synoptic. Our data teams work with providers to help them minimize this time by reducing upstream processing, or using simpler techniques. Ultimately providers control this latency. Synoptic works diligently to acquire data with the lowest-latency available.
-
Acquisition latency - the time between when the data was available, and when synoptic acquired it. Certain inefficient delivery methods (such as providing multiple days of data per push) also result in significant re-processing of existing data, adding seconds of latency. Synoptic can tune scheduling and optimize processing to minimize this time, but it is a function of the provider.
-
Synoptic platform latency. This is the factor Synoptic maintains full responsibility for. Synoptic employs a distributed event-based architecture to process billions of daily observations with under 10 seconds of total latency. During this time data are organized into our foundational data model, given quality checks, including spatial and temporal comparisons, and made available to streaming (e.g. push streaming), request-based ( Weather API) applications, and other data delivery systems.
The total latency for any observation can be queried from Weather API’s Latency data service. You can also view an aggregated representation of latency per network on the Data Availability Dashboard .
How do I minimize latency when accessing data?
Synoptic’s Push Streaming is the minimum-latency mechanism to access observation data. This is a WebSocket-based utility for receiving observations within 2 seconds of their deliverable accessibility.
Some users will thrash Weather API to achieve minimal latency without the push streaming service. This is generally difficult to stay within concurrency requirements, and will perform poorly for more than a small number of stations.
Factors that cause latency to vary
Latency varies from our upstream sources for a variety of reasons. Shared systems or dependencies on networking managed by others is a common source of delay. Scheduling and conflicts within upstream systems can also delay observations. Issues at the observing station or telemetry - including bandwidth limitations, and variable sizes of reports (some stations report more variables at certain times) will also alter the time it takes for data to be available to Synoptic.
Once received by Synoptic, we added latency should not vary substantially, as our platform elastically scales to meet demand. Periods were a significant number of reports are received faster than our scale-in capacity (such as short term backfills) may result in brief slowdowns on the order of seconds. Synoptic is constantly innovating on our platform to further minimize potential latency impacts.
Synoptic will alert our http://status.synopticdata.com page if there are factors within our control causing an increase to latency beyond several minutes.
Synoptic does not currently publish any general latency metrics, due to the fact almost all latency experienced by customers is upstream.
Refer to Dataset-specific documentation to learn more about latency characteristics of selected datasets.