Data Formatting Best Practices

Synoptic is extremely flexible regarding accepted data representations. We have a team of data scientists and engineers on staff who use powerful tools and cloud technology to quickly understand and ingest any data format. This is how Synoptic has been able to aggregate legacy TAC formats like METAR, SHEF, and SYNOP, binary formats like NetCDF, BUFR, and HDF, and structured formats like CSV, JSON, and XML.

We receive these formats of data in a variety of ways. Most commonly this will be data files, exchanged by Pushing data via Amazon S3 or classic S/FTP, LDM. We also perform ingest via streaming methods, including MQTT and WebSocket. Additionally we pull data in many formats from various forms of web endpoints, such as APIs.

Across all these techniques, there are a few best practices for downstream usability of your data.

1. Use metric units

Synoptic’s system, and many downstream applications require the use of metric or SI units. For the most part Synoptic’s internal system will store your observation data in its metric units (see our variable list to see our storage units). When we convert units we attempt to retain your precision where possible, but some conversions are imprecise (such as Fahrenheit to celsius). Sending data as metric allows us to skip this change. Data are re-converted to English or other units by our API and other delivery services.

2. Use predictable structure keys

When providing structured data, such as JSON or XML, use structures which provide constantly predictable keys. This means the data is structured and repeatable. It also may produce a less efficient structure, for instance

CODE

{
  "temperature_1":12,
  "temperature_2":15,
  "wind_speed_1":5
}

Conversely, a structure which uses predictable keys for this data would be

CODE

{
  "measurements":[
  {
    "variable":"temperature",
    "sensor":1,
    "value":12
  },
  {
    "variable":"temperature",
    "sensor":2,
    "value":15
  },
  {
    "variable":"wind_speed",
    "sensor":1,
    "value":5
  },
  ]
}

This is less efficient, yes, but is much easier to read with software.