ETL

Data Catalog: Series

608 views September 11, 2018 September 14, 2018 admin 0

In the UDM nomenclature, time series are formally called Series. Series must belong to Reports, but may belong to more than one Report. Likewise, most Reports present information on more than one Series.

Our Excel users can find the complete list of all Series in the UDM Data Catalog Excel file, under tab Series:

For ETL data pulls, the relevant OData feed with information on Series is /UDM_Catalog_Series. The fields are:

  • SeriesID (int, not null)
  • SeriesName (nvarchar(400), not null)
  • SeriesNameTranslated (nvarchar(400), not null)
  • SeriesNameCode (nvarchar(400), not null)
  • UnitID (int, not null, foreign key: UDM_Catalog_Units, field: UnitID)
  • Scale (int, not null)
  • LastModified (datetime2(6), not null)
  • Active (bit, not null)

SeriesID is the primary key in the table supporting this data feed. SeriesName is the name for the time series. We store all names in their original language. Each SeriesName is unique. SeriesNameTranslated provides the series name in English language. SeriesNameCode is the name of the series with all characters other than A-Z, a-z, 0-9 removed or replaced with underscore sign. For example, SeriesID 90165 has SeriesName MRiRW Serwatka w proszku, średnia ważona cena w euro, SeriesNameTranslated is MRiRW Whey Powder, Weighted Average Price in Euros, and SeriesNameCode is MRiRW_Whey_Powder_Weighted_Average_Price_in_Euros.

The field UnitID is the foreign key for /UDM_Catalog_Units, and identifies the unit of measure for each series in the database, such as $, %, $/cwt, MT, mil liters, etc. Scale defines the number of decimal points. For example, if Scale is 4, that means that the variable will be stored with precision of 1/10000, or 0.0001 units of measure. The maximum scale allowed in the UDM database is six. In the above example, UnitName is €/100kg.

The timestamp field LastModified can be used to facilitate incremental pulls. No record is ever hard-deleted from the underlying UDM database table. Instead, soft-deleted records have Active field set to 0. Soft-delete principle in conjunction with the timestamp of last change allows the users to optimize their data pulls by only requesting those fields that have been modified since the last time the data feed was pulled. For example, the request to the server may be:

http://udmdirect.dairymarkets.com/Universal/UDM_Catalog_Series?$filter=LastModified gt DateTime’2018-09-01T15:30:00.0000000′

or more simply:

http://udmdirect.dairymarkets.com/Universal/UDM_Catalog_Series?$filter=LastModified gt DateTime’2018-09-01′

Was this helpful?