Syndication Exports v3

The API enables partners to build their own navigation and representation of solute's price comparison data. It is accessed via an HTTPS connection using Basic Authentication (RFC 2617).

Export Description

The export feed structure is made up of base and delta exports. The delta exports are generated every 20 minutes and include only changes to the previous data state. The base exports are generated in much larger intervals including all data and represent the new reference point for the following changes.

The export files are available for download via HTTPS with this API (see section Exports).

File and Export Types

The export files are files in the format returned by the API. Each file has got its own data structure in which related info is stored.

If the format is JSON, each line of the file contains a valid JSON-object. Every returned file is UTF-8 encoded. If the format is CSV, the file starts with a header row with the names of the columns. The columns are semicolon-separated.

There are following types of export files. Some are delivered in as base and also as delta.

  • offersv3: base and delta (JSON)
  • shopsv3: base (JSON)
  • clicklog_export: daily-base (CSV)

Detailed explanation for all files:

  • An "Offers Base Feed" includes all offers.
  • An "Offers Delta Feed" includes only offers which were deleted, added or updated after the previous last base/delta.
  • A "Shops Base Feed" includes all shops. You are only allowed to deliver traffic to shops which are included in this feed with is_online = True.
  • A "clicklog_export Base Feed" includes all clicks you generated on a specified day.

In addition to those features, a cleaning process regularly deletes old files. This process deletes all export files (both delta feeds and base feeds) except for the newer files (~last 14 days).

Column Names & Order

The content of the exports are configured individually.

How To Use

  1. Use the exports endpoint of the API to get a list of available files.
  2. Download the file using export-files endpoint of the API to download the export.
  3. Execute a full import of the recent base export.
  4. Bind the shops and offers data together.
  5. Watch out for the next offers delta feed and shops feed.
  6. Evaluate the operations and time stamps in the delta feed (offers).
  7. If the operation is “delete”, exclude the data entity from your system. If the operation is “update”, update all of the data related to the given feed info. There is no indication which part of the data has changed. If the operation is “add”, insert the new data into your system. Sometimes it is possible that a "update" never had a "insert" row.
  8. Check if the shops are still online by using the shops base feed. There is no indication in the offers delta feed if the offer is online or offline. This is because an offer can't go offline independently from its shop or from the product that the offer belongs to. Exclude all offers whose shop is not listed in the current shops base feed or where is_online isn't True.
  9. Do the same procedure until the next complete base feed generation comes up. This workflow is suggested but not mandatory.

Every time a new base gets created, you have to import this new base. Based on different circumstances, there is the possibility that the consumed deltas aren't resulting in the next base - so consuming the bases is mandatory.

Get the next export to apply

We provide the endpoint /exports/next to help our customers keep their data up to date. The endpoint needs to know which data-source the customer is interested in (shopsv3, offersv3 or clicklog_export) and uses a token to obtain the next export to handle.

Usage

Initial base

The request for an initial base export is done by requesting the /exports/next endpoint without a token.

If an initial base is available, the export info of the base to download and a new token are returned. If no initial base is available the status code of the response is 204. No export info and no token are returned in this case. This should be repeated (at an interval of around 5 minutes) until the status code of the response is 200. In this case, the export info of the initial base export and a token are returned.

Each id-item inside export_file_ids of export-info is used to download the export using the /export-files/{export_file_id} endpoint. The token should be stored by the customer for future requests to the /exports/next endpoint.

Get next export

The current token is used to request the /exports/next endpoint for newer exports.

If there is no newer export the status-code of the response is 204. In this case request /exports/next again with the token until the status code of the response is 200 and a new export-info and a new token are returned (at an interval of around 5 minutes).

Each id-item inside export_file_ids of export-info is used to download the export-file using the /export-files/{export_file_id} endpoint. The new token should be stored by the customer. It is required for future requests of the /exports/next endpoint.

Whenever you want to search for newer exports use the previously collected token and make a request to the /exports/next endpoint. If status-code of the response is 200 a newer export-file is available and a new export-info and a new token are returned. Call the /export-files/{export_file_id} endpoint for each id inside the export_file_ids of export-info to download the export-file(s). Now update the stored token with the new one for next request.