1 of 5

Synchronization

Let Ergonode orchestrate the synchronization process and worry only about your business logic

At the heart of every PIM-including ecosystem, there's data integration.

A need to keep your data well-organized thanks to PIM and easily accessible to eCommerce system(s).

We do realize that creating the synchronization process is not an easy task. At the very minimum, it most usually requires:

a queue system like RabbitMQ, Amazon SQS, etc
a worker
well-developed and robust logic to handle the process
- progress state
- both process and entry statuses
- history
- frontend application displaying all that data
persistent storage i.e. database
business logic implementation

which is a lot and usually not that easy to do, especially, in the distributed environment of multiple services.

That's why at the core of the Apps engine we created an entire synchronization process orchestration with the intent of you only being left with implementing your desired business logic and the minimum effort needed around it.

Need only an incremental export of data you have recently changed? We've got you covered.

Does it have to handle full export each time? It's possible with an App.

Creating synchronization App

To start with a synchronization App enable at least one of those features in your Manifest.

synchronization - enables the basic integration. During every synchronization, you'll only receive data changed from the last iteration
synchronization_full - enables the possibility of running full integration on every execution

Both features may be enabled next to each other or you can pick just one.

If your App generates a file, enable features according to your needs

synchronization_file_download - enables file download for each synchronization
synchronization_file_download_latest - generates constant URL under last-generated file will be available for feed-like functionalities

Data flow

Overview of the data flow during synchronization process

Synchronization start

This phase is optional based on events the App subscribes to.

Status: Preparing

Data preparation

This is the internal phase.

Status: Preparing

Apps engine gathers the data required for the synchronization process.

This process can take some time. Its length depends on the amount of data to process and the cache size needed to be built.

Most usually it'll be the longest on the first run of the synchronization and significantly shorter on the following one, even if it is the same full synchronization.

Planning

This is the internal phase.

Status: Preparing

In this phase synchronization plan is created resulting in entries in the history created and put in Planned status.

Attributes synchronization

This phase is optional based on events the App subscribes to.

Status: Execution

Firstly, as Attributes are the base of the Ergonode data model, their data is synchronized.

Categories synchronization

This phase is optional based on events the App subscribes to.

Status: Execution

As a second Categories data is delivered to an App.

Products synchronization

This phase is optional based on events the App subscribes to.

Status: Execution

As products utilize all of the previously synchronized resources they are the last ones to be sent.

Products relations synchronization

This phase is optional based on events the App subscribes to.

Status: Execution

Lastly, relation data is being sent - variants, children assignment, and product relation attribute values.

This is to avoid circular reference problems where product A depends on B and B on A - all the products should exist in this phase already.

Relations payloads are always provided with product_updated events.

Synchronization end

This phase is optional based on events the App subscribes to.

Status: Finished

Synchronization endpoints

A place where the synchronization business logic is implemented

Synchronization endpoints have a very similar role to message handlers in a message queue system.

They receive the event(message) with data and react to its payload.

There are several endpoints corresponding to events in the Manifest configuration.

Each endpoint receives an appropriate X-APP-TOKEN HTTP header with JWT that allows authentication of the caller.

JWT for all synchronization endpoint requests is extended with an extra context claim synchronization_id.

This claim is a UUID and allows identification of the synchronization process.

Every endpoint should respond with a 2xx HTTP code to be treated as a success.

Synchronization endpoints

All the endpoints follow the semantic [PUT] /consume/{event}.

synchronization.events contains an array of detailed payloads containing actual changes - the full list can be found here.

Those events allow retrieving information about what has changed and reacting to that change.

This field for DELETED events is always an empty array.

/consume/attribute_created

subscribed event: attribute_created

example payload:

{
  "name": "attribute_created",
  "resource_id": {
    "id": "Attribute_code",
    "type": "attribute_code"
  },
  "synchronization": {
    "resource_customs": null,
    "events": []
  }
}

/consume/attribute_updated

subscribed event: attribute_updated

/consume/attribute_deleted

subscribed event: attribute_deleted

example payload:

{
  "name": "attribute_deleted",
  "resource_id": {
    "id": "Attribute_code",
    "type": "attribute_code"
  },
  "synchronization": {
    "resource_customs": {"your": "custom"},
    "events": []
  }
}

/consume/category_created

subscribed event: category_created

example payload:

{
  "name": "category_created",
  "resource_id": {
    "id": "Category_code",
    "type": "category_code"
  },
  "synchronization": {
    "resource_customs": null,
    "events": []
  }
}

/consume/category_updated

subscribed event: category_updated

example payload:

{
  "name": "category_updated",
  "resource_id": {
    "id": "Category_code",
    "type": "category_code"
  },
  "synchronization": {
    "resource_customs": {"your": "custom"},
    "events": []
  }
}

/consume/category_deleted

subscribed event: category_deleted

example payload:

{
  "name": "category_deleted",
  "resource_id": {
    "id": "Category_code",
    "type": "category_code"
  },
  "synchronization": {
    "resource_customs": {"your": "custom"},
    "events": []
  }
}

/consume/product_created

subscribed event: product_created

example payload:

{
  "name": "product_created",
  "resource_id": {
    "id": "Product SKU",
    "type": "sku"
  },
  "synchronization": {
    "resource_customs": null,
    "events": []
  }
}

/consume/product_updated

subscribed event: product_updated

example payload:

{
  "name": "product_updated",
  "resource_id": {
    "id": "Product SKU",
    "type": "sku"
  },
  "synchronization": {
    "resource_customs": {"your": "custom"},
    "events": []
  }
}

/consume/product_deleted

subscribed event: product_deleted

example payload:

{
  "name": "product_updated",
  "resource_id": {
    "id": "Product SKU",
    "type": "sku"
  },
  "synchronization": {
    "resource_customs": {"your": "custom"},
    "events": []
  }
}

/consume/synchronization_started

subscribed event: synchronization_started

example payload:

{
    "name": "synchronization_started"
}

/consume/synchronization_ended

subscribed event: synchronization_ended

example payload:

{
    "name": "synchronization_started"
}

Resource customs

In the above examples, you might have noticed the resource_customs field.

This mechanic allows storing small pieces of information as JSON in the engine.

The most common use case for this would be storing the ID of the eCommerce entity next to the resource. It'll be passed with every following request so it's easier for your business logic to create the requests to the eCommerce system without the need of reaching to the database.

To persist the custom in the response of the endpoint add the following payload

{
  "resource_customs": {
    "id": 1,
    "other_relevent_information": "here"
  }
}

The resource custom will only be persisted on 2xx(non-204) HTTP responses.

Errors handling

Handling of the request payload may fail.

In such case, there are a couple of scenarios:

failed connection(including 502 and 503 HTTP errors) - the event will be retried a couple of times before aborting completely
5xx and 3xx HTTP responses
- for a single resource, the event is not retried as of an internal error and the history entry is marked with a generic message
- for a synchronization error event is going to be retried eventually leading to the failed synchronization status
4xx HTTP responses
- for a single resource
  - a custom behavior may be applied via an appropriate response
    { "custom_message": "your custom error message", "retryable": false }
    retryable parameter determines whether the event can be retried. Otherwise, the fail and custom error message will be applied immediately
  - the event is not retried and the history entry is marked with a generic message
- for a synchronization error event is going to be retried eventually leading to the failed synchronization status

File download endpoint

If an App purpose it to generate a file as a result of synchronization enable any of the following features based on your needs

To allow access to Ergonode for generated files you need to implement the following endpoint.

/download-file

Use synchronization_id claim from JWT to determine which file should be returned.

The endpoint should return the binary file.

The response can contain content-type and content-length HTTP headers so that those are passed to the browser on file download.