Data engineering
Retail data flows: planning acceptance tests and recovery with a data contract
A job completing without errors is insufficient grounds for accepting a retail integration. You need a versioned contract and evidence of results in the receiving applications. This guide provides a completed stock contract example, its acceptance cases and a delivery approval record. Example values are fictional and must be replaced by rules agreed with your business stakeholders.
Start with the business decisions you need to protect
The first workshop brings together the owners of source and target applications, operations staff and a business representative. Choose a specific journey: publishing availability, transferring an order or making a product sellable. Identify the authoritative system for each piece of information and the person empowered to resolve disagreements.
Next, define acceptable data freshness, behaviour when updates are late and evidence of successful receipt. A positive technical response may only confirm that a request has been accepted. If the target processes it afterwards, acceptance testing must check the final outcome. This distinction prevents approval of a flow whose data remains blocked downstream.
The scoping deliverable is a specification for each flow: scope, owner, dependencies, timing objectives, rejection rules and acceptance criteria. Agree thresholds with business stakeholders based on their constraints. A number chosen solely to make testing easier is not a useful commitment.
Write a contract that defines what the data means
A schema describes structure; a contract also defines meaning. For inventory, specify whether the quantity represents physical stock, reserved stock or availability to sell. Define the item and location granularity, units, the meaning of zero and the meaning of a missing value. State whether each exchange replaces a current state or represents a movement.
For orders, document the business key, line identifiers and permitted status transitions. For catalogues, distinguish products, variants and commercial offers. A valid record in the reference system may still be unsuitable for publication on a channel: publication requirements must be independently testable.
The AsyncAPI specification describes messages, headers, payloads and correlation, among other things. It can formalise the technical part of an asynchronous exchange. Add responsibilities, business rules and service commitments. For a file exchange, a versioned column definition accompanied by the same rules may be sufficient.
- Identity: business key, event identifier and scope of uniqueness.
- Timing: business event time, receipt time and the rule for ordering versions.
- Quality: required fields, permitted values and handling of unknown references.
- Evolution: expected compatibility, affected consumers and the change procedure.
Deliverable: a completed stock contract to adapt
Fictional teaching example, AVAILABLE-STOCK contract v1. It covers an absolute snapshot per variant and warehouse. The producer, integrator and consumer receive the same document; changes are versioned and reviewed with them.
- Source and target: ERP to storefront; business owner: availability manager; support: named operations team.
- Key: variant + warehouse. Value: sellable quantity in units; zero means unavailable, a missing value is rejected.
- Message: stable identifier, increasing source version per key, business timestamp and absolute quantity. A product missing from a partial batch remains unchanged.
- Ordering: retain the latest version; different content carrying the same version raises a conflict for investigation.
- Recovery: select affected messages, preserve their identity and reconcile final state with the source before closure.
- Freshness criterion: agree a threshold with business stakeholders before approval; record commercial behaviour when it is exceeded.
- Evidence: contract v1 + inputs + initial state + target result + decision log + business owner approval.
Hypothetical example: inventory updates arriving out of order
Imagine a fictional retailer whose ERP publishes inventory snapshots to an online shop. For one variant and warehouse, version 42 reports seven available units. Version 43 reports four. After an outage, version 42 arrives a second time, after version 43. These figures illustrate a test scenario, not a project outcome.
The proposed contract specifies an increasing version number for each variant and warehouse pair. The target retains the applied version and ignores older snapshots. The expected result is therefore four units after all three messages arrive. Testing must also establish that two simultaneous operations cannot bypass this check: comparison and writing must be protected together.
This solution suits complete availability snapshots. For movements that must be added together, ignoring every older event could lose a legitimate adjustment: identify movements already applied and detect gaps instead. Likewise, a timestamp alone does not establish a reliable order when several producers use clocks that are not coordinated.
Treat duplicates, ordering and write confirmation separately
Acceptance cases distinguish three properties: a repeated request creates no additional effect, an older version does not replace a newer state, and a lost confirmation leads to retrieving the outcome or investigating. Passing one test does not establish the other two.
Document the target’s ability to retrieve an operation and identifier retention. If a remote API cannot establish the outcome, the contract needs reconciliation and an investigation state before another creation attempt.
Prepare selective recovery that operators can monitor
Classify incidents before defining retries. Temporary unavailability may justify limited attempts with delays between them. An unknown item code calls for a correction or the arrival of the required reference data. A lost response requires checking whether the write succeeded. Retrying all three situations indiscriminately makes diagnosis harder.
In Azure Service Bus, a dead-letter queue allows inspection of unprocessed messages and supports organised resubmission; it is not cleaned up automatically. Whatever tool you use, assign an owner to rejected records, provide an actionable reason and define how each case is closed.
The recovery procedure specifies which objects to select, which dependencies to restore and which checks to perform before replay. Preserve business identity while recording each attempt. Include a preview of the selected scope and a throughput limit: a large recovery run should not prevent new orders from progressing.
Balance freshness, throughput and complexity
Frequent updates can reduce delay but place more demands on the target. Batch processing simplifies some reconciliation tasks at the cost of additional waiting time. Decide by use case: the acceptable delay for enriching a product description may differ from that for publishing commercial availability.
Measure the age of business data and the oldest pending object as well as execution times. Fast processing of stale data remains inadequate. Also prepare periodic source-to-target comparisons, using a consistent time boundary to avoid reporting false discrepancies while updates are in progress.
In a Semarchy xDI or Stambia environment, keep validation rules and recovery procedures easy to identify. If a .NET component supplements the setup, make its responsibility explicit. Distributing the same rule across several processes increases the risk that implementations diverge when a correction is made.
Acceptance checklist before production deployment
Each test case should combine input data, an initial state, an action and an expected result in the target. Retain evidence alongside the contract and processing versions. Matching row counts are insufficient: two orders might be swapped, or an incorrect value might have replaced the correct one.
- Normal operation: verify business values, identifiers and the final status in every target.
- Duplicate: resend the same event and confirm there is no second business effect.
- Out-of-order delivery: reverse two versions and check the final state against the contract.
- Interruption: stop after the target write but before confirmation, then recover.
- Partial rejection: mix valid and invalid objects and account for every outcome.
- Missing dependency: send an order with an unknown reference, then resolve it.
- Evolution: test an added field, a missing required field and an unknown value.
- Catch-up: replay a backlog alongside live traffic and measure freshness and throughput.
- Reconciliation: identify missing, additional and inconsistent objects.
- Operations: have the team responsible for support execute the recovery procedure.
Approve delivery using shared evidence
Common mistakes include testing only normal operation, confusing receipt with integration and postponing recovery design until after launch. Request a delivery package containing the versioned contract, acceptance results, known discrepancies and operating procedures. Rolling back application code does not automatically correct data already written: plan how to handle that data.
To prepare acceptance testing with Sculpture Binaire, gather an existing contract, an anonymised exchange, observed incidents and business criteria. Hugues Dehen, based in Raismes near Valenciennes, can help define the evidence record and necessary changes to the flows.