Freelance retail data engineer in Lille: scoping reliable data flows

1. Start with a business decision and an observable scope

Start by selecting a workflow: making sellable stock visible, sending an order for fulfilment or publishing a product variant. Identify its producer, its consumers and the person authorised to resolve inconsistencies. An application can be authoritative for physical quantities without determining how much stock can be offered for sale.

Then examine representative messages, extraction schedules and available incident records. Establish where the final evidence resides: a receipt acknowledgement, a stored object or an accepted business status. Scoping must distinguish these stages. A successful technical response does not always prove that an order is ready for fulfilment.

• Define the objects, channels and locations covered, together with exclusions.
• Agree the expected data freshness and the time allowed to catch up after an interruption.
• Name the owners of business rules, corrections and operational support.

2. Write a data contract that teams can verify

A data contract describes the commitments between a producer and its consumers. For each important field, document its meaning, unit, origin and missing-value behaviour. For stock, distinguish physical, reserved and available-to-sell quantities. For orders, specify the identities of the channel, order and individual lines.

Add temporal rules: business event time, extraction time, time zone and object version. Does an absent value mean unknown, unchanged or deleted? Does a file represent a complete snapshot or only changes? These decisions prevent a partial delivery from being interpreted as an instruction to delete missing references.

JSON Schema can check types, required fields and whether additional properties are accepted. Supplement this structural validation with business rules. Version the contract and test each change with consumers: even a new optional field can cause problems for a strict reader.

3. Hypothetical example: an old stock position arrives after the new one

Consider a fictional retailer sending stock snapshots by item and warehouse. For reference SKU-A in warehouse DEPOT-N, version 41 reports eight sellable units; version 42 reports five. An interruption delays version 41, which arrives after version 42. These values are solely an illustration of the scenario.

Here, the contract specifies a source version that increases for each item-warehouse pair. The target applies five units, then ignores version 41 while recording the reason. The version check and write must form an atomic operation: two simultaneous workers must not each validate their own read and subsequently overwrite the result.

This rule suits an absolute stock position. For movements that add or subtract units, ignoring an older event could lose a necessary operation. Each movement must instead be identified, missing sequences checked and reconciliation planned. The choice between snapshots and movements therefore belongs in the contract.

4. Define evidence of reliability for each type of flow

Stock: measure the age of the position actually applied, across the relevant scope. Also define the commercial behaviour when that age exceeds the agreed threshold: temporarily retaining a value, restricting availability or suspending its exposure. The business team must make this decision.

Orders: reconcile source and target identifiers, lines, quantities and expected states. A partially accepted order needs an explicit handling rule. Define permitted transitions so that a late status cannot move a dispatched order back to an earlier state.

Catalogue: check product-variant relationships, channel requirements and withdrawal rules. Acceptance of an import can precede actual publication. Retain the target system’s detailed processing results and distinguish between an object being received, accepted and published.

5. Design recovery before going live

Classify failures as transient incidents, data requiring correction or uncertain outcomes. Temporary unavailability may justify a limited number of spaced retries; an invalid business identifier needs correction. Avoid multiplying attempts across the connector, service and scheduler.

A lost response after order creation leaves an uncertain outcome. Look up the target object or use an idempotency key: a stable identity that allows a request to be repeated without creating a second effect. Define its scope and retention period against the intended recovery horizon.

When you control the producing application, a transactional outbox can record the business change and the event awaiting publication together. A separate process then handles delivery. This option requires a suitable transaction boundary and dedicated operational support.

The recovery record should identify the affected objects, their known target state, the correction applied and the checks after replay. Selective recovery makes analysis easier; replaying a complete batch remains possible when its effects have been verified. Retain an audit trail without unnecessarily copying personal data from orders.

6. Balance frequency, complexity and operational capacity

Batch processing provides clear reconciliation points, at the cost of waiting between runs. Event-driven exchanges can reduce that wait, but require more decisions about sequencing and operations. Choose according to the freshness the business needs, application capabilities and the team available.

In a Semarchy xDI or Stambia environment supplemented by .NET services, assign each responsibility clearly: extraction, validation, target calls, tracking and recovery. Avoid having two components maintain competing states for the same delivery. The contract should remain understandable independently of the component executing it.

7. Turn acceptance testing into a reproducible demonstration

Prepare test data with inputs, initial state and expected results. Each scenario should produce evidence that can be inspected in the target and in execution tracking. Freshness and catch-up thresholds must be agreed with the business; no universal figure can replace that decision.

• Check missing fields, null values, units and incompatible versions.
• Send the same order twice and confirm a single business effect.
• Reverse two stock versions and verify the expected final state.
• Disconnect after the target write, then verify how the uncertain outcome is resolved.
• Introduce an invalid line and check exactly which records are rejected.
• Test partial cancellation, a withdrawn variant and a late status change.
• Replay corrected objects and reconcile identifiers, quantities and statuses.
• Simulate an interruption, then measure catch-up while new data continues to arrive.
• Have the person responsible for operations execute the recovery procedure.

8. Avoid scoping mistakes and prepare the handover

Common mistakes are concrete: replacing unknown data with zero, using receipt time as business sequence, deleting rejected records without resolution or validating only row counts. Two equally sized batches can contain different orders. Reconcile identifiers and the values that determine business behaviour.

When assessing an engagement, ask for a versioned contract, acceptance scenarios, reconciliation evidence and a usable recovery procedure. Include the necessary access and an owner for each alert. These deliverables make the scope easier to estimate and the handover verifiable.

Hugues Dehen, a freelance developer based in Valenciennes, brings together .NET, data integration and Semarchy xDI/Stambia at Sculpture Binaire. To scope your requirements in the Lille metropolitan area, prepare an anonymised flow example, its characteristic failure and the expected business outcome. These provide a concrete starting point for discussion.