Migrating from DataStage to Semarchy xDI: the freelance guide

Many IT departments today inherit an IBM DataStage estate built over ten or fifteen years: hundreds of jobs, nested sequences, business logic scattered across stages and scheduling scripts. Licence costs, scarce skills and the desire to push transformation down into the database are leading more and more companies to evaluate Semarchy xDI (formerly Stambia). The question is then no longer "is it possible", but "how do we switch over without breaking Monday morning's reporting chain". Here is what a DataStage to Semarchy xDI migration really involves, the pitfalls that cost the most, and the method I apply as a freelancer on this type of project.

Two very different integration philosophies

The first mistake is to treat this migration as a simple tool-to-tool translation. DataStage is a classic ETL: data flows through a proprietary execution engine that performs the transformations before loading the target. Semarchy xDI is a code-generating ELT: the tool does not move the data, it generates SQL (or native code) executed by the target database itself, from reusable components called templates.

This difference has very concrete consequences. A DataStage job heavy on sort, aggregate and lookup stages has no "stage for stage" equivalent: its logic must be rethought as a set-based SQL problem, which is often much faster, but requires understanding what the job actually did - not just what it looked like it did. Conversely, the error-recovery, reject and logging mechanisms, highly codified in DataStage, must be rebuilt using xDI's own mechanisms (variables, actions, template-level reject handling).

Step 1: map before you convert

No serious migration starts by opening the designer. It starts with an exhaustive inventory of the existing estate, extracted from DSX exports or the DataStage repository: list of jobs and sequences, real sources and targets, parameters, scheduling dependencies, run frequencies and data volumes.

This work almost always holds the same surprise: a significant share of the jobs is dead or redundant. Test jobs left in production, processing whose target nobody reads any more, duplicates left over from an old reorganisation. Cross-referencing the inventory with the last twelve months of execution logs and the actual usage of target tables often takes 20 to 40% out of scope before you even start. It is the best-spent euro of the project: every job not migrated is a job that will never need testing, nor maintenance.

Step 2: build the xDI foundation before the flows

The temptation to migrate the first job in the first week is strong. It is almost always a mistake. What drives the total cost of an xDI migration is not the first flow: it is the conventions laid down before it.

• Metadata and models: reverse-engineering source and target schemas, stable naming, environment handling (dev / test / production) through variables rather than duplicated projects.
• Templates: choosing the standard integration templates, and above all identifying the rare cases that justify a custom one. A poorly mastered in-house template is technical debt that survives for ten years.
• Logging and rejects: a single log and reject schema, applied to every flow, so that operations teams can read any process without knowing it beforehand.
• Delivery: exporting xDI deliverables, Git versioning and automated deployment, ideally through an Azure DevOps pipeline - the most frequently forgotten topic in ETL migrations, and the one that makes the difference two years later.

Step 3: migrate in batches, never big bang

The most effective breakdown follows functional domains, not technical difficulty: a first self-contained batch (reference data, for example), then the flows that depend on it. Each batch follows the same cycle: conversion, unit testing, parallel run, business validation, cut-over.

The parallel run is the only genuinely reliable testing technique on an ETL migration: for several cycles, the old DataStage job and the new xDI flow run side by side on the same data, and the targets are compared row by row. The discrepancies found at that point are precious - they almost always reveal an implicit business rule written down nowhere: a rounding rule, a date filter, a null-handling behaviour inherited from a 2014 fix. Finding those gaps during testing costs half a day; finding them after cut-over costs user trust.

The pitfalls that cost the most

• Forgetting the scheduler. Dependencies between jobs often live outside DataStage, in Control-M, an in-house scheduler or shell scripts. Migrating the flows without migrating the execution plan means delivering half a project.
• Reproducing row-by-row logic. Transposing iterative DataStage logic into an xDI loop gives a slow, unreadable flow. ELT is set-based thinking; that is where the 10x gains in processing time are made.
• Underestimating the target database. Since transformation runs in the database, the migration shifts the load onto it. Indexes, statistics, partitioning and execution windows must be reviewed - with the same care as the flows themselves.
• Not transferring skills. A migration that succeeds technically but is never handed over creates a dependency. Skills transfer to the internal team is part of the project, not an afterthought.

How long, and with which profile?

On the estates I have worked with, a standard flow (one to three sources, one target, classic transformations) is converted and tested in half a day to two days once the foundation is in place. Complex flows - dense business logic, history management, reconciliations - naturally take longer. The useful order of magnitude is therefore not "the number of jobs", but the number of genuinely live jobs after pruning, weighted by complexity: that is exactly what the mapping phase produces, and what allows honest estimating.

In terms of profile, this project requires a rare double skill set: knowing the source ETL well enough to read an undocumented legacy job, and mastering xDI well enough not to reproduce the old tool's habits in it. That is precisely the ground I cover day to day - you will find the details in my articles on being a freelance Semarchy xDI consultant and a freelance Stambia developer, as well as on my expertise page.

Freelancer or systems integrator for this project?

A DataStage to xDI migration is a bounded, highly technical project whose value comes from deep knowledge of both tools, not from headcount. Going through a freelancer removes the layers of intermediaries: you talk directly to the person converting the flows, start-up takes days, and the budget matches real production time. On larger scopes, the model that works best remains an experienced freelancer who lays the foundation, industrialises the conversion and trains the internal team that will take over.

Remote work and availability

This type of assignment works very well remotely: mapping from exports, development on your environments, continuous reviews and demos. I work remotely for companies all across France, and I can be on site in the Hauts-de-France region (Valenciennes, Lille, Douai, Cambrai, Maubeuge) when the context requires it - scoping workshops, cut-over committees, skills transfer.

In summary

Migrating from DataStage to Semarchy xDI is not a translation, it is a paradigm shift - from engine-based ETL to code-generating ELT. Successful projects share four reflexes: map and prune before converting, lay the xDI foundation before the first flow, migrate in batches with a systematic parallel run, and treat scheduling and the target database as first-class components. Done properly, this migration does more than replace a tool: it cuts the bill, speeds up processing and finally makes the flow estate readable.