INNORIX in Practice, Multi-Stage Flow

Once again, not all dataarrived on time today.

A company’s data analytics work starts at 8 a.m. every day.Five systems prepare the data it needs, but once again, each one is ready at a different time.

Scenario
Through a fictional company whose data is ready at a different time every day, we look at the run conditions between jobs and how data moves.

08:00MONDAY MORNING

Today’s data readiness

Transaction systemReady07:12
Customer systemReady07:38
Service operations systemReady07:51
External data systemTransferringTBD
Data processing systemProcessingTBD
Scheduled analytics start
08:00
Data ready
3 / 5
Actual work status
Waiting for data
  1. 07:00Data created
  2. 08:00Analytics scheduled
  3. 09:00Downstream work
  4. 12:00Results in use

It’s time to start the analytics work. But data from two systems still isn’t ready.

From Monday morning to Tuesday morning, we follow the company’s data and work.

INNORIX in Practice

Monday morning07:00

Monday 07:00

Five systems start their day.

Every morning, the company uses the data created the day before for customer service, operations analysis, business analytics and more.

To support this, five separate systems each produce their own data.

MONDAY, 07:00

Data needed this morning

Source systemToday’s dataExpected readyExpected size
Transaction systemTransaction records07:1584 GB
Customer systemCustomer changes07:3012 GB
Service operations systemService usage logs07:45264 GB
External data systemInbound external data07:5048 GB
Data processing systemProcessed business data07:55186 GB

Today’s schedule

  1. 08:00Update customer service data
  2. 08:00Analyze service operations
  3. 08:00Integrated analytics on all data
  4. 10:00Use analytics results
  5. 12:00Wrap up morning work
If all goes to plan, every dataset is ready before 8 a.m.

Live contextNow

07:00Monday morning

Data operations team07:00

We have started preparing the data for this morning’s analytics.

Data readiness0 / 5

  • Transactions0%
  • Customer data0%
  • Service ops0%
  • External data0%
  • Processed results0%

Downstream jobs now

  • Customer serviceWaiting
  • Operations statusWaiting
  • Integrated analyticsWaiting

Monday 07:51

Data for some of the work is ready.

Transaction data was ready at 7:12 a.m. Customer data and service operations data followed one after another.

But not every job at the company needs all five datasets.

DATA REQUIREMENTS

Each job needs different data.

Required dataCustomer serviceOperations statusIntegrated analytics
TransactionsReadyReady—Ready
Customer dataReadyReady—Ready
Service ops dataReady—ReadyReady
External dataPreparing——Waiting
Processed resultsPreparing——Waiting
Start condition2 / 21 / 13 / 5

Customer service job

Ready to run

2 of 2 required data ready

Operations status job

Ready to run

1 of 1 required data ready

Integrated analytics job

Waiting

2 required data not ready

The data for integrated analytics isn’t ready yet,but other jobs can already start.

Live contextNow

07:51Monday morning

Service operations team07:51

All data for customer service is ready.

Data readiness3 / 5

  • TransactionsDone
  • Customer dataDone
  • Service opsDone
  • External data48%
  • Processed results32%

Downstream jobs now

  • Customer serviceReady to run
  • Operations statusReady to run
  • Integrated analyticsWaiting

Monday 08:00

It’s the scheduled start time.

It’s 8 a.m.

The external data system is still transferring over the network, and the data processing system is handling more data than usual.

The company’s integrated analytics needs all five datasets.

08:00

MONDAY MORNING

Data ready
3 / 5
Scheduled start
08:00
Integrated analytics delivery
Waiting

Analytics operations team

Some data still isn’t ready. Integrated analytics is on hold.

Reaching the start time and being able to start are two different conditions.

Live contextNow

08:00Monday morning

Analytics operations team08:00

External data and processed results are not ready yet. Integrated analytics is on hold.

Data readiness3 / 5

  • TransactionsDone
  • Customer dataDone
  • Service opsDone
  • External data68%
  • Processed results42%

Downstream jobs now

  • Customer serviceCompleted
  • Operations statusIn progress
  • Integrated analyticsWaiting

Monday 08:24

Should every job wait for the last dataset?

External data has arrived. But the last dataset integrated analytics needs is still being processed.

Meanwhile, the customer service and service operations data has already been delivered to their own environments.

MONDAY, 08:24

The three jobs are in different states.

Customer service data delivery

Completed

07:38 — 07:42

Required data

  • TransactionsCompleted
  • Customer dataCompleted

Destination

Customer service server

2 / 2required data ready

Operations data delivery

Completed

07:51 — 08:13

Required data

  • Service ops dataCompleted

Destination

Operations analytics server

1 / 1required data ready

Integrated analytics data delivery

Waiting

Waiting for run conditions

Required data

  • TransactionsCompleted
  • Customer dataCompleted
  • Service ops dataCompleted
  • External dataCompleted
  • Processed resultsWaiting

Destination

Integrated analytics environment

4 / 5required data ready

When one job is delayed,unrelated jobs don’t have to wait with it.

Live contextNow

08:24Monday morning

Customer service team08:24

The customer service files have been delivered. We’ll check the integrated analytics results later.

Data readiness4 / 5

  • TransactionsDone
  • Customer dataDone
  • Service opsDone
  • External dataDone
  • Processed results76%

Downstream jobs now

  • Customer serviceCompleted
  • Operations statusCompleted
  • Integrated analyticsWaiting

Monday 08:47

The last required data is ready.

The data processing system has produced its last result file.

But a file existing doesn’t mean the data is fully ready. The job writing it may still be running, or only some of the related files may have been created.

So the company has defined, for each system, how to confirm that its data is ready.

DATA READINESS

When a file appears vs. when the data is ready

08:43

Writing files

/processed/2026-11-16/

  • result_001.parquetCompleted
  • result_002.parquetCompleted
  • result_003.parquetWriting
Next transfer waiting

08:47

Data ready

/processed/2026-11-16/

  • result_001.parquetCompleted
  • result_002.parquetCompleted
  • result_003.parquetCompleted
  • _SUCCESSDone marker
Next transfer running

Daily integrated analytics

INNORIX Flow

Data conditions required to run

  • Transactions deliveredMet
  • Customer data deliveredMet
  • Service ops data deliveredMet
  • External data deliveredMet
  • Processed results readyMet

Run conditionAll required conditions met

Delivery to the analytics environment started

Only after confirming the data the work needs is readydoes the next data movement begin.

Live contextNow

08:47Monday morning

Data operations team08:47

All required data is ready. Sending it to the analytics environment.

Data readiness5 / 5

  • TransactionsDone
  • Customer dataDone
  • Service opsDone
  • External dataDone
  • Processed resultsDone

Downstream jobs now

  • Customer serviceCompleted
  • Operations statusCompleted
  • Integrated analyticsTransferring

Monday 09:06

Analytics results aren’t ready all at once either.

At 8:49 a.m., the required data reached the integrated analytics environment, and the existing analytics system began processing it.

This system, too, produces different kinds of results at different times.

MONDAY, 09:06

Three jobs waiting on analytics results

The company’s existing analytics system

Daily analytics processing

Per-customer resultsGenerated
Per-service resultsGenerating72%
Full integrated resultsGenerating38%

Per-customer results

09:06

Transfer started

Customer service environment

Per-service results

Preparing

Waiting

Service management environment

Full integrated results

Preparing

Waiting

Business analytics environment

Even before the analytics system finishes everything,jobs that need a finished result carry on.

Live contextNow

09:06Monday morning

Analytics team09:06

Per-customer analytics results came out first.

Data readiness5 / 5

  • TransactionsDone
  • Customer dataDone
  • Service opsDone
  • External dataDone
  • Processed resultsDone

Downstream jobs now

  • Customer serviceCompleted
  • Operations statusCompleted
  • Integrated analyticsGenerating results

MONDAY, 10:15

This morning’s work seemed to be done.

  • Customer dataCompleted
  • Service operationsCompleted
  • Integrated analyticsCompleted
  • Analytics resultsCompleted

Transaction system team10:15

Some of the transaction records we sent this morning need corrections.We’re sending the corrected transaction data. Please apply it to the morning analytics results too.transaction_correction_1015.csv286 MB, newly created file

Does finished work have to start over from scratch?

INNORIX in Practice

Monday morning07:00

Monday 10:15

Unscheduled data arrives.

Some transaction records have been corrected.

The company already has analytics results built from the morning data. Resending all the source data and rerunning the entire analysis is not what it needs.

The existing analytics system already has a separate job for processing correction data.

ADDITIONAL DATA

New data needs its own processing conditions.

Scheduled morning work

594 GB

All source data

  • TransactionsCompleted
  • Customer dataCompleted
  • Service ops dataCompleted
  • External dataCompleted
  • Processed resultsCompleted

Earlier deliveries stay as they are

Extra correction job

286 MB

Newly created correction data

  • transaction_correction_1015.csvCreated 10:15
  • Correction-only Flow runs
  • Delivered to the correction system

What follows

  1. 10:15Correction data created
  2. 10:16Delivered to the correction environment
  3. 10:24Existing system processes the correction
  4. 10:28Corrected analytics results generated
  5. 10:29Results delivered to related services
Recurring work and exceptional workcan be set up in different ways.

Live contextNow

10:15Monday

Transaction system team10:15

Some of today’s transaction records need correcting. We’ve sent a correction file.

Data readiness5 / 5

  • TransactionsDone
  • Customer dataDone
  • Service opsDone
  • External dataDone
  • Processed resultsDone

Downstream jobs now

  • Customer serviceCompleted
  • Operations statusCompleted
  • Integrated analyticsApplying correction

Additional data

Correction data286 MB

Monday 11:40

How did today’s work actually go?

The operator reviews the results of the morning’s work.

Each job started and finished at a different time than planned, and an unscheduled extra job came up too.

MONDAY, 11:40

Scheduled work vs. what actually ran

  • Data delivery
  • Existing analytics processing
  • Extra correction job

INNORIX STUDIO

Today

Runs

FlowStartedCompletedStatus
Customer data07:3807:42Completed
Service operations07:5108:13Completed
Integrated analytics08:4708:49Completed
Analytics results09:0609:50Completed
Correction data10:1510:16Completed
Corrected results10:2910:31Completed

Data operations team

This morning’s data deliveries and the extra correction are both complete.

Today’s work didn’t follow the schedule,but each job ran once the data it needed was ready.

Live contextNow

11:40Monday

Data operations team11:40

We’ve confirmed the morning data delivery and the applied corrections.

Data readiness5 / 5

  • TransactionsDone
  • Customer dataDone
  • Service opsDone
  • External dataDone
  • Processed resultsDone

Downstream jobs now

  • Customer serviceCompleted
  • Operations statusCompleted
  • Integrated analyticsCompleted

THE NEXT DAY, Tuesday 07:32

Today was a little different from yesterday.

It’s the next morning.

There was less data to process than usual, and external data arrived early. All the required data from the five systems was ready at 7:32 a.m.

Jobs with no fixed run time don’t need to wait until the scheduled 8 a.m.

MONDAY & TUESDAY

Same work, different start times

Monday

08:47

All required data ready

Ready 47 minutes later than scheduled

Tuesday

07:32

All required data ready

Ready 28 minutes earlier than scheduled

The run conditions are the same as yesterday.Only the time they were met has changed.

Live contextNow

07:32Tuesday morning

Analytics operations team07:32

All data was ready early today. Starting the analytics work.

Data readiness5 / 5

  • TransactionsDone
  • Customer dataDone
  • Service opsDone
  • External dataDone
  • Processed resultsDone

Downstream jobs now

  • Customer serviceCompleted
  • Operations statusCompleted
  • Integrated analyticsTransferring

Mon–Fri

The week that followed

The company’s work repeats every day. But the time its data was ready differed every day.

FIVE BUSINESS DAYS

Actual start times over one week

DayData readyWhat was different
Monday08:47External and processed data delayed
Tuesday07:32All data ready early
Wednesday08:13More service usage logs
Thursday09:17External data transfer briefly paused
Friday08:01A fairly typical day
Even when the same work repeats,no two days are quite the same.

Live contextNow

Mon–FriThe week that followed

Data readiness5 / 5

  • TransactionsDone
  • Customer dataDone
  • Service opsDone
  • External dataDone
  • Processed resultsDone

Downstream jobs now

  • Customer serviceCompleted
  • Operations statusCompleted
  • Integrated analyticsCompleted

EVERY BUSINESS DAY

Tomorrow, too, the data won’t allbe ready at the same time.

Some work can start early,some has to wait for data that isn’t ready yet,and sometimes new, unscheduled work comes up.

Scheduled

Monday
08:00
Tuesday
08:00
Wednesday
08:00
Thursday
08:00
Friday
08:00

Actual data ready

Monday
08:47
Tuesday
07:32
Wednesday
08:13
Thursday
09:17
Friday
08:01

Work has a schedule.And it has conditions for when it can start.

Companies design their data movement with both in mind.

INNORIX

THE TECHNOLOGY BEHIND THIS STORY

Multi-Stage Flow

Set up and run downstream file transfers based on real business conditions, not just fixed schedules: data readiness, file creation and the completion of upstream transfers.

Explore the product