INNORIX in Practice, Multi-Stage Flow

When a file moves,the next job begins.

Work finished in one system becomes the start of another, and data created by one team carries on into another team's work.Follow a company whose separate jobs move together as one connected chain.

Scenario
A story set in a fictional company that shows how multi-stage file transfers link work across systems.

Monday 06:00

MONDAY MORNING

06:00

Today's first data has been created.

This morning, several teams' work starts from this data.

  1. 06:00DoneDaily business data createdBusiness system
  2. 06:12In progressData delivered to analyticsINNORIX
  3. 07:30WaitingAnalysis results createdData analytics system
  4. 08:00WaitingAnalysis results deliveredService operations

The first business data is created

INNORIX in Practice

The companyBefore INNORIX

The company, Before INNORIX

Each job was already automated.

Every day, the company's core business system creates new data. The analytics system analyzes the data it receives, and the service operations system puts the results to use.

Each system did its own work automatically. But moving data from one system to the next was still left to separate programs and to an operator.

When an earlier job ran late, the next job's run time had to be changed. When some data went missing, jobs that had already run had to be checked again.

The company began building this data movement as INNORIX Flows.

Automating each job and connecting one job to the next turned out to be different problems.

BEFORE

Each system was already doing its own job.

  • Business systemCreates daily business dataAutomatic
  • Analytics systemAnalyzes data and creates resultsAutomatic
  • Service operations systemUses analysis results in the serviceAutomatic

Work the operator managed between systems

  1. Check that all business data was created06:00
  2. Deliver data to the analytics environment06:30
  3. Check that analysis results were created07:50
  4. Deliver results to service operations08:00
  5. Find failed transfers and rerun themAs needed
Each job was automated,but the work in between still had to be managed.

Live contextNow

Before INNORIXWork waiting on each other

Work in progress

  • Business data creationRuns automatically
  • Analysis workAwaiting data
  • Service operationsAwaiting analysis results

Next up

Set up data delivery between systems

One Monday, 06:00

Today's first data has been created.

Monday, 6 a.m. The company's business system writes the previous day's transactions and service usage records to files.

Once the files are saved, a Flow starts to deliver them. Data bound for the analytics system and data that must be archived long-term each head to their own destination.

The analytics and service operations teams haven't arrived yet. But today's work has already begun.

The moment data is created, preparation for the next job begins.

06:00

Daily data processing

Business system, Daily output

Data created
  • transactions.csv12.4 GB
  • service_logs.parquet84.6 GB
  • daily_summary.csv240 MB

Analytics environment

  • transactions.csv
  • service_logs.parquet
Delivered

Long-term archive

  • All raw data
  • Daily business records
Delivered
  1. 06:00
  2. 06:05
  3. 06:30
  4. 06:42

The raw data archive is complete too.

Live contextNow

06:00The first data is created

Work in progress

  • Daily data creationDone
  • Delivery to analyticsTransferring
  • Raw data archiveTransferring

Next up

Analytics data delivered

One Monday, 06:30

The first transfer is done. And new work begins.

6:30 a.m. All the data the analytics environment needs has been delivered.

The analytics system confirms the delivery and starts its own analysis. The separate transfer that archives the raw data is done too.

Next, the result files created when the analysis finishes need to be delivered. That moment isn't a preset time, but whenever the analysis results are actually ready.

A finished transfer was not the end of the work.

06:30

Daily data processing

  1. INNORIXDeliver data to analytics06:12 — 06:30Done
  2. Company's analytics systemData analysis06:30 — 07:42Done
  3. Company's analytics systemCreate analysis result files07:42daily_analysis.parquetservice_metrics.jsonCreated
  4. INNORIXDeliver results to the service environment07:42 — 08:16Done
  1. 06:12
  2. 06:30
  3. 07:10
  4. 07:42
  5. 08:16

The data the service operations team needs is ready.

Live contextNow

06:30The system that receives it

Work in progress

  • Analytics data deliveryDone
  • Raw data archiveDone
  • External analytics systemAnalyzing
  • Analysis result deliveryAwaiting file

Next up

Analysis result files created

One Monday, 08:00

Analysis results reach the teams that need them.

7:42 a.m. The analytics system creates today's analysis result files.

As soon as the result files are ready, a Flow delivers them to the service operations environment. Results marked for long-term retention go to a separate storage environment.

8:16 a.m. Every file needed has been delivered. When the service operations team arrives at 9 a.m., they check today's data and get to work.

By the time staff arrived, the work across several systems was already done.

08:16

44 minutes before staff arrive

Work done for today's tasks

  1. 06:00Raw data createdBusiness systemDone
  2. 06:30Analytics data deliveredINNORIXDone
  3. 06:42Raw data archivedINNORIXDone
  4. 07:42Analysis results createdData analytics systemDone
  5. 08:16Result data delivered for serviceINNORIXDone

09:00

  • Infrastructure operationsDaily data archived
  • Data analyticsToday's analysis results ready
  • Service operationsService data ready
Each team starts its own work.

Live contextNow

08:00Work ready before the office opens

Work in progress

  • External system analysisDone
  • Results delivered for serviceDone
  • Analysis result archiveDone

Next up

09:00, service operations team starts work

One Monday, 10:30

Today's data is now needed for another job.

10:30 a.m. The new service project team asks for today's raw data and analysis results.

The project team uses a validation environment separate from service operations. It processes the data it receives, then needs its validation results delivered back.

Alongside the existing work, the operator sets up a data delivery Flow the project can use again and again.

The new work starts without changing or stopping the morning's routine jobs.

New work was added from the same data.

New service project team10:32

From now on, our service validation environment needs fresh data every day too.Please send the files we need from the raw data and analysis results to the validation server. Once validation is done, please deliver the result files back so the project team can use them.

Infrastructure operations10:35

We'll set up a separate data delivery job for the project, apart from the existing work.

Requested data and delivery conditions

DataDestinationRun condition
Daily raw dataValidation serverWhen the file is created
Analysis result dataValidation serverWhen the result file is created
Validation result dataProject storageWhen validation results are created

INNORIX STUDIO

Flows

FlowTypeStatus
Daily data processingExisting operationsActive
Service data deliveryExisting operationsActive
Historical data archiveExisting operationsActive
Project validation dataNew workAdded

Live contextNow

10:30A new work request

Work in progress

  • Existing daily workDone
  • Raw data for the projectTransferring
  • Project result deliveryAwaiting result file

Next up

Prepare data for project validation

One Monday, 13:20

The connection dropped while data was on its way to validation.

The project's validation server hits a temporary network problem. Not all the data it needs has been delivered yet.

The validation work that depends on that data doesn't run; it waits. Meanwhile, the existing service and archive jobs, which run separately, carry on.

Once the connection is back, the rest of the interrupted transfer resumes. Only after the transfer has actually finished does the validation system start its work.

The affected work waits; work that doesn't need to wait keeps going.

13:20

PROJECT VALIDATION

15:40, the validation results reached the project team.

New service project

Project raw data deliveryDone

  • Validation system workExternal systemValidated
  • Validation result deliveryINNORIXDelivered

Existing work running separately

  • Service operations data deliveryOperating normally
  • Daily data archiveDone
  • Analysis result archiveDone
  1. 13:20 Disconnected
  2. 13:34 Reconnected
  3. 13:56 Done
  4. 15:40 Results delivered

The validation system created its result files, and the results were delivered to the project team's storage.

Live contextNow

13:20An unexpected connection failure

Work in progress

  • Project data deliveryAwaiting recovery
  • External validation systemAwaiting data
  • Validation result deliveryAwaiting result file
  • Existing service workOperating normally

Next up

Reconnect the validation environment

One Monday, 16:30

How far did this afternoon's work get?

Someone on the project team asks for the delivery status of the validation results.

The operator checks the run history of the project's Flow: when the raw data was delivered, the connection failure, when the transfer completed, and the transfers after the validation system produced its results.

Without digging through every internal log of each system, the data movement INNORIX ran, and its transfer results, are clear at a glance.

You can see not just the last transfer result, but the whole path the data took.

Project team16:32

Can we see how the data used in today's validation was delivered, and when the result files were delivered?

INNORIX STUDIO

Project validation data

Runs

TransferCompletedStatus
Raw data for validation13:56Completed
Analysis results for validation12:14Completed
Project validation results15:40Completed

Raw data for validation, run details

  1. 13:00Transfer started
  2. 13:20Connection lost
  3. 13:34Reconnected, transfer resumed
  4. 13:56Transfer completed

Amazon S3 / Project validation server

Infrastructure operations16:35

The validation server had a brief connection failure this afternoon, but the raw data delivery completed at 13:56. The validation result files were delivered to project storage at 15:40.

Live contextNow

16:30The path one job took

Work in progress

  • Project raw data deliveryDone
  • External validationDone
  • Validation result deliveryDone

Next up

Project team reviews validation results

One Monday, 22:00

Today's last output sets up tomorrow's first job.

Even after staff leave for the day, the company's business systems keep creating new data.

Today's work results and data marked for archiving are delivered to the right storage on set schedules and conditions.

Early the next morning, as new data is created, the data delivery for analysis starts all over again.

Today's last data movement prepares tomorrow's work.

22:00

MONDAY — TUESDAY

Work that follows on set schedules and data creation conditions

Today's last jobsets up tomorrow's first.

Live contextNow

22:00Preparing tomorrow's work

Work in progress

  • Daily result data archiveTransferring
  • Tomorrow's data creationScheduled
  • Tomorrow's analytics deliveryAwaiting file

Next up

Tuesday 06:00, a new day of work begins

SIX MONTHS LATER

Then six months passed.

The company decided to bring in a new analytics system.

  • Existing workDaily data processingRunning every day
  • Existing workService data preparationRunning every day
  • New workNew analytics projectData delivery added

The existing work carries on.

INNORIX in Practice

The companyBefore INNORIX

The company changes, 6 months later

The company brings in a new analytics system.

On top of the existing analytics system, a new analytics project begins. The two analytics environments use different data and produce different results.

The operator reworks only the parts of the existing daily data delivery Flow that need it, and adds data delivery for the new analytics environment.

A new system has arrived, yet the service operations team keeps receiving its data exactly as before.

Adding new work didn't mean rebuilding everything.

The company's new analytics project

Existing data, used in a new way.

Existing analytics system

Work
Daily service analysis
Data used
Daily transaction data, service usage records
Where results are used
Existing service operations system
Operating normally

New analytics system

Work
Long-term data analysis
Data used
Historical transaction data, accumulated analysis results
Where results are used
New analytics project environment
New Flow added

INNORIX STUDIO

Flows

FlowTrigger, DestinationStatus
Daily data processingDaily, existing analyticsActive
Service data deliveryOn file creation, service operationsActive
Historical data archiveDaily, long-term archiveActive
Long-term analytics dataWeekly, new analyticsNew
New work was added,but existing teams' ways of working didn't change.

Live contextNow

6 months laterA new business system joins

Work in progress

  • Existing analytics FlowOperating normally
  • Existing service FlowOperating normally
  • New analytics FlowAdding & validating

How the work has changed

  1. Routine data processing
  2. Project validation
  3. New analytics system linked

Next up

New analytics system goes live

The company changes, 1 year later

The company's existing data powers new work.

The company launches a new AI project. The project team needs not only raw data but also the results the existing analytics system has produced.

The needed files go to a new compute environment, and once the external AI system creates its outputs, they are delivered on to a validation environment.

The AI project's data movement joins the company's existing Flow operations too.

New technology arrives, but delivering data and preparing the next job goes on.

AI project team

We're starting a new project using the raw and analysis data the company has built up.Please set it up so the training data is delivered, and once training finishes, the outputs go to a separate validation environment.

Existing company data used in the project

DataSource
Historical raw dataLong-term archive
Daily analysis dataExisting analytics system
Long-term analysis dataNew analytics system

AI PROJECT, DATA OPERATIONS

  1. INNORIXDeliver training dataDone
  2. AI compute systemTrain the model and create outputsIn progress
  3. INNORIXDeliver training outputs to validationAwaiting result file
  4. External validation systemValidate training outputsAwaiting input data
New technology arrives,but data still starts the next job the same way.

Live contextNow

1 year laterAn AI project begins

Work in progress

  • Existing data workOperating normally
  • AI training data deliveryTransferring
  • AI output deliveryAwaiting file

How the work has changed

  1. Routine data processing
  2. Project validation
  3. New analytics system linked
  4. AI project data linked

Next up

Prepare training data for the AI system

The company changes, 2 years later

The person who built the first Flows moves to another role.

Over the past two years, the company's systems and data movement have grown.

The new operator goes through the Flows now running to learn what data is created where, and under which conditions it is delivered.

From the run history they review completed jobs and jobs that ran into problems, then take over the routine work currently running.

The day after the handover, data created before dawn still reaches every system.

The person who built the work has changed, but the company's data movement carries on.

Operations handover

The company's data movement, built over two years

INNORIX STUDIO

Flows

Flow nameTriggerStatus
Daily data processingOn file creationActive
Service data deliveryOn result file creationActive
Historical data archiveDailyActive
Project validation dataOn file creationActive
Long-term analytics dataWeeklyActive
AI training data deliveryOn requestActive
AI output deliveryOn result file creationActive

Previous operator

Each Flow shows what it delivers and when it runs.Jobs that run after an earlier transfer completes are kept apart from jobs that run when a new result file is created. For anything that had problems, check the run history.

New operator

I've reviewed the routine jobs and each project's data delivery. I'll take over operations from tomorrow.

The next morning

06:00

New daily business data has been created.

Daily data processingToday's first transferRunning

People change, but the company's work carries on.

Live contextNow

2 years laterA new operator takes over

Work in progress

  • Daily data FlowOperating normally
  • Analytics & service FlowOperating normally
  • AI project FlowOperating normally
  • Operations handoverIn progress

How the work has changed

  1. Routine data processing
  2. Project validation
  3. New analytics system linked
  4. AI project data linked

Next up

The business keeps running

TODAY

Today, once again,many jobs start from one piece of data.

Over the past two years, a new analytics system arrived, and a new service and an AI project began.The company's work changed, and so did the people running it.

Where the story began

  1. Daily business data created
  2. Delivered to analytics
  3. Analysis results created
  4. Delivered to the service

The company, 2 years later

  1. Daily data processing
  2. Analytics & service data delivery
  3. Long-term data archive
  4. Service validation
  5. New analytics project
  6. AI training & output data delivery

The company never merged all its work into a single system.Nor did it overhaul its existing systems for every new job.Each system stays focused on the work it is meant to do.

Work finished in one placestarts new work somewhere else.

In between, INNORIX delivers the data wherever it is needed.

INNORIX

THE TECHNOLOGY BEHIND THIS STORY

Multi-Stage Flow

INNORIX connects data movement across systems and teams by setting the order and run conditions of file transfers.

Explore the product