Work finished in one system becomes the start of another, and data created by one team carries on into another team's work.Follow a company whose separate jobs move together as one connected chain.
Scenario
A story set in a fictional company that shows how multi-stage file transfers link work across systems.
Monday 06:00
MONDAY MORNING
06:00
Today's first data has been created.
This morning, several teams' work starts from this data.
06:00DoneDaily business data createdBusiness system
06:12In progressData delivered to analyticsINNORIX
07:30WaitingAnalysis results createdData analytics system
Scene preview, 06:00The first data starts the day's work.
Work across many systems that gets moving before anyone arrives.
A company's day, and two years of change, starting from one piece of dataRead from the start ↓
INNORIX in Practice
The companyBefore INNORIX
The company, Before INNORIX
Each job was already automated.
Every day, the company's core business system creates new data. The analytics system analyzes the data it receives, and the service operations system puts the results to use.
Each system did its own work automatically. But moving data from one system to the next was still left to separate programs and to an operator.
When an earlier job ran late, the next job's run time had to be changed. When some data went missing, jobs that had already run had to be checked again.
The company began building this data movement as INNORIX Flows.
Automating each job and connecting one job to the next turned out to be different problems.
BEFORE
Each system was already doing its own job.
Business systemCreates daily business dataAutomatic
Analytics systemAnalyzes data and creates resultsAutomatic
Service operations systemUses analysis results in the serviceAutomatic
Work the operator managed between systems
Check that all business data was created06:00
Deliver data to the analytics environment06:30
Check that analysis results were created07:50
Deliver results to service operations08:00
Find failed transfers and rerun themAs needed
Each job was automated,but the work in between still had to be managed.
Live contextNow
Before INNORIXWork waiting on each other
Work in progress
Business data creationRuns automatically
Analysis workAwaiting data
Service operationsAwaiting analysis results
Next up
Set up data delivery between systems
One Monday, 06:00
Today's first data has been created.
Monday, 6 a.m. The company's business system writes the previous day's transactions and service usage records to files.
Once the files are saved, a Flow starts to deliver them. Data bound for the analytics system and data that must be archived long-term each head to their own destination.
The analytics and service operations teams haven't arrived yet. But today's work has already begun.
The moment data is created, preparation for the next job begins.
06:00
Daily data processing
Business system, Daily output
Data created
transactions.csv12.4 GB
service_logs.parquet84.6 GB
daily_summary.csv240 MB
Analytics environment
transactions.csv
service_logs.parquet
Delivered
Long-term archive
All raw data
Daily business records
Delivered
06:00
06:05
06:30
06:42
The raw data archive is complete too.
Live contextNow
06:00The first data is created
Work in progress
Daily data creationDone
Delivery to analyticsTransferring
Raw data archiveTransferring
Next up
Analytics data delivered
One Monday, 06:30
The first transfer is done. And new work begins.
6:30 a.m. All the data the analytics environment needs has been delivered.
The analytics system confirms the delivery and starts its own analysis. The separate transfer that archives the raw data is done too.
Next, the result files created when the analysis finishes need to be delivered. That moment isn't a preset time, but whenever the analysis results are actually ready.
Company's analytics systemCreate analysis result files07:42daily_analysis.parquetservice_metrics.jsonCreated
INNORIXDeliver results to the service environment07:42 — 08:16Done
06:12
06:30
07:10
07:42
08:16
The data the service operations team needs is ready.
Live contextNow
06:30The system that receives it
Work in progress
Analytics data deliveryDone
Raw data archiveDone
External analytics systemAnalyzing
Analysis result deliveryAwaiting file
Next up
Analysis result files created
One Monday, 08:00
Analysis results reach the teams that need them.
7:42 a.m. The analytics system creates today's analysis result files.
As soon as the result files are ready, a Flow delivers them to the service operations environment. Results marked for long-term retention go to a separate storage environment.
8:16 a.m. Every file needed has been delivered. When the service operations team arrives at 9 a.m., they check today's data and get to work.
By the time staff arrived, the work across several systems was already done.
10:30 a.m. The new service project team asks for today's raw data and analysis results.
The project team uses a validation environment separate from service operations. It processes the data it receives, then needs its validation results delivered back.
Alongside the existing work, the operator sets up a data delivery Flow the project can use again and again.
The new work starts without changing or stopping the morning's routine jobs.
New work was added from the same data.
N
New service project team10:32
From now on, our service validation environment needs fresh data every day too.Please send the files we need from the raw data and analysis results to the validation server. Once validation is done, please deliver the result files back so the project team can use them.
I
Infrastructure operations10:35
We'll set up a separate data delivery job for the project, apart from the existing work.
Requested data and delivery conditions
Data
Destination
Run condition
Daily raw data
Validation server
When the file is created
Analysis result data
Validation server
When the result file is created
Validation result data
Project storage
When validation results are created
INNORIX STUDIO
Flows
Flow
Type
Status
Daily data processing
Existing operations
Active
Service data delivery
Existing operations
Active
Historical data archive
Existing operations
Active
Project validation data
New work
Added
Live contextNow
10:30A new work request
Work in progress
Existing daily workDone
Raw data for the projectTransferring
Project result deliveryAwaiting result file
Next up
Prepare data for project validation
One Monday, 13:20
The connection dropped while data was on its way to validation.
The project's validation server hits a temporary network problem. Not all the data it needs has been delivered yet.
The validation work that depends on that data doesn't run; it waits. Meanwhile, the existing service and archive jobs, which run separately, carry on.
Once the connection is back, the rest of the interrupted transfer resumes. Only after the transfer has actually finished does the validation system start its work.
The affected work waits; work that doesn't need to wait keeps going.
13:20
PROJECT VALIDATION
15:40, the validation results reached the project team.
New service project
Project raw data deliveryDone
Validation system workExternal systemValidated
Validation result deliveryINNORIXDelivered
Existing work running separately
Service operations data deliveryOperating normally
Daily data archiveDone
Analysis result archiveDone
13:20 Disconnected
13:34 Reconnected
13:56 Done
15:40 Results delivered
The validation system created its result files, and the results were delivered to the project team's storage.
Live contextNow
13:20An unexpected connection failure
Work in progress
Project data deliveryAwaiting recovery
External validation systemAwaiting data
Validation result deliveryAwaiting result file
Existing service workOperating normally
Next up
Reconnect the validation environment
One Monday, 16:30
How far did this afternoon's work get?
Someone on the project team asks for the delivery status of the validation results.
The operator checks the run history of the project's Flow: when the raw data was delivered, the connection failure, when the transfer completed, and the transfers after the validation system produced its results.
Without digging through every internal log of each system, the data movement INNORIX ran, and its transfer results, are clear at a glance.
You can see not just the last transfer result, but the whole path the data took.
P
Project team16:32
Can we see how the data used in today's validation was delivered, and when the result files were delivered?
INNORIX STUDIO
Project validation data
Runs
Transfer
Completed
Status
Raw data for validation
13:56
Completed
Analysis results for validation
12:14
Completed
Project validation results
15:40
Completed
Raw data for validation, run details
13:00Transfer started
13:20Connection lost
13:34Reconnected, transfer resumed
13:56Transfer completed
Amazon S3 / Project validation server
I
Infrastructure operations16:35
The validation server had a brief connection failure this afternoon, but the raw data delivery completed at 13:56. The validation result files were delivered to project storage at 15:40.
Live contextNow
16:30The path one job took
Work in progress
Project raw data deliveryDone
External validationDone
Validation result deliveryDone
Next up
Project team reviews validation results
One Monday, 22:00
Today's last output sets up tomorrow's first job.
Even after staff leave for the day, the company's business systems keep creating new data.
Today's work results and data marked for archiving are delivered to the right storage on set schedules and conditions.
Early the next morning, as new data is created, the data delivery for analysis starts all over again.
Today's last data movement prepares tomorrow's work.
22:00
MONDAY — TUESDAY
Work that follows on set schedules and data creation conditions
18:0021:0000:0003:0006:00
Daily results
Long-term archive
Overnight data
Next-day analytics
After staff have left
Today's last jobsets up tomorrow's first.
Live contextNow
22:00Preparing tomorrow's work
Work in progress
Daily result data archiveTransferring
Tomorrow's data creationScheduled
Tomorrow's analytics deliveryAwaiting file
Next up
Tuesday 06:00, a new day of work begins
SIX MONTHS LATER
Then six months passed.
The company decided to bring in a new analytics system.
Existing workDaily data processingRunning every day
Existing workService data preparationRunning every day
New workNew analytics projectData delivery added
The existing work carries on.
INNORIX in Practice
The companyBefore INNORIX
The company changes, 6 months later
The company brings in a new analytics system.
On top of the existing analytics system, a new analytics project begins. The two analytics environments use different data and produce different results.
The operator reworks only the parts of the existing daily data delivery Flow that need it, and adds data delivery for the new analytics environment.
A new system has arrived, yet the service operations team keeps receiving its data exactly as before.
Adding new work didn't mean rebuilding everything.
New work was added,but existing teams' ways of working didn't change.
Live contextNow
6 months laterA new business system joins
Work in progress
Existing analytics FlowOperating normally
Existing service FlowOperating normally
New analytics FlowAdding & validating
How the work has changed
Routine data processing
Project validation
New analytics system linked
Next up
New analytics system goes live
The company changes, 1 year later
The company's existing data powers new work.
The company launches a new AI project. The project team needs not only raw data but also the results the existing analytics system has produced.
The needed files go to a new compute environment, and once the external AI system creates its outputs, they are delivered on to a validation environment.
The AI project's data movement joins the company's existing Flow operations too.
New technology arrives, but delivering data and preparing the next job goes on.
A
AI project team
We're starting a new project using the raw and analysis data the company has built up.Please set it up so the training data is delivered, and once training finishes, the outputs go to a separate validation environment.
Existing company data used in the project
Data
Source
Historical raw data
Long-term archive
Daily analysis data
Existing analytics system
Long-term analysis data
New analytics system
AI PROJECT, DATA OPERATIONS
INNORIXDeliver training dataDone
AI compute systemTrain the model and create outputsIn progress
INNORIXDeliver training outputs to validationAwaiting result file
External validation systemValidate training outputsAwaiting input data
New technology arrives,but data still starts the next job the same way.
Live contextNow
1 year laterAn AI project begins
Work in progress
Existing data workOperating normally
AI training data deliveryTransferring
AI output deliveryAwaiting file
How the work has changed
Routine data processing
Project validation
New analytics system linked
AI project data linked
Next up
Prepare training data for the AI system
The company changes, 2 years later
The person who built the first Flows moves to another role.
Over the past two years, the company's systems and data movement have grown.
The new operator goes through the Flows now running to learn what data is created where, and under which conditions it is delivered.
From the run history they review completed jobs and jobs that ran into problems, then take over the routine work currently running.
The day after the handover, data created before dawn still reaches every system.
The person who built the work has changed, but the company's data movement carries on.
Operations handover
The company's data movement, built over two years
INNORIX STUDIO
Flows
Flow name
Trigger
Status
Daily data processing
On file creation
Active
Service data delivery
On result file creation
Active
Historical data archive
Daily
Active
Project validation data
On file creation
Active
Long-term analytics data
Weekly
Active
AI training data delivery
On request
Active
AI output delivery
On result file creation
Active
P
Previous operator
Each Flow shows what it delivers and when it runs.Jobs that run after an earlier transfer completes are kept apart from jobs that run when a new result file is created. For anything that had problems, check the run history.
N
New operator
I've reviewed the routine jobs and each project's data delivery. I'll take over operations from tomorrow.
The next morning
06:00
New daily business data has been created.
Daily data processingToday's first transferRunning
People change, but the company's work carries on.
Live contextNow
2 years laterA new operator takes over
Work in progress
Daily data FlowOperating normally
Analytics & service FlowOperating normally
AI project FlowOperating normally
Operations handoverIn progress
How the work has changed
Routine data processing
Project validation
New analytics system linked
AI project data linked
Next up
The business keeps running
TODAY
Today, once again,many jobs start from one piece of data.
Over the past two years, a new analytics system arrived, and a new service and an AI project began.The company's work changed, and so did the people running it.
Where the story began
Daily business data created
Delivered to analytics
Analysis results created
Delivered to the service
The company, 2 years later
Daily data processing
Analytics & service data delivery
Long-term data archive
Service validation
New analytics project
AI training & output data delivery
The company never merged all its work into a single system.Nor did it overhaul its existing systems for every new job.Each system stays focused on the work it is meant to do.
Work finished in one placestarts new work somewhere else.
In between, INNORIX delivers the data wherever it is needed.
INNORIX
THE TECHNOLOGY BEHIND THIS STORY
Multi-Stage Flow
INNORIX connects data movement across systems and teams by setting the order and run conditions of file transfers.