INNORIX in Practice, Multi-Stage Flow

What happened to a company runninghundreds of automation programs?

File transfer programs, built one at a time whenever a need came up.Before long, the company’s core work depended on hundreds of them. Follow the last 8 years of this company, and what comes next.

Scenario
Through a fictional company running its own in-house file transfer programs, we look at how data movement work is designed and operated.

TodayOperating environment

EIGHT YEARS LATER

386

File transfer programsrunning at the company

64 Flow

Built at different times, by different developers, for different jobs.

/data_operations/

  • transfer_daily.py2018
  • backup_send.sh2019
  • report_upload_v3.py2021
  • transfer_retry_final.sh2022
  • service_data_new_v2.ps12024
  • export_legacy_fixed.py2026

and 380 more programs

24
Servers
11
Teams
17
Developers
  1. 8 years agoThe first program
  2. Today386 programs
  3. One Monday
  4. 1 year later

386 programs running on 24 servers

8 years of automation programs, and 1 year of change

Read from the start ↓

INNORIX in Practice

The company’s last 8 years8 years ago

The company’s last 8 years, 8 years ago

It started with three.

Back then, building it in-house was the simplest way.

The company’s first file transfer program was built to move files created on the business server every night to the archive server.

Then the analytics team asked for data, and the service operations team asked for result files. Two more programs were added.

A developer set the destinations and run times and wrote some simple error handling. The three programs ran without any real trouble.

2018

One server, three programs

The company had three file transfer jobs that had to run every day.

Operations server

ProgramPurposeSchedule
daily_transfer.shDaily data delivery06:00
backup_files.shSource data archiving23:00
send_report.pyAnalysis result delivery08:00
3
File transfer programs
1
Server
1
Developer
Back then, this was enough.

What the company needed back then was three file transfer jobs.

Live contextNow

8 years agoIt started with three.

Operating environment

3Existing programs running

Servers involved1

Current work

  • Daily data deliveryRunning normally
  • Source data archivingRunning normally
  • Analysis result deliveryRunning normally

Up next

New business system rollout

The company’s last 8 years, 5 years ago

One program at a time, as needed.

Every new job meant a new program.

As the company grew, so did its business systems, and new storage and analytics environments came in.

The developer who built the first programs reused existing code to build new file transfer programs. Developers on other teams built the programs their own work needed.

Requirements varied a little from job to job: send only certain files, or save files to a separate folder after transfer. The file transfer programs grew to 68.

2026

386

File transfer programs in operation

Servers
24
Teams
11
Developers
17

Each square is one file transfer program.

  1. 2018
  2. 2021
  3. 2023
  4. 2026

Hundreds of programs behind the company’s core work

Programs built by different developers at different times now run across the company.

Every program was built for a reason.

Live contextNow

5 years agoNew work keeps coming.

Operating environment

68Existing programs running

Servers involved7

Current work

  • New system data integrationIn development
  • Changes to existing programsIn progress
  • Scheduled data deliveryRunning normally

Up next

More automation programs per job

The company’s last 8 years, 3 years ago

There was more to build than file transfer.

At some point, moving files meant building a lot more.

As business units multiplied, simple data delivery was no longer enough. Requests followed for follow-up transfers after completion, sorting by file conditions, and retries when something failed.

To meet them, developers added run conditions and error handling to each program.

Some programs wrote transfer results to log files; others sent emails. How things were built depended on when, and by whom.

The company was now running 217 file transfer programs.

How one file transfer program grows

The original request was simple.

Work request

Please send the files created every night to the analytics server.

  1. First buildSend the files to the analytics serverBasic file transfer
  2. 2 weeks laterPlease send only the files to be analyzed.File selection and filtering added
  3. 1 month laterPlease resend if the connection drops.Error handling and retry logic added
  4. 3 months laterWe need to see the transfer results.Run logs and alerts added
  5. 6 months laterOnce it’s done, please send it to another server too.Follow-up transfer and run conditions added
  6. 1 year laterCan we switch to the new storage environment?Storage integration reworked
Even a single file transfer jobneeded ongoing development and maintenance.

The work was automated, but so was the work of managing the automation, and it kept growing.

Live contextNow

3 years agoThe work outgrew the programs

Operating environment

217Existing programs running

Servers involved16

Current work

  • Per-job transfer conditionsIn progress
  • Per-program error handlingManaged one by one
  • Transfer result checksLog by log

Up next

More programs and more to manage

One Monday, 09:12

The analysis results never arrived.

An ordinary Monday morning, and the analysis results had not arrived.

The service operations team called: the analysis result files they needed for the morning had not arrived.

The operator works out which server runs the program behind that job. After going through the logs of several programs, they find a connection error to the external storage environment.

The connection had already recovered, but the program had stopped. The operator had to find out whether it was safe to rerun it from the start.

09:12

MONDAY MORNING

Service operations team09:12

The analysis results we need this morning haven’t arrived yet. Could you take a look?Data neededdaily_analysis.parquet

Infrastructure operations team09:16

We’re tracking down the program that delivers this data and the server it runs on.

The operator’s checks

  • 09:20Ask the business owner which program it is
  • 09:32Find the server running the transfer program
  • 09:45Check the run log files
  • 09:51Find a connection error to external storage

report_upload_v3.log

  1. 08:00:01Job started
  2. 08:03:14Uploading analysis files
  3. 08:05:42Connection timeout
  4. 08:05:42Process terminated
  • Current network connectionNormal
  • Program run statusStopped
The connection had already recovered.But the data had not been delivered.

The network problem was fixed, but the work still hadn’t started.

Live contextNow

09:12The data never arrived.

Operating environment

386Existing programs running

Servers involved24

Current work

  • Analysis results for the serviceFailed
  • Morning service workWaiting for data
  • Program run historyChecking

Up next

Check whether the program can be rerun

One Monday, 10:40

Who built this program?

The developer who built it had long since moved on to other work.

The failing program was built five years ago. The employee who first wrote it had moved to another department.

The current operator reads through the code and logs to work out which files went through, and whether files already sent can safely be sent again.

Once this program finished, a separate program ran the follow-up file transfer, so its run conditions had to be checked too.

10:40

Morning business data delayed

report_upload_v3.py

  • First written2021-04-12
  • Last modified2022-08-19
  • Original developerMoved to another team
  • Current operatorInfrastructure operations
  • Related programs4

Infrastructure operations team

The connection is back. Can we just run the program again?

Development team

First we need to check whether some files already went through. It looks like a downstream program also runs when this one finishes. We’ll also check whether a rerun could process anything twice.

What to check before a rerun

Earlier transfer results
Which files already went through?
Rerun conditions
Does it resend every file?
Downstream programs
What triggers them?
Duplicates
Will it affect data already processed?
Just to run the program again,they first had to understand how it worked.

Before restarting the transfer, they had to understand how the program was built.

Live contextNow

10:40Who built this program?

Operating environment

386Existing programs running

Servers involved24

Current work

  • Review the existing source codeIn progress
  • Check for duplicate transfersIn progress
  • Check downstream programsIn progress

Up next

Check remaining files and resume transfer

One Monday, 14:30

Another development request arrived.

While they were fixing one problem, another development request came in.

The morning’s data delivery problem was solved. But in the afternoon, a request came in to deliver data to a new analytics environment.

It wasn’t just moving files: only certain files were to be selected, and once the transfer finished, the result data had to go on to a separate environment.

The development team looks for reusable code in existing programs and reviews the new run conditions and error handling.

At this company, almost every new data movement job turned into a new development job.

New analytics project team14:30

We’ll start using a new analytics server next week.Please set it up so the source data we need is delivered from the business server, and once the analysis is done, the result files go to the project storage.

What the new job needs

  • Pick the source dataSpecific file types and date conditions
  • Deliver to the analytics environmentNew server connection and file transfer
  • Check result filesConfirm the analysis results were created
  • Follow-up transferDeliver to the project storage environment
  • Error handlingRetry and recover interrupted transfers
  • Operational checksRun results and file delivery history

Development team: new job review

  • Check whether existing programs can be reused
  • Build the new environment integration and run conditions
  • Implement failure handling and follow-up transfer logic
  • Test runs and deploy to production
One more file transfer jobturns into one more development project.

Running file transfer programs and building new ones, over and over again.

Live contextNow

14:30A new work request

Operating environment

386Existing programs running

Servers involved24

Current work

  • Morning transfer failureResolved
  • New analytics environment linkUnder dev review
  • Run conditions and recoveryDesign planned

Up next

Schedule the new program’s development

ONE WEEK LATER

The company decided to take another lookat every program it had built.

386

File transfer programs in operation today

Each program handled a different job.

But inside them were features built again and again.

How much of this do we really need to build ourselves?

INNORIX in Practice

The company’s last 8 years8 years ago

A new choice, 1 week later

Looking at the work, not the programs.

The company started looking at its work, not its programs.

The infrastructure operations and development teams began surveying what each of the 386 running programs actually did.

They found the same roles built again and again in different programs: finding data, selecting files, transferring, recovering connections, logging runs and sending follow-up transfers.

At the same time, some programs ran the company’s own data processing logic. Replacing every program was never the right answer.

Survey of existing automation programs

Different programs were doing the same things over and over.

SelectTransferRecoverFollow-upCustom
daily_transfer.pyCommon data movement functionCommon data movement functionCommon data movement functionNoneNone
backup_files.shCommon data movement functionCommon data movement functionCommon data movement functionNoneNone
report_upload.pyCommon data movement functionCommon data movement functionCommon data movement functionCommon data movement functionNone
analytics_job.pyCommon data movement functionCommon data movement functionNoneCommon data movement functionBusiness-specific processing
service_export.shCommon data movement functionCommon data movement functionCommon data movement functionCommon data movement functionNone
data_transform.pyNoneCommon data movement functionNoneNoneBusiness-specific processing
archive_result.pyCommon data movement functionCommon data movement functionCommon data movement functionNoneNone
  • Common data movement function
  • Business-specific processing

Repeated data movement functions

  • Selecting files to transfer
  • Transfer between servers and cloud
  • Handling connection failures and errors
  • Run conditions and follow-up transfers
  • Keeping file transfer history

Business-specific functions

  • The company’s data analysis logic
  • Data generation in business systems
  • Industry-specific data processing
  • Service-specific business logic
  • Processing unique to existing systems
Keep the software the business really needs,and separate out the repeated data movement.

What had to go was not the programs themselves, but the development and operations work being repeated for no reason.

Live contextNow

1 week laterWhat 386 programs really do

Operating environment

386Existing programs running

Servers involved24

Current work

  • Survey existing programsIn progress
  • Classify file transfer functionsIn progress
  • Separate business-specific logicIn progress

Up next

Choose the first job to switch

A new choice, 1 month later

The first job, built as a Flow.

The company changed how it operated, one job at a time.

The first job chosen for the switch was the daily analytics data delivery. Several programs were involved: selecting files, transferring, retrying and delivering results.

The development team keeps the existing systems that run the analysis, and the programs that do the company’s own data processing, exactly as they are.

Instead, the part that delivers the files and starts the follow-up transfer when completion conditions are met was built as an INNORIX Flow.

They decided to compare the results with the old way for a while before putting it into production.

First switch: daily analytics data processing

The company doesn’t change every program at once. It started with the analytics data delivery job: it runs on a schedule, and the work before and after each transfer is clearly defined.

  1. How it ran before
  2. Separating the work
  3. How it runs now

Daily analytics data

INNORIX Flow
Trigger
Daily source data is ready
Endpoints
Business server, analytics server
Transfer rules
Selected files only, recover on error

Existing external system

The company’s data analysis

The existing analytics program creates the result files.

Follow-up data delivery

Deliver analysis result files

Runs when result files are created

The existing analytics system stays, while data movement and follow-up transfers run in the Flow environment.

The work itself doesn’t change.What changes is how data moves between the steps.

Without rebuilding existing systems, they started by changing the data movement.

Live contextNow

1 month laterThe first job, built as a Flow

Operating environment

386Existing programs running

1Flows built

Servers involved24

Current work

  • Existing analytics systemKept
  • New data delivery FlowPilot run
  • Compare with existing resultsIn progress

Up next

Verify the production switchover

A new choice, 2 months later

They didn’t shut down the old programs right away.

OPERATING VERIFICATION

The new way of operating has to produce the same results as the old one.

  1. Normal transfer
  2. Connection failure test
  3. Follow-up transfer check

Existing program

Downstream program run

File delivery
Completed
Error check
Normal
What happens next
Check a separate run log

INNORIX Flow

Follow-up transfer run

File delivery
Completed
Error check
Normal
What happens next
Check the related transfer history

Checked before switching over

  • Transfer targets and result filesMatch
  • Storage locations and handling rulesMatch
  • Downstream work in external systemsVerified
  • Failure recovery and rerun behaviorVerified
Nothing was switched over until it was verified.

Live contextNow

2 months laterThey didn’t shut down the old programs right away.

Operating environment

386Existing programs running

1Flows built

Servers involved24

Current work

  • Existing programsIn operation
  • Compare results in a test environmentIn progress
  • Verify failure recovery and rerunsIn progress

Up next

The first switchover

A new choice, 3 months later

The work of 28 programs

What 28 programs used to handle now runs as 6 Flows.

After verifying the first switched job, the company moved on to switching the related data movement work.

The file transfer and run control work of 28 existing programs was rebuilt as 6 Flows. Each Flow carries the endpoints, run conditions and follow-up transfers its job needs.

The old programs weren’t deleted right away; they were kept for a while as the new results were checked. The company’s own data processing programs kept running as before.

First switchover complete

How it ran before

28File transfer programs

  • Daily file discovery and selection, 4
  • Source data transfer, 6
  • Transfer retry, 4
  • Analysis result watchers, 3
  • Result data delivery, 7
  • Run logging and alerts, 4

How it runs now

6Flows, one per job

  • Daily source data deliveryIn operation
  • Scheduled analytics data prepIn operation
  • Analysis result forwardingIn operation
  • Service operations data deliveryIn operation
  • Project data deliveryIn operation
  • Long-term data archivingIn operation

Existing company systems

Data generation, analysis, business processing and the company’s own programs keep running as before.

Not how many programs they manage,but which work they manage became clear.

They started managing the work they run, not the number of programs.

Live contextNow

3 months laterThe first switchover

Operating environment

358Existing programs running

6Flows built

Servers involved24

Current work

  • First switched areaIn operation
  • Existing programsStopped and archived
  • Next switch candidatesUnder review

Up next

Extend the switch to other areas

The company changes, 6 months later

A similar request came in again.

The request looked familiar, but the response was different.

A new analytics environment came in, and with it the need to deliver source data and analysis results.

Developers first looked at what extra processing the existing systems needed, and worked out the file movement requirements with the operations team.

The operations team checks the connected devices and storage, then builds the data and run conditions it needs as a Flow.

The development team could focus on the project’s data analysis and service features.

New project team

We need the daily data in the new analytics environment too.Please deliver the source data, and when the analysis is done, send the result files to the project storage.

Much like the request that came in six months ago.

6 months ago

Development team’s work

  • Survey existing programs
  • Write integration code for the new environment
  • Implement file selection and run conditions
  • Build error handling and retries
  • Implement follow-up transfers
  • Implement logging and run history
  • Test and deploy to production

Today

Operations team’s work

  • Connect the new devices and storage
  • Select the data to transfer
  • Set up run conditions and follow-up transfers
  • Test the transfer and go live

Development team

Building new analytics features

Builds the data processing and analytics features unique to the business.

Development work and data movement operationsnow play different roles.

A new file transfer job no longer had to mean a new program.

Live contextNow

6 months laterA new development request arrives.

Operating environment

214Existing programs running

37Flows built

Servers involved24

Current work

  • New analysis logicDev team
  • Data delivery FlowBuilt by operations
  • Existing workRunning normally

Up next

New project goes live

The company changes, 1 year later

The same failure happened again.

08:32

ONE YEAR LATER, MONDAY

Service operations data has been delivered.

Runs

Daily service data
Analysis result data deliveryCompleted

Source: Analytics storage / Target: Service environment

Run history

  1. 08:00Scheduled job started
  2. 08:05Connection lost
  3. 08:12Connection recovered, transfer resumed
  4. 08:32File transfer completed
Follow-up data deliveryStarted
  1. 08:05 Interrupted
  2. 08:12 Recovered
  3. 08:18 Resumed
  4. 08:32 Completed

What’s different now

The job owner can see the run results, how it recovered, and the status of the follow-up transfer.

The company didn’t become one where nothing fails.What changed is how it responds.

Live contextNow

1 year laterA new way of operating

Operating environment

82Existing programs running

64Flows built

Servers involved24

Current work

  • Recurring data movementRun as Flows
  • Business-specific logicExisting programs kept
  • New work designBuilt around the work

Up next

Ongoing operations

The company changes, 1 year later

Did all of the company’s programs disappear?

What the company manages has changed.

Over the year, the switch happened step by step, starting with the file transfer work that recurs.

Of the 386 programs, 304 have stopped running. 82 remain, because they carry business-specific logic or depend on existing systems.

The operations team checks each job’s run conditions, endpoints and results across 64 Flows.

When a new file transfer request comes in, they first check whether an existing Flow can be extended or a new Flow built.

ONE YEAR OF TRANSITION

The company did not switch every program.

At the start of the switch

386

Programs in operation

1 year later

82programs kept

82 kept304 stopped

64Flows in operation

Programs still running

  • Business-specific data processing
  • Programs tied to existing systems
  • Special-purpose automation
  • Programs to review for a later switch

Work run as Flows

  • Data delivery between servers and cloud
  • Condition-based file transfer
  • Follow-up transfers after completion
  • Transfer recovery and result checks
They separated what to build in-house from what to run on a platform.

These figures are operating changes set in a fictional company scenario and do not represent any real customer’s results.

The company still has its own in-house programs. It just doesn’t build every piece of data movement itself.

The company today, Today

What developers should build

The company still builds new software.

New services and analytics features are being built, and so is the software unique to the business.

But there is far less need to rebuild, in every program, the code that selects and delivers files, handles errors and starts the next transfer when one finishes.

Developers build the software the business needs; operators manage the company’s data movement.

MONDAY, 10:00

A new project has started.

Service planning team

We’re building a service on top of existing analytics data and new user data. Please get the data delivery ready as well.

Development team

Projects in progress

  • Designing new service features
  • Developing data analysis logic
  • Building user-facing features
  • Validating and improving the service

Builds the new software the business needs.

Infrastructure operations team

INNORIX, Flows

  • Existing analytics data deliveryIn operation
  • New user data deliveryNew
  • Service result data deliveryNew
  • Scheduled data archivingIn operation

Builds and runs the data movement the service needs.

As before, the company builds new software.It just doesn’t rebuild every data movement feature each time.

Focus on what you need to build, and run what you need to run.

Live contextNow

TodayWhat developers should build

Operating environment

82Existing programs running

64Flows built

Servers involved24

Current work

  • Business-specific service developmentIn progress
  • Data movement workRunning normally
  • New work requestsReviewed as Flows

Up next

New projects ahead

TODAY

This company stillbuilds new software.

New services launch, new analytics environments arrive,and the software unique to the business keeps being built.

Where the story began

386
File transfer programs

Operations built around individual programs

The company, one year later

82
Existing programs
64
Flow

Business-specific work and data movement, separated

  • The existing systems the business needs stayed in place.
  • The company keeps building its own software.
  • The data movement it used to build again and again now runs as Flows.

Focus on what you need to build,and run what you need to run.

INNORIX

THE TECHNOLOGY BEHIND THIS STORY

Multi-Stage Flow

INNORIX: build the file transfer and run control you kept re-developing as Flows, and run them alongside the systems you already have.

Explore the product