What happened to a company runninghundreds of automation programs?
File transfer programs, built one at a time whenever a need came up.Before long, the company’s core work depended on hundreds of them. Follow the last 8 years of this company, and what comes next.
Scenario
Through a fictional company running its own in-house file transfer programs, we look at how data movement work is designed and operated.
TodayOperating environment
EIGHT YEARS LATER
386
File transfer programsrunning at the company
64 Flow
Built at different times, by different developers, for different jobs.
Back then, building it in-house was the simplest way.
The company’s first file transfer program was built to move files created on the business server every night to the archive server.
Then the analytics team asked for data, and the service operations team asked for result files. Two more programs were added.
A developer set the destinations and run times and wrote some simple error handling. The three programs ran without any real trouble.
2018
One server, three programs
The company had three file transfer jobs that had to run every day.
Operations server
Program
Purpose
Schedule
daily_transfer.sh
Daily data delivery
06:00
backup_files.sh
Source data archiving
23:00
send_report.py
Analysis result delivery
08:00
3
File transfer programs
1
Server
1
Developer
Back then, this was enough.
What the company needed back then was three file transfer jobs.
Live contextNow
8 years agoIt started with three.
Operating environment
3Existing programs running
Servers involved1
Current work
Daily data deliveryRunning normally
Source data archivingRunning normally
Analysis result deliveryRunning normally
Up next
New business system rollout
The company’s last 8 years, 5 years ago
One program at a time, as needed.
Every new job meant a new program.
As the company grew, so did its business systems, and new storage and analytics environments came in.
The developer who built the first programs reused existing code to build new file transfer programs. Developers on other teams built the programs their own work needed.
Requirements varied a little from job to job: send only certain files, or save files to a separate folder after transfer. The file transfer programs grew to 68.
2026
386
File transfer programs in operation
Servers
24
Teams
11
Developers
17
Each square is one file transfer program.
2018
2021
2023
2026
Hundreds of programs behind the company’s core work
Programs built by different developers at different times now run across the company.
Every program was built for a reason.
Live contextNow
5 years agoNew work keeps coming.
Operating environment
68Existing programs running
Servers involved7
Current work
New system data integrationIn development
Changes to existing programsIn progress
Scheduled data deliveryRunning normally
Up next
More automation programs per job
The company’s last 8 years, 3 years ago
There was more to build than file transfer.
At some point, moving files meant building a lot more.
As business units multiplied, simple data delivery was no longer enough. Requests followed for follow-up transfers after completion, sorting by file conditions, and retries when something failed.
To meet them, developers added run conditions and error handling to each program.
Some programs wrote transfer results to log files; others sent emails. How things were built depended on when, and by whom.
The company was now running 217 file transfer programs.
How one file transfer program grows
The original request was simple.
Work request
Please send the files created every night to the analytics server.
First buildSend the files to the analytics serverBasic file transfer
2 weeks laterPlease send only the files to be analyzed.File selection and filtering added
1 month laterPlease resend if the connection drops.Error handling and retry logic added
3 months laterWe need to see the transfer results.Run logs and alerts added
6 months laterOnce it’s done, please send it to another server too.Follow-up transfer and run conditions added
1 year laterCan we switch to the new storage environment?Storage integration reworked
Even a single file transfer jobneeded ongoing development and maintenance.
The work was automated, but so was the work of managing the automation, and it kept growing.
Live contextNow
3 years agoThe work outgrew the programs
Operating environment
217Existing programs running
Servers involved16
Current work
Per-job transfer conditionsIn progress
Per-program error handlingManaged one by one
Transfer result checksLog by log
Up next
More programs and more to manage
One Monday, 09:12
The analysis results never arrived.
An ordinary Monday morning, and the analysis results had not arrived.
The service operations team called: the analysis result files they needed for the morning had not arrived.
The operator works out which server runs the program behind that job. After going through the logs of several programs, they find a connection error to the external storage environment.
The connection had already recovered, but the program had stopped. The operator had to find out whether it was safe to rerun it from the start.
09:12
MONDAY MORNING
S
Service operations team09:12
The analysis results we need this morning haven’t arrived yet. Could you take a look?Data neededdaily_analysis.parquet
I
Infrastructure operations team09:16
We’re tracking down the program that delivers this data and the server it runs on.
The operator’s checks
09:20Ask the business owner which program it is
09:32Find the server running the transfer program
09:45Check the run log files
09:51Find a connection error to external storage
report_upload_v3.log
08:00:01Job started
08:03:14Uploading analysis files
08:05:42Connection timeout
08:05:42Process terminated
Current network connectionNormal
Program run statusStopped
The connection had already recovered.But the data had not been delivered.
The network problem was fixed, but the work still hadn’t started.
Live contextNow
09:12The data never arrived.
Operating environment
386Existing programs running
Servers involved24
Current work
Analysis results for the serviceFailed
Morning service workWaiting for data
Program run historyChecking
Up next
Check whether the program can be rerun
One Monday, 10:40
Who built this program?
The developer who built it had long since moved on to other work.
The failing program was built five years ago. The employee who first wrote it had moved to another department.
The current operator reads through the code and logs to work out which files went through, and whether files already sent can safely be sent again.
Once this program finished, a separate program ran the follow-up file transfer, so its run conditions had to be checked too.
10:40
Morning business data delayed
report_upload_v3.py
First written2021-04-12
Last modified2022-08-19
Original developerMoved to another team
Current operatorInfrastructure operations
Related programs4
I
Infrastructure operations team
The connection is back. Can we just run the program again?
D
Development team
First we need to check whether some files already went through. It looks like a downstream program also runs when this one finishes. We’ll also check whether a rerun could process anything twice.
What to check before a rerun
Earlier transfer results
Which files already went through?
Rerun conditions
Does it resend every file?
Downstream programs
What triggers them?
Duplicates
Will it affect data already processed?
Just to run the program again,they first had to understand how it worked.
Before restarting the transfer, they had to understand how the program was built.
Live contextNow
10:40Who built this program?
Operating environment
386Existing programs running
Servers involved24
Current work
Review the existing source codeIn progress
Check for duplicate transfersIn progress
Check downstream programsIn progress
Up next
Check remaining files and resume transfer
One Monday, 14:30
Another development request arrived.
While they were fixing one problem, another development request came in.
The morning’s data delivery problem was solved. But in the afternoon, a request came in to deliver data to a new analytics environment.
It wasn’t just moving files: only certain files were to be selected, and once the transfer finished, the result data had to go on to a separate environment.
The development team looks for reusable code in existing programs and reviews the new run conditions and error handling.
At this company, almost every new data movement job turned into a new development job.
N
New analytics project team14:30
We’ll start using a new analytics server next week.Please set it up so the source data we need is delivered from the business server, and once the analysis is done, the result files go to the project storage.
What the new job needs
Pick the source dataSpecific file types and date conditions
Deliver to the analytics environmentNew server connection and file transfer
Check result filesConfirm the analysis results were created
Follow-up transferDeliver to the project storage environment
Error handlingRetry and recover interrupted transfers
Operational checksRun results and file delivery history
Development team: new job review
Check whether existing programs can be reused
Build the new environment integration and run conditions
Implement failure handling and follow-up transfer logic
Test runs and deploy to production
One more file transfer jobturns into one more development project.
Running file transfer programs and building new ones, over and over again.
Live contextNow
14:30A new work request
Operating environment
386Existing programs running
Servers involved24
Current work
Morning transfer failureResolved
New analytics environment linkUnder dev review
Run conditions and recoveryDesign planned
Up next
Schedule the new program’s development
ONE WEEK LATER
The company decided to take another lookat every program it had built.
386
File transfer programs in operation today
Each program handled a different job.
But inside them were features built again and again.
How much of this do we really need to build ourselves?
INNORIX in Practice
The company’s last 8 years8 years ago
A new choice, 1 week later
Looking at the work, not the programs.
The company started looking at its work, not its programs.
The infrastructure operations and development teams began surveying what each of the 386 running programs actually did.
They found the same roles built again and again in different programs: finding data, selecting files, transferring, recovering connections, logging runs and sending follow-up transfers.
At the same time, some programs ran the company’s own data processing logic. Replacing every program was never the right answer.
Survey of existing automation programs
Different programs were doing the same things over and over.
Select
Transfer
Recover
Follow-up
Custom
daily_transfer.py
Common data movement function
Common data movement function
Common data movement function
None
None
backup_files.sh
Common data movement function
Common data movement function
Common data movement function
None
None
report_upload.py
Common data movement function
Common data movement function
Common data movement function
Common data movement function
None
analytics_job.py
Common data movement function
Common data movement function
None
Common data movement function
Business-specific processing
service_export.sh
Common data movement function
Common data movement function
Common data movement function
Common data movement function
None
data_transform.py
None
Common data movement function
None
None
Business-specific processing
archive_result.py
Common data movement function
Common data movement function
Common data movement function
None
None
Common data movement function
Business-specific processing
Repeated data movement functions
Selecting files to transfer
Transfer between servers and cloud
Handling connection failures and errors
Run conditions and follow-up transfers
Keeping file transfer history
Business-specific functions
The company’s data analysis logic
Data generation in business systems
Industry-specific data processing
Service-specific business logic
Processing unique to existing systems
Keep the software the business really needs,and separate out the repeated data movement.
What had to go was not the programs themselves, but the development and operations work being repeated for no reason.
Live contextNow
1 week laterWhat 386 programs really do
Operating environment
386Existing programs running
Servers involved24
Current work
Survey existing programsIn progress
Classify file transfer functionsIn progress
Separate business-specific logicIn progress
Up next
Choose the first job to switch
A new choice, 1 month later
The first job, built as a Flow.
The company changed how it operated, one job at a time.
The first job chosen for the switch was the daily analytics data delivery. Several programs were involved: selecting files, transferring, retrying and delivering results.
The development team keeps the existing systems that run the analysis, and the programs that do the company’s own data processing, exactly as they are.
Instead, the part that delivers the files and starts the follow-up transfer when completion conditions are met was built as an INNORIX Flow.
They decided to compare the results with the old way for a while before putting it into production.
First switch: daily analytics data processing
The company doesn’t change every program at once. It started with the analytics data delivery job: it runs on a schedule, and the work before and after each transfer is clearly defined.
How it ran before
Separating the work
How it runs now
find_daily_files.pyFind files to transfer
send_analytics.shSend to the analytics server
retry_transfer.pyRetry failed transfers
analytics_engineThe company’s own data analysis
watch_result.pyWatch for result files
publish_result.shForward analysis results
write_transfer_log.pyLog transfer runs
Several programs each play their own part, connected to a separate analytics system.
Work to keep
The company’s analytics system
Data analysis
Data processing
Generating analysis results
Work to switch
Data movement
Selecting and delivering files
Transfer recovery
Forwarding result files
Logging transfer runs
Separate the company’s own analytics, and build the parts that move data as a Flow.
Daily analytics data
INNORIX Flow
Trigger
Daily source data is ready
Endpoints
Business server, analytics server
Transfer rules
Selected files only, recover on error
Existing external system
The company’s data analysis
The existing analytics program creates the result files.
Follow-up data delivery
Deliver analysis result files
Runs when result files are created
The existing analytics system stays, while data movement and follow-up transfers run in the Flow environment.
The work itself doesn’t change.What changes is how data moves between the steps.
Without rebuilding existing systems, they started by changing the data movement.
Live contextNow
1 month laterThe first job, built as a Flow
Operating environment
386Existing programs running
1Flows built
Servers involved24
Current work
Existing analytics systemKept
New data delivery FlowPilot run
Compare with existing resultsIn progress
Up next
Verify the production switchover
A new choice, 2 months later
They didn’t shut down the old programs right away.
OPERATING VERIFICATION
The new way of operating has to produce the same results as the old one.
Normal transfer
Connection failure test
Follow-up transfer check
Existing program
Normal run
File delivery
Completed
Error check
Normal
What happens next
Existing process kept
INNORIX Flow
Normal run
File delivery
Completed
Error check
Normal
What happens next
Results confirmed
Existing program
Connection error
File delivery
Stopped after a partial delivery
Error check
Check each log separately
What happens next
Check the rerun policy
INNORIX Flow
Transfer recovery under way
File delivery
Completed after the connection recovered
Error check
Shown in the run details
What happens next
Follow-up runs after the transfer completes
Existing program
Downstream program run
File delivery
Completed
Error check
Normal
What happens next
Check a separate run log
INNORIX Flow
Follow-up transfer run
File delivery
Completed
Error check
Normal
What happens next
Check the related transfer history
Checked before switching over
Transfer targets and result filesMatch
Storage locations and handling rulesMatch
Downstream work in external systemsVerified
Failure recovery and rerun behaviorVerified
Nothing was switched over until it was verified.
Live contextNow
2 months laterThey didn’t shut down the old programs right away.
Operating environment
386Existing programs running
1Flows built
Servers involved24
Current work
Existing programsIn operation
Compare results in a test environmentIn progress
Verify failure recovery and rerunsIn progress
Up next
The first switchover
A new choice, 3 months later
The work of 28 programs
What 28 programs used to handle now runs as 6 Flows.
After verifying the first switched job, the company moved on to switching the related data movement work.
The file transfer and run control work of 28 existing programs was rebuilt as 6 Flows. Each Flow carries the endpoints, run conditions and follow-up transfers its job needs.
The old programs weren’t deleted right away; they were kept for a while as the new results were checked. The company’s own data processing programs kept running as before.
First switchover complete
How it ran before
28File transfer programs
Daily file discovery and selection, 4
Source data transfer, 6
Transfer retry, 4
Analysis result watchers, 3
Result data delivery, 7
Run logging and alerts, 4
How it runs now
6Flows, one per job
Daily source data deliveryIn operation
Scheduled analytics data prepIn operation
Analysis result forwardingIn operation
Service operations data deliveryIn operation
Project data deliveryIn operation
Long-term data archivingIn operation
Existing company systems
Data generation, analysis, business processing and the company’s own programs keep running as before.
Not how many programs they manage,but which work they manage became clear.
They started managing the work they run, not the number of programs.
Live contextNow
3 months laterThe first switchover
Operating environment
358Existing programs running
6Flows built
Servers involved24
Current work
First switched areaIn operation
Existing programsStopped and archived
Next switch candidatesUnder review
Up next
Extend the switch to other areas
The company changes, 6 months later
A similar request came in again.
The request looked familiar, but the response was different.
A new analytics environment came in, and with it the need to deliver source data and analysis results.
Developers first looked at what extra processing the existing systems needed, and worked out the file movement requirements with the operations team.
The operations team checks the connected devices and storage, then builds the data and run conditions it needs as a Flow.
The development team could focus on the project’s data analysis and service features.
N
New project team
We need the daily data in the new analytics environment too.Please deliver the source data, and when the analysis is done, send the result files to the project storage.
Much like the request that came in six months ago.
6 months ago
Development team’s work
Survey existing programs
Write integration code for the new environment
Implement file selection and run conditions
Build error handling and retries
Implement follow-up transfers
Implement logging and run history
Test and deploy to production
Today
Operations team’s work
Connect the new devices and storage
Select the data to transfer
Set up run conditions and follow-up transfers
Test the transfer and go live
Development team
Building new analytics features
Builds the data processing and analytics features unique to the business.
Development work and data movement operationsnow play different roles.
A new file transfer job no longer had to mean a new program.
Live contextNow
6 months laterA new development request arrives.
Operating environment
214Existing programs running
37Flows built
Servers involved24
Current work
New analysis logicDev team
Data delivery FlowBuilt by operations
Existing workRunning normally
Up next
New project goes live
The company changes, 1 year later
The same failure happened again.
08:32
ONE YEAR LATER, MONDAY
Service operations data has been delivered.
Runs
Daily service data
Analysis result data deliveryCompleted
Source: Analytics storage / Target: Service environment
Run history
08:00Scheduled job started
08:05Connection lost
08:12Connection recovered, transfer resumed
08:32File transfer completed
Follow-up data deliveryStarted
08:05 Interrupted
08:12 Recovered
08:18 Resumed
08:32 Completed
What’s different now
The job owner can see the run results, how it recovered, and the status of the follow-up transfer.
The company didn’t become one where nothing fails.What changed is how it responds.
Live contextNow
1 year laterA new way of operating
Operating environment
82Existing programs running
64Flows built
Servers involved24
Current work
Recurring data movementRun as Flows
Business-specific logicExisting programs kept
New work designBuilt around the work
Up next
Ongoing operations
The company changes, 1 year later
Did all of the company’s programs disappear?
What the company manages has changed.
Over the year, the switch happened step by step, starting with the file transfer work that recurs.
Of the 386 programs, 304 have stopped running. 82 remain, because they carry business-specific logic or depend on existing systems.
The operations team checks each job’s run conditions, endpoints and results across 64 Flows.
When a new file transfer request comes in, they first check whether an existing Flow can be extended or a new Flow built.
ONE YEAR OF TRANSITION
The company did not switch every program.
At the start of the switch
386
Programs in operation
1 year later
82programs kept
82 kept304 stopped
64Flows in operation
Programs still running
Business-specific data processing
Programs tied to existing systems
Special-purpose automation
Programs to review for a later switch
Work run as Flows
Data delivery between servers and cloud
Condition-based file transfer
Follow-up transfers after completion
Transfer recovery and result checks
They separated what to build in-house from what to run on a platform.
These figures are operating changes set in a fictional company scenario and do not represent any real customer’s results.
The company still has its own in-house programs. It just doesn’t build every piece of data movement itself.
The company today, Today
What developers should build
The company still builds new software.
New services and analytics features are being built, and so is the software unique to the business.
But there is far less need to rebuild, in every program, the code that selects and delivers files, handles errors and starts the next transfer when one finishes.
Developers build the software the business needs; operators manage the company’s data movement.
MONDAY, 10:00
A new project has started.
S
Service planning team
We’re building a service on top of existing analytics data and new user data. Please get the data delivery ready as well.
Development team
Projects in progress
Designing new service features
Developing data analysis logic
Building user-facing features
Validating and improving the service
Builds the new software the business needs.
Infrastructure operations team
INNORIX, Flows
Existing analytics data deliveryIn operation
New user data deliveryNew
Service result data deliveryNew
Scheduled data archivingIn operation
Builds and runs the data movement the service needs.
As before, the company builds new software.It just doesn’t rebuild every data movement feature each time.
Focus on what you need to build, and run what you need to run.
Live contextNow
TodayWhat developers should build
Operating environment
82Existing programs running
64Flows built
Servers involved24
Current work
Business-specific service developmentIn progress
Data movement workRunning normally
New work requestsReviewed as Flows
Up next
New projects ahead
TODAY
This company stillbuilds new software.
New services launch, new analytics environments arrive,and the software unique to the business keeps being built.
Where the story began
386
File transfer programs
Operations built around individual programs
The company, one year later
82
Existing programs
64
Flow
Business-specific work and data movement, separated
The existing systems the business needs stayed in place.
The company keeps building its own software.
The data movement it used to build again and again now runs as Flows.
Focus on what you need to build,and run what you need to run.
INNORIX
THE TECHNOLOGY BEHIND THIS STORY
Multi-Stage Flow
INNORIX: build the file transfer and run control you kept re-developing as Flows, and run them alongside the systems you already have.