INNORIX in Practice / Server & Cloud-to-Cloud Transfer
A day when not everything went to plan
A day that started like any other. Look inside a company’s data operations as one unexpected problem follows another.
A company’s data operations run on set schedules and plans.
In the morning the data for analysis is prepared; in the afternoon the files for new services are delivered; in the evening the day's business data is archived.
But today was different.
The network failed before dawn, and by mid-morning the server that should receive data had run out of space. In the afternoon a server rebooted mid-transfer, and just before close someone had to confirm whether an important file had been delivered.
A day when not everything went to plan.
How did this company finish its day?
WEDNESDAY
18:00
The company's data operations log
End of business day
Daily operations
Current statusEnd of business day
06:12
09:48
11:24
14:16
16:40
18:00
A story built from a day of data operations in a fictional company.
A company’s data operations run on set schedules and plans.
In the morning the data for analysis is prepared, in the afternoon the files for new services are delivered, and in the evening the day's business data is archived.
But today was different.
The network failed before dawn, and by mid-morning the server that should receive data had run out of space. In the afternoon a server rebooted mid-transfer, and just before close someone had to confirm whether an important file had been delivered.
A day when not everything went to plan.
How did this company finish its day?
This company runs servers in its own data center alongside several cloud environments.
Data from the business systems is kept in Amazon S3 and analyzed in Google Cloud. Customer services run on Microsoft Azure, and an in-house Linux Server handles long-term retention.
Every day, many kinds of transfers run between these systems.
Some must finish before the morning’s work begins. Others deliver the data for afternoon analytics or service updates.
THE COMPANY
The company's data environment
Infrastructure
Business use
Main data
Windows Server
Business systems
Business transaction data
Amazon S3
Shared data storage
Business and service data
Google Cloud
Data analytics
Analysis input and results
Microsoft Azure
Service operations
Service input data
Linux Server
Long-term retention
Archive data
Today's scheduled data jobs
Starts
Transfer job
Size
Needed by
06:00
Morning business data delivery
480 GB
09:00
09:00
Business data archiving
2.4 TB
15:00
10:00
Large analytics data delivery
3.2 TB
14:00
13:00
Service data update
100,000 files
16:00
15:30
Daily settlement data delivery
18.6 GB
17:30
At 6 a.m., the first data job of the day started.
So far, everything is going to plan.
Live contextNow
06:00Wednesday
Affected business work
Morning work prep
Scheduled for
09:00
Current status
Running as planned
Today's incidents
None
Other work
All transfers normal
NETWORK INTERRUPTION, 06:12
The network dropped while the morning’s work was being prepared.
It is 6:12 a.m.
A transfer from the company’s business server to Amazon S3 has suddenly stopped.
This morning’s work needs 480 GB of data in total. 128 GB has been delivered so far, and 352 GB is still to go.
This data must be ready by 9 a.m.
06:12, INNORIX STUDIO
Morning business data delivery
StatusWaiting for connection
Source
Windows Server
Target
Amazon S3
480 GB128 GB, Waiting
06:0006:1206:20
Total data
480 GB
Transferred
128 GB
Remaining
352 GB
Until the connection is back, this transfer switches to waiting.
Transfers between other servers and clouds that do not use this network keep running.
The company has a problem with one network connection. Its data operations as a whole have not stopped.
06:20, RUNS
Other jobs running at the same time
Flow
Status
Morning business data deliveryWindows Server / Amazon S3
Waiting
Service reference data deliveryGoogle Cloud / Azure
The search shows the file was delivered successfully at 3:42 p.m.
The operations team checks the transfer start and finish times and the destination path.
Live contextNow
16:41Wednesday
Affected business work
Daily settlement
Scheduled for
17:30
Current status
Searching transfer history
Today's incidents
06:12 Network connection lostResolved
09:48 Insufficient storageResolved
11:24 Server rebootResolved
14:16 Partial file failureResolved
16:40 File check requestChecking
16:43
The problem was not the file transfer.
FILE TRANSFER HISTORY
daily_settlement.parquet
Final resultCompleted
Source
Google Cloud Storage
/settlement/output/
Started 15:30
Target
Azure Blob Storage
/settlement/input/
Completed 15:42
File size
11.2 GB
Started
15:30:08
Completed
15:42:16
Transfer errors
None
Final result
Completed
File transfer
Delivered 15:42
Settlement work
Done 17:18 after the downstream program was fixed
The operations team sent the service operations team the delivery time and storage location.
Checking the file at that path and the downstream program, the service operations team found that the program reading the settlement data had briefly stopped.
There was no need to resend the file.
The service operations team gets the downstream program running again and completes settlement with the data already delivered.
At 5:18 p.m., settlement is done.
Live contextNow
16:43Wednesday
Affected business work
Daily settlement
Scheduled for
17:30
Current status
Fixing the downstream system
Today's incidents
06:12 Network connection lostResolved
09:48 Insufficient storageResolved
11:24 Server rebootResolved
14:16 Partial file failureResolved
16:40 File check requestChecking
END OF BUSINESS DAY, 18:00
A day when not everything went to plan is over.
Today brought more problems than usual.
The network dropped, storage ran out and a server rebooted. Some files were not delivered because of access permissions, and a business system could not process a file that had already arrived.
The operations team reviews how today’s data operations turned out.
DAILY OPERATIONS REVIEW
Today’s problems and how they ended
Time
What happened
How it was handled
Final status
06:12
Network connection lost
Resumed after reconnecting
Recovered
09:48
Insufficient storage
Resumed after freeing space
Recovered
11:24
Server reboot
Resumed after server reconnected
Recovered
14:16
Some files not delivered
Permissions fixed, failed files retried
Completed
16:40
Important file check
History checked, downstream system fixed
Resolved
It was not a day without problems.
It was a day when each problem got the handling it needed, when it happened.
Some problems were handled automatically once the connection came back. Others needed the operations team to act.
And where the file transfer was fine, the transfer history helped narrow the cause down to another system.
So how did the company’s work go today?
Compare today’s transfer problems with the company’s actual business schedule.
BUSINESS TIMELINE
Data delivery times against the business schedule
06:0009:0012:0015:0018:00
Morning work starts
Data archiving
Afternoon analytics
Service data update
Daily settlement
Data transfer (start to ready)
Scheduled time
Business task
Data ready
Scheduled
Result
Morning work starts
08:14
09:00
Started on time
Data archiving
12:08
15:00
Done before deadline
Afternoon analytics
13:42
14:00
Started on time
Service data update
14:42
16:00
Applied on time
Daily settlement
15:42
17:30
Done 17:18
Today’s problems changed when several file transfers ran.
But the data each task needed was ready before its scheduled time, and the final settlement work also finished before the deadline once the downstream system was fixed.
What this company tracks closely is not just whether a file transfer hit an error.
It is which data an error affected, how it was handled and what it meant for the business in the end.
Live contextNow
18:00Wednesday
Affected business work
End of business day
Scheduled for
18:00
Current status
Today's work done
Today's incidents
06:12 Network connection lostResolved
09:48 Insufficient storageResolved
11:24 Server rebootResolved
14:16 Partial file failureResolved
16:40 File check requestResolved
A DAY WHEN THINGS WENT WRONG
Today, not everythingwent to plan.
Today's problems
What happened to the work
06:12The network connection dropped.
08:14Morning business data ready
09:48Storage ran out.
12:08Data archived
11:24A server rebooted.
13:42Analytics data ready
14:16Some files were not delivered.
14:42All service files delivered
16:40An important file had to be checked.
17:18Daily settlement done
18:00 The company finished its work for the day.
Unexpected things can happen at any time.
What matters is whether the businesscan keep going after a problem.
THE TECHNOLOGY BEHIND THIS STORY
Server & Cloud-to-Cloud Transfer
Run file transfers between different servers and clouds, handle the exceptions that come up along the way, and confirm where every file finally landed.