INNORIX in Practice / Server & Cloud-to-Cloud Transfer

A day when not everything
went to plan

A day that started like any other. Look inside a company’s data operations as one unexpected problem follows another.

A company’s data operations run on set schedules and plans.

In the morning the data for analysis is prepared; in the afternoon the files for new services are delivered; in the evening the day's business data is archived.

But today was different.

The network failed before dawn, and by mid-morning the server that should receive data had run out of space. In the afternoon a server rebooted mid-transfer, and just before close someone had to confirm whether an important file had been delivered.

A day when not everything went to plan.

How did this company finish its day?

WEDNESDAY

18:00

The company's data operations log

End of business day

Daily operations

Current statusEnd of business day
A story built from a day of data operations in a fictional company.

INNORIX in Practice

Wednesday, Morning06:00

THE COMPANY, 06:00

A day like any other begins.

A company’s data operations run on set schedules and plans.

In the morning the data for analysis is prepared, in the afternoon the files for new services are delivered, and in the evening the day's business data is archived.

But today was different.

The network failed before dawn, and by mid-morning the server that should receive data had run out of space. In the afternoon a server rebooted mid-transfer, and just before close someone had to confirm whether an important file had been delivered.

A day when not everything went to plan.

How did this company finish its day?

This company runs servers in its own data center alongside several cloud environments.

Data from the business systems is kept in Amazon S3 and analyzed in Google Cloud. Customer services run on Microsoft Azure, and an in-house Linux Server handles long-term retention.

Every day, many kinds of transfers run between these systems.

Some must finish before the morning’s work begins. Others deliver the data for afternoon analytics or service updates.

THE COMPANY

The company's data environment

InfrastructureBusiness useMain data
Windows ServerBusiness systemsBusiness transaction data
Amazon S3Shared data storageBusiness and service data
Google CloudData analyticsAnalysis input and results
Microsoft AzureService operationsService input data
Linux ServerLong-term retentionArchive data

Today's scheduled data jobs

StartsTransfer jobSizeNeeded by
06:00Morning business data delivery480 GB09:00
09:00Business data archiving2.4 TB15:00
10:00Large analytics data delivery3.2 TB14:00
13:00Service data update100,000 files16:00
15:30Daily settlement data delivery18.6 GB17:30

At 6 a.m., the first data job of the day started.

So far, everything is going to plan.

Live contextNow

06:00Wednesday

Affected business work
Morning work prep
Scheduled for
09:00
Current status
Running as planned

Today's incidents

None

Other work

All transfers normal

NETWORK INTERRUPTION, 06:12

The network dropped while the morning’s work was being prepared.

It is 6:12 a.m.

A transfer from the company’s business server to Amazon S3 has suddenly stopped.

This morning’s work needs 480 GB of data in total. 128 GB has been delivered so far, and 352 GB is still to go.

This data must be ready by 9 a.m.

06:12, INNORIX STUDIO

Morning business data delivery

StatusWaiting for connection

Source
Windows Server
Target
Amazon S3
Total data
480 GB
Transferred
128 GB
Remaining
352 GB

Until the connection is back, this transfer switches to waiting.

Transfers between other servers and clouds that do not use this network keep running.

The company has a problem with one network connection. Its data operations as a whole have not stopped.

06:20, RUNS

Other jobs running at the same time

FlowStatus
Morning business data deliveryWindows Server / Amazon S3Waiting
Service reference data deliveryGoogle Cloud / AzureRunning
Overnight analysis results archiveGoogle Cloud / Amazon S3Completed
Service operations log deliveryAzure / Archive ServerRunning

Live contextNow

06:12Wednesday

Affected business work
Morning work prep
Scheduled for
09:00
Current status
Waiting for connection

Today's incidents

  • 06:12 Network connection lostIn progress

Other work

Independent transfers normal

06:36

The network connection is back.

24 minutes later, the company’s network connection came back.

INNORIX runs the interrupted transfer again.

Instead of resending data that was already delivered, it sends the rest from the last point it can resume from.

TRANSFER RECOVERY

Morning business data delivery

StatusCompleted

Stopped
06:12
Resumed
06:36
Completed
08:14

At 8:14 a.m., all of the data was delivered.

The network failure shifted the transfer schedule, but the data needed for the 9 a.m. start was ready.

Live contextNow

06:36Wednesday

Affected business work
Morning work prep
Scheduled for
09:00
Current status
Resuming from where it stopped

Today's incidents

  • 06:12 Network connection lostResumed

09:00

Staff start work as planned.

FIRST BUSINESS DEADLINE

Morning business data ready

StatusCompleted

Problem
Network connection lost
Data ready
08:14
Work starts
09:00

The data was ready 46 minutes before work began.

The first problem of the day is solved.

But 48 minutes later, a new problem appears elsewhere in the company’s data operations.

Live contextNow

09:00Wednesday

Affected business work
Morning work starts
Scheduled for
09:00
Current status
Data ready at 08:14

Today's incidents

  • 06:12 Network connection lostResolved

INSUFFICIENT STORAGE, 09:48

The server that should receive the data is out of space.

Since 9 a.m., the company had been moving business data kept in Amazon S3 to its in-house long-term archive server.

With 1.2 TB of the 2.4 TB delivered, the destination ran out of storage.

The network is fine, and the cloud sending the data is running normally.

But the destination server cannot store any more files.

ARCHIVE SERVER, STORAGE CAPACITY

Business data archiving

Archive ServerInsufficient space

20 TB / 20 TB

  • Existing data 15.8 TB
  • To clean up 3 TB
  • Delivered today 1.2 TB

INNORIX STUDIO, RUN DETAILS

Business data archivingWaiting for storage

Total size
2.4 TB
Transferred
1.2 TB
Remaining
1.2 TB
Free space
0 GB

This transfer is set up to wait until storage is freed.

The operations team sees the low-storage status in INNORIX and uses its existing storage management system to find data that can be cleaned up.

This problem is different from the network drop.

Waiting alone may not fix it, so the operations team has to act.

Live contextNow

09:48Wednesday

Affected business work
Data archiving
Scheduled for
15:00
Current status
Storage must be freed

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageAction needed

10:30

The operations team freed up storage.

Following the company’s retention policy, the operations team cleared 3 TB of data whose retention period had expired.

With space free on the destination server, the waiting transfer runs again.

10:30, STORAGE FREED

Archive Server

Free space 3 TBSpace available

17 TB / 20 TB

Ops team, storage management

INNORIX, transfer resumed

TaskStatus
Ops team frees storageDone
Business data archivingResuming
Data left to send1.2 TB
Transfer rebuildNot needed

All the operations team had to do was free up storage.

They did not reselect the 2.4 TB, sort through delivered files to find what was left, or rebuild the transfer from scratch.

At 12:08 p.m., all of the data is archived.

Live contextNow

10:30Wednesday

Affected business work
Data archiving
Scheduled for
15:00
Current status
Resumed after freeing space

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResumed

WEDNESDAY, 11:24

This time, a serverrebooted mid-transfer.

INNORIX STUDIO, ACTIVE TRANSFER

Large analytics data deliveryInterrupted

1.86 TB / 3.2 TB, 58.1%

Source

Linux Compute Server

Disconnected

Target

Google Cloud Storage

Connected

The analytics work starting at 2 p.m. needs the full 3.2 TB.

INNORIX in Practice

Wednesday, Morning06:00

UNEXPECTED REBOOT, 11:24

When a server restarts, does the transfer have to start over?

The morning’s connection failure has been fixed, and the storage shortage is being handled.

This time, though, the problem is not the network but the server sending the data.

Since 10 a.m., large analytics data had been moving from a Linux Compute Server to Google Cloud Storage.

With 1.86 TB of the 3.2 TB delivered, the server rebooted without warning.

Every program running on the server lost its connection.

The analytics team needs this data from 2 p.m.

DEVICE AND TRANSFER HISTORY

Linux Compute Server

TimeDevice statusTransfer status
10:00:00ConnectedRunning
11:24:08DisconnectedInterrupted
11:27:42Server restartedWaiting
11:30:54ConnectedRecovery check
11:31:06ConnectedResuming

Transfer in progress before the reboot

analytics_dataset.bin 3.2 TB

The server starts up normally and INNORIX reconnects.

The source file and the transfer state are still there, so the interrupted transfer can recover and deliver the rest of the data.

Live contextNow

11:24Wednesday

Affected business work
Afternoon analytics
Scheduled for
14:00
Current status
Waiting for server

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResumed
  • 11:24 Server rebootIn progress

Other work

Independent transfers normal

11:31

The server is back and the transfer has resumed.

RESUMING AFTER REBOOT

The 3.2 TB file that was in transit

analytics_dataset.binCompleted

Already delivered
1.86 TB
Not resent
Remaining
1.34 TB
Total delivered
3.2 TB

At 1:42 p.m., all of the analytics data is delivered.

Live contextNow

11:31Wednesday

Affected business work
Afternoon analytics
Scheduled for
14:00
Current status
Sending the remainder

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResumed
  • 11:24 Server rebootResumed

14:00

The analytics team starts its afternoon work.

AFTERNOON ANALYTICS

The 3.2 TB needed for analysis is ready.

Google Cloud Storage / analytics/input/

analytics_dataset.bin

Available

3.2 TB, received 13:42

Data ready
13:42
Analytics starts
14:00

Despite the server reboot, the analytics team starts on schedule.

Just 16 minutes later, another problem turns up in the delivery of the company’s service data.

Live contextNow

14:00Wednesday

Affected business work
Afternoon analytics
Scheduled for
14:00
Current status
Data ready at 13:42

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResolved
  • 11:24 Server rebootResolved

PARTIAL TRANSFER FAILURE, 14:16

37 of 100,000 files were not delivered.

The company’s service operations team plans to apply new service data at 4 p.m.

A total of 100,000 files have to go from Amazon S3 to Azure Blob Storage.

During the transfer, a read permission problem showed up on some source files.

37 files could not be read, but the rest were fine.

The company had set a policy to keep delivering the other files even when individual files fail.

INNORIX STUDIO, RUN DETAILS

Service data update

Amazon S3 / Azure Blob StoragePartial failure

Total files
100,000
Succeeded
99,963
Failed
37

Failed files

File nameReason
service_reference_02841.jsonAccess denied
service_reference_03918.jsonAccess denied
service_reference_12753.jsonAccess denied
service_reference_18462.jsonAccess denied

Some of the 37 failed files

The success rate is 99.963%.

But the service operations team does not need a high success rate. It needs every file the service depends on to be in place.

The remaining 37 files have to be delivered too.

Live contextNow

14:16Wednesday

Affected business work
Service data update
Scheduled for
16:00
Current status
37 files need handling

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResolved
  • 11:24 Server rebootResolved
  • 14:16 Partial file failureAction needed

14:28

The problem with the 37 files is found.

The operations team checked the list of failed files and why they failed.

They fix the access permissions on those files in the source cloud, then have INNORIX resend only the failed files.

The 99,963 files already delivered are not selected again.

RETRY FAILED FILES, 14:28

Resend failed files

Already delivered

99,963

Not resent

To resend

37

Fixed files only

Files delivered

100,000 / 100,000

Completed

At 2:42 p.m., the last 37 files were delivered.

The operations team focused on the files with problems and did not resend everything else.

Live contextNow

14:28Wednesday

Affected business work
Service data update
Scheduled for
16:00
Current status
Retrying failed files only

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResolved
  • 11:24 Server rebootResolved
  • 14:16 Partial file failureIn progress

16:00

The service operations team applies the data as planned.

SERVICE DATA UPDATE

Every file needed for the service update is ready.

Files received
100,000
All delivered
14:42
Data applied
16:00

All four transfer problems from today have been handled.

Every team at the company is getting on with its planned work.

But just before the day wraps up, a new request reaches the operations team.

Live contextNow

16:00Wednesday

Affected business work
Service data update
Scheduled for
16:00
Current status
Service data applied

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResolved
  • 11:24 Server rebootResolved
  • 14:16 Partial file failureResolved

WHERE IS THE FILE?, 16:40

Just before close, one important file has to be traced.

16:40, SERVICE OPERATIONS TEAM

“Please check whether today’s settlement data reached the service system.”

File to check

daily_settlement.parquet

Created in
Google Cloud Storage
Delivered to
Azure Blob Storage
Work due by
17:30

The service operations team found that the program processing settlement results had not picked up today’s settlement data.

Today brought a network drop, a storage shortage, a server reboot and some failed file deliveries.

No one knows which of these problems, if any, affected this file.

The operations team needs to find out how far the file actually got.

Live contextNow

16:40Wednesday

Affected business work
Daily settlement
Scheduled for
17:30
Current status
Checking delivery history

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResolved
  • 11:24 Server rebootResolved
  • 14:16 Partial file failureResolved
  • 16:40 File check requestChecking

16:41

Searching all of today’s transfer records for the file.

Instead of logging in to each server and cloud to hunt for the file, the operations team searches today’s transfer history in INNORIX.

INNORIX STUDIO, FILE SEARCH

Search transfer history

Search: daily_settlement.parquet, period: today, all devices

File nameSizeCompletedStatus
daily_settlement.parquetGoogle Cloud Storage / Azure Blob Storage11.2 GB15:42Completed

1 matching files

The search shows the file was delivered successfully at 3:42 p.m.

The operations team checks the transfer start and finish times and the destination path.

Live contextNow

16:41Wednesday

Affected business work
Daily settlement
Scheduled for
17:30
Current status
Searching transfer history

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResolved
  • 11:24 Server rebootResolved
  • 14:16 Partial file failureResolved
  • 16:40 File check requestChecking

16:43

The problem was not the file transfer.

FILE TRANSFER HISTORY

daily_settlement.parquet

Final resultCompleted

Source

Google Cloud Storage

/settlement/output/

Started 15:30

Target

Azure Blob Storage

/settlement/input/

Completed 15:42

File size
11.2 GB
Started
15:30:08
Completed
15:42:16
Transfer errors
None
Final result
Completed

File transfer

Delivered 15:42

Settlement work

Done 17:18 after the downstream program was fixed

The operations team sent the service operations team the delivery time and storage location.

Checking the file at that path and the downstream program, the service operations team found that the program reading the settlement data had briefly stopped.

There was no need to resend the file.

The service operations team gets the downstream program running again and completes settlement with the data already delivered.

At 5:18 p.m., settlement is done.

Live contextNow

16:43Wednesday

Affected business work
Daily settlement
Scheduled for
17:30
Current status
Fixing the downstream system

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResolved
  • 11:24 Server rebootResolved
  • 14:16 Partial file failureResolved
  • 16:40 File check requestChecking

END OF BUSINESS DAY, 18:00

A day when not everything went to plan is over.

Today brought more problems than usual.

The network dropped, storage ran out and a server rebooted. Some files were not delivered because of access permissions, and a business system could not process a file that had already arrived.

The operations team reviews how today’s data operations turned out.

DAILY OPERATIONS REVIEW

Today’s problems and how they ended

TimeWhat happenedHow it was handledFinal status
06:12Network connection lostResumed after reconnectingRecovered
09:48Insufficient storageResumed after freeing spaceRecovered
11:24Server rebootResumed after server reconnectedRecovered
14:16Some files not deliveredPermissions fixed, failed files retriedCompleted
16:40Important file checkHistory checked, downstream system fixedResolved

It was not a day without problems.

It was a day when each problem got the handling it needed, when it happened.

Some problems were handled automatically once the connection came back. Others needed the operations team to act.

And where the file transfer was fine, the transfer history helped narrow the cause down to another system.

So how did the company’s work go today?

Compare today’s transfer problems with the company’s actual business schedule.

BUSINESS TIMELINE

Data delivery times against the business schedule

Business taskData readyScheduledResult
Morning work starts08:1409:00Started on time
Data archiving12:0815:00Done before deadline
Afternoon analytics13:4214:00Started on time
Service data update14:4216:00Applied on time
Daily settlement15:4217:30Done 17:18

Today’s problems changed when several file transfers ran.

But the data each task needed was ready before its scheduled time, and the final settlement work also finished before the deadline once the downstream system was fixed.

What this company tracks closely is not just whether a file transfer hit an error.

It is which data an error affected, how it was handled and what it meant for the business in the end.

Live contextNow

18:00Wednesday

Affected business work
End of business day
Scheduled for
18:00
Current status
Today's work done

Today's incidents

  • 06:12 Network connection lostResolved
  • 09:48 Insufficient storageResolved
  • 11:24 Server rebootResolved
  • 14:16 Partial file failureResolved
  • 16:40 File check requestResolved

A DAY WHEN THINGS WENT WRONG

Today, not everythingwent to plan.

  1. 06:12The network connection dropped.

    08:14Morning business data ready

  2. 09:48Storage ran out.

    12:08Data archived

  3. 11:24A server rebooted.

    13:42Analytics data ready

  4. 14:16Some files were not delivered.

    14:42All service files delivered

  5. 16:40An important file had to be checked.

    17:18Daily settlement done

18:00 The company finished its work for the day.

Unexpected things can happen at any time.

What matters is whether the businesscan keep going after a problem.

THE TECHNOLOGY BEHIND THIS STORY

Server & Cloud-to-Cloud Transfer

Run file transfers between different servers and clouds, handle the exceptions that come up along the way, and confirm where every file finally landed.

Explore the product