INNORIX in Practice / Embedded Web Upload & Download

The file our customer wants
to submit is 500 GB.

A change that started with one customer inquiry.

See how a business changes when the amount of data its web service can accept changes.

A web service that has run smoothly for years receives an inquiry from a new customer.

The data the customer wants to submit is 500 GB, on a different scale from any file this service has handled.

The operations and development teams start looking for a way to accept it. And this one request ends up changing the range of services the company offers.

The story runs from the Monday morning the inquiry arrives, through completing the new customer’s work, to two years later, when the company serves customers with even larger data.

TWO YEARS LATER

Projects of every size come in.

The company now serves different customers.

500 GB

The source file the customer wants to submit

Customer projects

Project A
12 GB
Project B
820 GB
Project C
4.2 TB
The story of one fictional web service and the customers who use it.

INNORIX in Practice

Monday09:12

Monday, 09:12

A new customer inquiry has arrived.

The company runs a web service: customers submit source data, and the company analyzes and processes it and delivers the results.

Customers create a project on the website, submit the files it needs and download the finished results.

On Monday morning, an inquiry arrives from a new enterprise customer.

NEW CUSTOMER INQUIRY, 09:12

Inquiry about large source data

Enterprise customer, new customer

Hello. We would like to use your data processing service.

The source data for this project, which needs analysis and processing, is about 500 GB. We plan to run projects of a similar size regularly.

Can we submit the data directly on your website?

original_dataset.tar

Data to submit, project source data

500 GB

Data sizes the web service takes today

  • Typical submission2~5 GB
  • Current single-file limit10 GB
  • New customer request500 GB

Until now, most data from this company’s customers was a few GB.

The web service’s existing file upload feature handled that well.

But this customer’s file is 50 times the largest file size allowed today.

The operations team asks the data processing lead whether they can take on the work.

Live contextNow

A new customer inquiryMonday 09:12

Customer inquiry

Can we submit 500 GB of source data directly on your website?

Requested data
500 GB
Current service limit
10 GB

Next task

Check whether the project is feasible

Monday, 10:30

We can process the data, but we can’t receive it.

The data processing team reviewed the work the customer requested.

The company’s analytics skills and processing infrastructure can easily handle the project.

Processing capacity was not the problem.

The customer had no way to get the data to the company.

INTERNAL PROJECT REVIEW

New customer project: review results

Data processing team

Can do the work
  • Data analysisPossible
  • Data processingPossible
  • Processing infrastructureIn place
  • Result data generationPossible

Web service operations team

Cannot submit data
Customer’s source file
500 GB
Web service file size limit
10 GB

Data processing lead10:34

Analysis and processing are no problem. Once we have the source data, we can run the project.

Service operations manager10:36

Our website can’t accept a file this size today. We’ll need to find another way to receive it.

The company already had the technology and infrastructure to serve the customer.

Yet it cannot start a new project because the customer cannot submit data to the web service.

File transfer was setting the limit on what services the company could offer.

Live contextNow

The service hits its limitMonday 10:30

Data processing lead

Processing it is not a problem. We just need the source data.

Can we process it?
Yes
Data submission
Looking for a way

Next task

Decide how the customer will deliver the data

Monday, 14:00

Should we ask the customer to split the file into 50 parts?

At an afternoon meeting, the team reviews several ways to receive the customer’s data.

They could split the original file, use a separate file sharing service or have a storage device shipped.

DATA DELIVERY OPTIONS

Ways to receive 500 GB of data

  • Split the file

    50 files, if split into 10 GB parts

    The customer sends it in parts

    Splitting, separate uploads, receipt checks and rebuilding the original.

  • File sharing service

    original_dataset.tar

    500 GB

    External download link

    Delivered outside the web service

    Use a separate service

    Check its size limits and security terms, then link the data back to the project.

  • Ship the data

    External storage

    500 GB data

    Prepare and ship a device

    Send a storage device

    Adds preparing and shipping a device, receiving it and copying the data.

Head of service operations

For this one project, we could use another method.

But if more customers like this keep coming, do we set up a separate delivery method every time?

This customer plans regular work of a similar size, not just a one-off project.

Asking every new customer to split files or use a separate delivery method is not how the company wants to run its service.

Instead of a workaround for this one project, the company decides to improve the web service itself.

Live contextNow

Talking it overMonday 14:00

Head of service operations

If more customers like this keep coming, do we set up a separate delivery method every time?

Delivery methods reviewed
3
Common problem
Extra work per customer

Next task

Decide how to improve the service

Tuesday

The team decided to keep the way customers work.

The development team reviews how the existing web service is built.

Sign-in, project creation, data processing and result delivery already run reliably.

There is no need to rebuild the whole web service.

The company decides to apply INNORIX file transfer technology to the existing file upload and download area.

APPLICATION INTEGRATION

The web service’s file submission screen

Before

Data workspace

Projects / New project

Create a project

Project name

Project Alpha

Original data

Choose file

Maximum file size: 10 GB

A 500 GB file cannot be submitted.
Submit project

With INNORIX

Data workspace

Projects / New project

Create a project

Project name

Project Alpha

Original data

original_dataset.tar

500 GB

Ready to upload
Submit project

Project details, sign-in, data processing and result delivery keep the existing service structure.

The team sets it up so customers create projects and submit data on the same screens as before.

Beyond the transfer feature, they also review storage capacity, server settings, authentication and per-project file storage policies.

The goal is not just to lift the file size limit, but to be ready to receive large data reliably.

Live contextNow

Deciding to improve the serviceTuesday

Development team

We keep the existing screens and apply it only to the file upload and download area.

What stays
Sign-in, projects, processing, results
What changes
File submission area

Next task

Test with data at real scale

1 week later

Transferring a real 500 GB file.

Before going live, the development team prepares a test file about the size of the customer’s real data.

They test not only a normal large-file transfer, but also situations customers might run into during their work.

PRE-RELEASE VALIDATION

Testing large data submission

Staging environment

Transfer completed

test_dataset.tar

Large single file

500 GB

500 GB / 500 GB

Test itemWhat we checkedResult
Large file uploadReceiving 500 GB of source dataPassed
Network disconnectionInterruption and recoveryPassed
Storage checkFile storage and capacity policyPassed
Source data checkIntegrity of received dataPassed
Project status linkNext job runs after completionPassed
Large result downloadDelivering results to the customerPassed

The team does not stop at confirming that the file was transferred.

They also check that the source data arrived complete, that it links to the right customer project, and that the next job does not start by mistake if the transfer runs into trouble.

The operations team gets ready to roll out the new file transfer feature without affecting existing customers.

Live contextNow

Testing with real data1 week later

Development team

We checked transfer, recovery and download with a 500 GB test file.

Test data
500 GB
Test items
6 / 6 passed

Next task

Tell the customer about the project

TWO WEEKS LATER

The company gets back to the customer.

Customer inquiry, Monday 09:12

Can we submit the data directly on your website?

Project invitation

You can start your project now.

Hello. We are ready to start the data processing project you asked about.

Please sign in to our website as usual, create a project and submit your source data directly.

There is no need to split files or prepare a storage device.

Start project

The customer who could not submit data two weeks ago returns to the same website.

INNORIX in Practice

Monday09:12

2 weeks later, 09:00

The customer submits a 500 GB file.

The customer signs in to the website and creates a new project.

They enter a project name and requirements, then choose the source file they prepared.

FIRST LARGE DATA PROJECT, 09:00

Data workspace

Projects, My account

Projects / New project

Create a project

Project name

Project Alpha

Processing requirements

Original dataset analysis and processing

Original data

original_dataset.tar

Upload started, 09:00

500 GB

Submitted

The customer did not split the file.

Nor did they upload the data to an outside file sharing service first.

Like every other customer, they create a project in the web service and submit the source data.

At the company, staff see that a new project has been created and the data transfer has started.

Live contextNow

The customer returns2 weeks later 09:00

Customer request
Submit source data
Data submitted
500 GB
Current work
Receiving a customer project
Project status
Transfer started

Next task

Finish the data transfer

2 weeks later, 09:48

The customer’s network connection drops.

It has been 48 minutes since the customer started submitting data.

With 205 GB of the 500 GB delivered, the network connection is interrupted.

In the past, when something went wrong during a large transfer, the customer had to start over from the beginning.

PROJECT ALPHA, 09:48

The customer’s web service screen

Submitting your project

Waiting for connection

original_dataset.tar

205 GB / 500 GB

41%

The network connection was interrupted. Data already sent is kept.

Project status
Waiting for source data
Data processing
Waiting to start
  1. 09:00 Start
  2. 09:48 Stopped
  3. 09:51 Restored
  4. 11:35 Done

The customer’s screen shows the transfer status.

Data processing at the company has not started yet.

It is set up so that a project is not processed until the file has fully arrived.

Live contextNow

The network connection drops2 weeks later 09:48

Customer transfer status

The network connection was interrupted.

Data received
205 GB
Data processing
Waiting for source data

Next task

Restore the connection and resume

2 weeks later, 09:51

No need to start over.

Three minutes later, the network connection is back.

The customer does not resend the 205 GB already delivered. The transfer simply continues.

TRANSFER CONTINUITY

original_dataset.tar

Transferred
205 GB
Remaining
295 GB
  • Already sent 41%
  • Still to send 59%

Transfer activity

  1. 09:00Upload started
  2. 09:48Connection interrupted
  3. 09:51Connection restored
  4. 09:51Upload resumed

The customer does not need to create a new project.

The company’s operations staff do not have to ask for the file again either.

The source data keeps arriving in the same project.

Live contextNow

The transfer resumes2 weeks later 09:51

Customer transfer status

The connection is back. The transfer picks up where it stopped.

Data remaining
295 GB
Data to resend
0 GB

Next task

Finish receiving the source data

2 weeks later, 11:35

500 GB of data has arrived.

The source data transfer that started in the morning is complete.

The web service verifies the file and the project details.

Once every check passes, a new job goes to the data processing lead.

DATA RECEIVED, 11:35

Project Alpha, Original data

Completed

original_dataset.tar

All source data received

500 GB

TimeProject activityStatus
09:00Project createdDone
09:00Source data upload startedDone
11:35All data receivedDone
11:38Source data verifiedDone
11:39Processing job createdDone
11:40Job handed to staffDone

Data processing lead11:42

We have checked the source data. We will start the analysis and processing you requested.

At the meeting two weeks ago, the data processing team said they could run the project as soon as the source data arrived.

Now 500 GB of data has actually reached the company’s processing environment.

The usual data processing work starts, with no workaround steps added just to move the file.

Live contextNow

Project received2 weeks later 11:35

Data processing lead

The data arrived intact. Starting the analysis.

Source data
500 GB
Project status
Received

Next task

Analyze and process the data

3 days later

The results for the customer are ready.

Analysis and processing are complete.

Staff add the result data and analysis report to the project page in the web service.

The customer signs in to the same website to view the results.

PROJECT RESULTS

Project Alpha

Source data submitted
500 GB
original_dataset.tar
Processed result data
186 GB
processed_dataset.tar

Available downloads

Completed

processed_dataset.tar

186 GB

analysis_report.pdf

24 MB

Download activity

Download completed

processed_dataset.tar

186 GB / 186 GB

From submitting 500 GB of source data to receiving 186 GB of results, everything happened in one web service.

The customer asks whether they can request future projects the same way.

The company no longer needs to set up a separate delivery method for this customer each time.

Live contextNow

Results delivered3 days later

Customer reply

We have the results. We will send next month’s request the same way.

Result data
186 GB
Project status
Completed

Next task

Prepare for the customer’s next project

THREE MONTHS LATER

This time, a customer wants to submit thousands of files.

The customer 3 months ago

500 GB

500 GB

original_dataset.tar

One large file

The new customer

840 GB

12,480 files

Many files across many folders

The new customer wants to send not one large file, but a great many source files.

INNORIX in Practice

Monday09:12

3 months later

Not every customer submits a single file.

A new customer wants to analyze data collected over many years.

The source data is spread across many folders, 12,480 files in all.

With the old web service, the company might have asked the customer to pack everything into one archive.

This time, the company looks at letting the customer submit the data in its existing structure.

NEW CUSTOMER PROJECT

Project Delta

Total data
840 GB
Source files
12,480
Source folders
48

Selected source data

  • Original dataset/
  • Region A/
  • January/
  • February/
  • March/
  • Region B/
  • January/
  • February/
  • March/
  • Reference data/

48 folders, 12,480 files, 840 GB

Uploading original data

Project Delta
9,485 / 12,480 files638.4 GB / 840 GB

Now, when the company meets a new customer, it can consider not just file size but also the number of files and how they are organized.

Every customer keeps their data differently.

But customers should not have to restructure their data just to use the company’s processing service.

Live contextNow

New customers3 months later

New customer

We want to submit source data spread across many folders, as it is.

Source data
840 GB
Files, folders
12,480, 48

Next task

Receive it with folders intact

6 months later

Several customers submit data at the same time.

As word of the service spread, project requests grew.

Now individual customers submitting a few GB and enterprise customers submitting hundreds of GB use the service at the same time.

Existing customers must be able to keep using the service while large new projects are processed.

TUESDAY, 10:42

Incoming customer projects

ProjectData sizeTransfer status
Project P-2014.8 GBCompleted
Project P-202320 GB64%
Project P-20312 GBCompleted
Project P-204860 GB38%
Project P-205780 MBCompleted
Project P-2061.2 TB21%

Service operations review

We check concurrent transfers per customer, network usage, storage and queued processing work together.

To keep up with growing customer work, the operations team reviews network, storage and processing resources and sets the operating policies it needs.

The company no longer just checks one customer’s file size. It manages the capacity of the whole service.

Live contextNow

Projects arriving at once6 months later

Service operations team

We track each customer’s transfer status, storage and queued processing work together.

Projects in progress
6
Data size
780 MB ~ 1.2 TB

Next task

Set operating policies

1 year later

Regular work from enterprise customers begins.

The customer who first submitted 500 GB has kept sending regular processing requests.

Some newly signed enterprise customers submit several TB of data every month.

The web service is growing from a place to receive one-off projects into a place where regular business work gets done.

CUSTOMER PROJECT HISTORY

Enterprise customer, Project history

Monthly data submitted (TB)

Peak 2.4 TB

Example: monthly data from one enterprise customer (Jan to Dec)

Recent projects

  • October data analysis

    Processed and delivered
    2.1 TB
  • November data analysis

    Processed and delivered
    1.9 TB
  • December data analysis

    Processed and delivered
    2.4 TB

Customers do not negotiate a new way to deliver data each time.

Nor does the company set up a separate file delivery process for each customer.

Repeat processing requests have become everyday work in the web service.

Live contextNow

A wider range of services1 year later

Enterprise customer

We received the December analysis results.

Submitted this month
2.4 TB
Project status
Processed and delivered

Next task

Next month’s regular project

2 years later

The company now serves different customers.

The company now takes on data processing projects of every size.

They range from small files from individual customers, to many files across many folders, to several TB of source data from enterprise customers.

CUSTOMER PROJECTS, TWO YEARS LATER

Data workspace

Project type, statusSource data
General data analysisDelivered2.8 GB4 files
Project source reviewProcessing48 GB12 files
Enterprise data analysisReviewing620 GB1 file
Field data processingReceiving data840 GB12,480 files
Regular data processingProcessing2.4 TB24 files
Large source analysisNew request4.2 TB183 files

2 years ago, the first inquiry

Can we submit 500 GB of source data directly on your website?

New customer inquiry

We want to analyze about 4 TB of source data. Here is the analysis we need for this project.

Service operations manager

We will review the requested analysis with our data processing team.

Two years ago, when the first inquiry came in, the operations team’s first question was how to receive 500 GB of data.

This customer works with even more data.

But what the team looks at first now is the analysis the customer wants, not how to receive the files.

The way the company evaluates new customers has changed.

Live contextNow

Different customers, a different service2 years later

New enterprise customer

We want to analyze about 4 TB of data.

New project
4.2 TB
Current work
Reviewing customer requirements

Next task

Plan the analysis and receive the project

TODAY

One customer inquirychanged the range of services the company offers.

2 years ago

Customer inquiry

500 GB

Source data the customer wants to submit

The operations team’s question

How can we receive this data?

Web service file limit, 10 GB

Today

Customer projects

Individual customer
2.8 GB
Enterprise customer
620 GB
Regular project
2.4 TB
New project
4.2 TB

The operations team’s focus today

What services can we offer our customers?

The company already had the technology to process its customers’ data.

What had to change to win new customers was how it received that data.

Large files no longerhold the web service back.

THE TECHNOLOGY BEHIND THIS STORY

Embedded Web Upload & Download

INNORIX embeds directly in your web services and websites to upload and download large files and large numbers of files.

Explore the product