INNORIX in Practice / Embedded Web Upload & Download
The file our customer wants to submit is 500 GB.
A change that started with one customer inquiry.
See how a business changes when the amount of data its web service can accept changes.
A web service that has run smoothly for years receives an inquiry from a new customer.
The data the customer wants to submit is 500 GB, on a different scale from any file this service has handled.
The operations and development teams start looking for a way to accept it. And this one request ends up changing the range of services the company offers.
The story runs from the Monday morning the inquiry arrives, through completing the new customer’s work, to two years later, when the company serves customers with even larger data.
TWO YEARS LATER
Projects of every size come in.
The company now serves different customers.
500 GB
The source file the customer wants to submit
Customer projects
Project A
12 GB
Project B
820 GB
Project C
4.2 TB
Inquiry
The change
First project
Two years on
The story of one fictional web service and the customers who use it.
THE STORY
How one customer’s request changed what a service could do
The company runs a web service: customers submit source data, and the company analyzes and processes it and delivers the results.
Customers create a project on the website, submit the files it needs and download the finished results.
On Monday morning, an inquiry arrives from a new enterprise customer.
NEW CUSTOMER INQUIRY, 09:12
Inquiry about large source data
Enterprise customer, new customer
Hello. We would like to use your data processing service.
The source data for this project, which needs analysis and processing, is about 500 GB. We plan to run projects of a similar size regularly.
Can we submit the data directly on your website?
original_dataset.tar
Data to submit, project source data
500 GB
Data sizes the web service takes today
Typical submission2~5 GB
Current single-file limit10 GB
New customer request500 GB
Until now, most data from this company’s customers was a few GB.
The web service’s existing file upload feature handled that well.
But this customer’s file is 50 times the largest file size allowed today.
The operations team asks the data processing lead whether they can take on the work.
Live contextNow
A new customer inquiryMonday 09:12
Customer inquiry
Can we submit 500 GB of source data directly on your website?
Requested data
500 GB
Current service limit
10 GB
Next task
Check whether the project is feasible
Monday, 10:30
We can process the data, but we can’t receive it.
The data processing team reviewed the work the customer requested.
The company’s analytics skills and processing infrastructure can easily handle the project.
Processing capacity was not the problem.
The customer had no way to get the data to the company.
INTERNAL PROJECT REVIEW
New customer project: review results
Data processing team
Can do the work
Data analysisPossible
Data processingPossible
Processing infrastructureIn place
Result data generationPossible
Web service operations team
Cannot submit data
Customer’s source file
500 GB
Web service file size limit
10 GB
Data processing lead10:34
Analysis and processing are no problem. Once we have the source data, we can run the project.
Service operations manager10:36
Our website can’t accept a file this size today. We’ll need to find another way to receive it.
The company already had the technology and infrastructure to serve the customer.
Yet it cannot start a new project because the customer cannot submit data to the web service.
File transfer was setting the limit on what services the company could offer.
Live contextNow
The service hits its limitMonday 10:30
Data processing lead
Processing it is not a problem. We just need the source data.
Can we process it?
Yes
Data submission
Looking for a way
Next task
Decide how the customer will deliver the data
Monday, 14:00
Should we ask the customer to split the file into 50 parts?
At an afternoon meeting, the team reviews several ways to receive the customer’s data.
They could split the original file, use a separate file sharing service or have a storage device shipped.
DATA DELIVERY OPTIONS
Ways to receive 500 GB of data
Split the file
50 files, if split into 10 GB parts
The customer sends it in parts
Splitting, separate uploads, receipt checks and rebuilding the original.
File sharing service
original_dataset.tar
500 GB
External download link
Delivered outside the web service
Use a separate service
Check its size limits and security terms, then link the data back to the project.
Ship the data
External storage
500 GB data
Prepare and ship a device
Send a storage device
Adds preparing and shipping a device, receiving it and copying the data.
Head of service operations
For this one project, we could use another method.
But if more customers like this keep coming, do we set up a separate delivery method every time?
This customer plans regular work of a similar size, not just a one-off project.
Asking every new customer to split files or use a separate delivery method is not how the company wants to run its service.
Instead of a workaround for this one project, the company decides to improve the web service itself.
Live contextNow
Talking it overMonday 14:00
Head of service operations
If more customers like this keep coming, do we set up a separate delivery method every time?
Delivery methods reviewed
3
Common problem
Extra work per customer
Next task
Decide how to improve the service
Tuesday
The team decided to keep the way customers work.
The development team reviews how the existing web service is built.
Sign-in, project creation, data processing and result delivery already run reliably.
There is no need to rebuild the whole web service.
The company decides to apply INNORIX file transfer technology to the existing file upload and download area.
APPLICATION INTEGRATION
The web service’s file submission screen
Before
Data workspace
Projects / New project
Create a project
Project name
Project Alpha
Original data
Choose file
Maximum file size: 10 GB
A 500 GB file cannot be submitted.
Submit project
With INNORIX
Data workspace
Projects / New project
Create a project
Project name
Project Alpha
Original data
original_dataset.tar
500 GB
Ready to upload
Submit project
Project details, sign-in, data processing and result delivery keep the existing service structure.
The team sets it up so customers create projects and submit data on the same screens as before.
Beyond the transfer feature, they also review storage capacity, server settings, authentication and per-project file storage policies.
The goal is not just to lift the file size limit, but to be ready to receive large data reliably.
Live contextNow
Deciding to improve the serviceTuesday
Development team
We keep the existing screens and apply it only to the file upload and download area.
What stays
Sign-in, projects, processing, results
What changes
File submission area
Next task
Test with data at real scale
1 week later
Transferring a real 500 GB file.
Before going live, the development team prepares a test file about the size of the customer’s real data.
They test not only a normal large-file transfer, but also situations customers might run into during their work.
PRE-RELEASE VALIDATION
Testing large data submission
Staging environment
Transfer completed
test_dataset.tar
Large single file
500 GB
500 GB / 500 GB
Test item
What we checked
Result
Large file upload
Receiving 500 GB of source data
Passed
Network disconnection
Interruption and recovery
Passed
Storage check
File storage and capacity policy
Passed
Source data check
Integrity of received data
Passed
Project status link
Next job runs after completion
Passed
Large result download
Delivering results to the customer
Passed
The team does not stop at confirming that the file was transferred.
They also check that the source data arrived complete, that it links to the right customer project, and that the next job does not start by mistake if the transfer runs into trouble.
The operations team gets ready to roll out the new file transfer feature without affecting existing customers.
Live contextNow
Testing with real data1 week later
Development team
We checked transfer, recovery and download with a 500 GB test file.
Test data
500 GB
Test items
6 / 6 passed
Next task
Tell the customer about the project
TWO WEEKS LATER
The company gets back to the customer.
Customer inquiry, Monday 09:12
Can we submit the data directly on your website?
Project invitation
You can start your project now.
Hello. We are ready to start the data processing project you asked about.
Please sign in to our website as usual, create a project and submit your source data directly.
There is no need to split files or prepare a storage device.
Start project
The customer who could not submit data two weeks ago returns to the same website.
INNORIX in Practice
Monday09:12
2 weeks later, 09:00
The customer submits a 500 GB file.
The customer signs in to the website and creates a new project.
They enter a project name and requirements, then choose the source file they prepared.
FIRST LARGE DATA PROJECT, 09:00
Data workspace
Projects, My account
Projects / New project
Create a project
Project name
Project Alpha
Processing requirements
Original dataset analysis and processing
Original data
original_dataset.tar
Upload started, 09:00
500 GB
Submitted
The customer did not split the file.
Nor did they upload the data to an outside file sharing service first.
Like every other customer, they create a project in the web service and submit the source data.
At the company, staff see that a new project has been created and the data transfer has started.
Live contextNow
The customer returns2 weeks later 09:00
Customer request
Submit source data
Data submitted
500 GB
Current work
Receiving a customer project
Project status
Transfer started
Next task
Finish the data transfer
2 weeks later, 09:48
The customer’s network connection drops.
It has been 48 minutes since the customer started submitting data.
With 205 GB of the 500 GB delivered, the network connection is interrupted.
In the past, when something went wrong during a large transfer, the customer had to start over from the beginning.
PROJECT ALPHA, 09:48
The customer’s web service screen
Submitting your project
Waiting for connection
original_dataset.tar
205 GB / 500 GB
41%
The network connection was interrupted. Data already sent is kept.
Project status
Waiting for source data
Data processing
Waiting to start
09:00 Start
09:48 Stopped
09:51 Restored
11:35 Done
The customer’s screen shows the transfer status.
Data processing at the company has not started yet.
It is set up so that a project is not processed until the file has fully arrived.
Live contextNow
The network connection drops2 weeks later 09:48
Customer transfer status
The network connection was interrupted.
Data received
205 GB
Data processing
Waiting for source data
Next task
Restore the connection and resume
2 weeks later, 09:51
No need to start over.
Three minutes later, the network connection is back.
The customer does not resend the 205 GB already delivered. The transfer simply continues.
TRANSFER CONTINUITY
original_dataset.tar
Transferred
205 GB
Remaining
295 GB
Already sent 41%
Still to send 59%
Transfer activity
09:00Upload started
09:48Connection interrupted
09:51Connection restored
09:51Upload resumed
The customer does not need to create a new project.
The company’s operations staff do not have to ask for the file again either.
The source data keeps arriving in the same project.
Live contextNow
The transfer resumes2 weeks later 09:51
Customer transfer status
The connection is back. The transfer picks up where it stopped.
Data remaining
295 GB
Data to resend
0 GB
Next task
Finish receiving the source data
2 weeks later, 11:35
500 GB of data has arrived.
The source data transfer that started in the morning is complete.
The web service verifies the file and the project details.
Once every check passes, a new job goes to the data processing lead.
DATA RECEIVED, 11:35
Project Alpha, Original data
Completed
original_dataset.tar
All source data received
500 GB
Time
Project activity
Status
09:00
Project created
Done
09:00
Source data upload started
Done
11:35
All data received
Done
11:38
Source data verified
Done
11:39
Processing job created
Done
11:40
Job handed to staff
Done
Data processing lead11:42
We have checked the source data. We will start the analysis and processing you requested.
At the meeting two weeks ago, the data processing team said they could run the project as soon as the source data arrived.
Now 500 GB of data has actually reached the company’s processing environment.
The usual data processing work starts, with no workaround steps added just to move the file.
Live contextNow
Project received2 weeks later 11:35
Data processing lead
The data arrived intact. Starting the analysis.
Source data
500 GB
Project status
Received
Next task
Analyze and process the data
3 days later
The results for the customer are ready.
Analysis and processing are complete.
Staff add the result data and analysis report to the project page in the web service.
The customer signs in to the same website to view the results.
PROJECT RESULTS
Project Alpha
Source data submitted
500 GB
original_dataset.tar
Processed result data
186 GB
processed_dataset.tar
Available downloads
Completed
processed_dataset.tar
186 GB
analysis_report.pdf
24 MB
Download activity
Download completed
processed_dataset.tar
186 GB / 186 GB
From submitting 500 GB of source data to receiving 186 GB of results, everything happened in one web service.
The customer asks whether they can request future projects the same way.
The company no longer needs to set up a separate delivery method for this customer each time.
Live contextNow
Results delivered3 days later
Customer reply
We have the results. We will send next month’s request the same way.
Result data
186 GB
Project status
Completed
Next task
Prepare for the customer’s next project
THREE MONTHS LATER
This time, a customer wants to submit thousands of files.
The customer 3 months ago
500 GB
500 GB
original_dataset.tar
One large file
The new customer
840 GB
12,480 files
Many files across many folders
The new customer wants to send not one large file, but a great many source files.
INNORIX in Practice
Monday09:12
3 months later
Not every customer submits a single file.
A new customer wants to analyze data collected over many years.
The source data is spread across many folders, 12,480 files in all.
With the old web service, the company might have asked the customer to pack everything into one archive.
This time, the company looks at letting the customer submit the data in its existing structure.
NEW CUSTOMER PROJECT
Project Delta
Total data
840 GB
Source files
12,480
Source folders
48
Selected source data
Original dataset/
Region A/
January/
February/
March/
Region B/
January/
February/
March/
Reference data/
48 folders, 12,480 files, 840 GB
Uploading original data
Project Delta
9,485 / 12,480 files638.4 GB / 840 GB
Now, when the company meets a new customer, it can consider not just file size but also the number of files and how they are organized.
Every customer keeps their data differently.
But customers should not have to restructure their data just to use the company’s processing service.
Live contextNow
New customers3 months later
New customer
We want to submit source data spread across many folders, as it is.
Source data
840 GB
Files, folders
12,480, 48
Next task
Receive it with folders intact
6 months later
Several customers submit data at the same time.
As word of the service spread, project requests grew.
Now individual customers submitting a few GB and enterprise customers submitting hundreds of GB use the service at the same time.
Existing customers must be able to keep using the service while large new projects are processed.
TUESDAY, 10:42
Incoming customer projects
Project
Data size
Transfer status
Project P-201
4.8 GB
Completed
Project P-202
320 GB
64%
Project P-203
12 GB
Completed
Project P-204
860 GB
38%
Project P-205
780 MB
Completed
Project P-206
1.2 TB
21%
Service operations review
We check concurrent transfers per customer, network usage, storage and queued processing work together.
To keep up with growing customer work, the operations team reviews network, storage and processing resources and sets the operating policies it needs.
The company no longer just checks one customer’s file size. It manages the capacity of the whole service.
Live contextNow
Projects arriving at once6 months later
Service operations team
We track each customer’s transfer status, storage and queued processing work together.
Projects in progress
6
Data size
780 MB ~ 1.2 TB
Next task
Set operating policies
1 year later
Regular work from enterprise customers begins.
The customer who first submitted 500 GB has kept sending regular processing requests.
Some newly signed enterprise customers submit several TB of data every month.
The web service is growing from a place to receive one-off projects into a place where regular business work gets done.
CUSTOMER PROJECT HISTORY
Enterprise customer, Project history
Monthly data submitted (TB)
Peak 2.4 TB
123456789101112
Example: monthly data from one enterprise customer (Jan to Dec)
Recent projects
October data analysis
Processed and delivered
2.1 TB
November data analysis
Processed and delivered
1.9 TB
December data analysis
Processed and delivered
2.4 TB
Customers do not negotiate a new way to deliver data each time.
Nor does the company set up a separate file delivery process for each customer.
Repeat processing requests have become everyday work in the web service.
Live contextNow
A wider range of services1 year later
Enterprise customer
We received the December analysis results.
Submitted this month
2.4 TB
Project status
Processed and delivered
Next task
Next month’s regular project
2 years later
The company now serves different customers.
The company now takes on data processing projects of every size.
They range from small files from individual customers, to many files across many folders, to several TB of source data from enterprise customers.
CUSTOMER PROJECTS, TWO YEARS LATER
Data workspace
Project type, status
Source data
General data analysisDelivered
2.8 GB4 files
Project source reviewProcessing
48 GB12 files
Enterprise data analysisReviewing
620 GB1 file
Field data processingReceiving data
840 GB12,480 files
Regular data processingProcessing
2.4 TB24 files
Large source analysisNew request
4.2 TB183 files
2 years ago, the first inquiry
Can we submit 500 GB of source data directly on your website?
New customer inquiry
We want to analyze about 4 TB of source data. Here is the analysis we need for this project.
Service operations manager
We will review the requested analysis with our data processing team.
Two years ago, when the first inquiry came in, the operations team’s first question was how to receive 500 GB of data.
This customer works with even more data.
But what the team looks at first now is the analysis the customer wants, not how to receive the files.
The way the company evaluates new customers has changed.
Live contextNow
Different customers, a different service2 years later
New enterprise customer
We want to analyze about 4 TB of data.
New project
4.2 TB
Current work
Reviewing customer requirements
Next task
Plan the analysis and receive the project
TODAY
One customer inquirychanged the range of services the company offers.
2 years ago
Customer inquiry
500 GB
Source data the customer wants to submit
The operations team’s question
How can we receive this data?
Web service file limit, 10 GB
Today
Customer projects
Individual customer
2.8 GB
Enterprise customer
620 GB
Regular project
2.4 TB
New project
4.2 TB
The operations team’s focus today
What services can we offer our customers?
The company already had the technology to process its customers’ data.
What had to change to win new customers was how it received that data.
Large files no longerhold the web service back.
THE TECHNOLOGY BEHIND THIS STORY
Embedded Web Upload & Download
INNORIX embeds directly in your web services and websites to upload and download large files and large numbers of files.