INNORIX in Practice, Fleet-Scale Distribution

At 9 a.m. tomorrow,
a new service starts nationwide.

How thousands of devices spread across the country get ready for one service launch. Follow the company through its 30 days of preparation.

D-01Final distribution

THE DAY BEFORE LAUNCH

D-0118:00

15 hours left
until the launch.

The last distribution of the new service files to devices nationwide is under way.

Sites

240

Devices

6,000

Where the new service starts at 9 a.m. tomorrow

Capital region90 sites, 2,250

Central region50 sites, 1,250

Southern region80 sites, 2,000

Other regions20 sites, 500

A story set in a fictional company, showing how files are actually distributed and operated to prepare a large service launch.

INNORIX in practiceScene 1 / 14

Launch preparationD-30

Launch preparation D-30Scene 1 / 14

In a month, a new service starts nationwide.

The company has set the launch date of the new service it spent a year preparing.

6,000 devices running at 240 sites across the country are due to offer the new service.

New service launch plan

D-DAY

09:00

Nationwide simultaneous launch

Planned launch date

Service scope

Nationwide

Sites

240

Devices

6,000

TeamLaunch task
Service planningSet the launch date and operating policy
DevelopmentPrepare the programs and service resources
DistributionDeliver the files every device needs
Field operationsCheck device environments and validate
Service operationsApply and start the service

The service development team prepares the programs and resources; the distribution team plans how to deliver them to every device in the country.

A service being developed and a service being able to start nationwide are two different things.

The launch date is set. Now the devices across the country have to be ready.

AT THIS MOMENTNow

D-30

Launch planning

Until launch

30 days

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Launch planning

  • Target devices6,000
  • Files delivered0

Launch readiness

  • Field validationPlanned
  • Common resources deliveredPlanned
  • Final files deliveredPlanned
  • Service startPlanned

Next

Prepare a real field validation environment

Next story, D-21Device environments checked →

Launch preparation D-21Scene 2 / 14

Not all 6,000 devices share one environment.

Over the years the company has brought in many kinds of devices at different times.

Operating systems, program environments and each site’s network are checked, and the service files are sorted accordingly.

Devices receiving the new service

Model A

Windows

3,000

Model program

4.2GB

Model configuration

Model B

Linux

1,800

Model program

3.6GB

Model configuration

Model C

Embedded Linux

1,200

Model program

2.8GB

Model configuration

Common service resources

Content and data every device uses

Files for all 6,000 devices

24GB

File environment kept on each device

Current service

/service/current/

Existing service keeps running

New service

/service/next/

Files prepared before launch

The common resources every device needs and the program files of each model are set up separately.

So the existing service is not affected, the distribution team decides to deliver the new service’s files to a separate path.

One service nationwide, but the files to prepare differ from device to device.

AT THIS MOMENTNow

D-21

Devices and file layout

Until launch

21 days

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Devices and file layout

  • Target devices6,000
  • Files delivered0

Launch readiness

  • Field validationPlanned
  • Common resources deliveredPlanned
  • Final files deliveredPlanned
  • Service startPlanned

Next

Pre-validation at 6 sites

Next story, D-14First distribution to 150 devices →

Field validation D-14Scene 3 / 14

Try it in the field before distributing nationwide.

The company has chosen 6 sites with different operating environments.

The new service files go to 150 of the 6,000 devices.

Nationwide target

6,000

Field validation target

150

6 sites chosen out of 240

Validation siteEnvironmentDevices
Site AHigh-speed network25
Site BStandard network25
Site CRestricted network25
Site DMostly older devices25
Site EMostly new devices25
Site FMixed devices25

Distribution team, 14:30

The new service files reached 150 devices at 6 sites.

Service validation team, 15:00

We will start validating the new service on the real devices.

Site staff and the development team run the programs on real devices and validate the service environment.

Validation shows that one model’s configuration file has to be changed.

Before distributing nationwide, check it in the real operating environment first.

AT THIS MOMENTNow

D-14

Field validation

Until launch

14 days

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Field validation

  • Target devices150
  • Files delivered150

Launch readiness

  • Field validationPlanned
  • Common resources deliveredPlanned
  • Final files deliveredPlanned
  • Service startPlanned

Next

Prepare the fixed file

Next story, D-12A problem found in the field →

Field validation D-12Scene 4 / 14

Not every file had to be sent again.

The development team analysed the problem found during validation.

Only the configuration file of the affected Model B devices needs fixing. The large common resources and the other models’ program files have not changed.

Field validation team10:20

Some service screens on Model B devices do not pick up their settings. Please check the configuration file of those devices.

Development team11:45

We found the cause and fixed the configuration file. The programs and common resources do not need to be sent again.

Changed files

FileSizeChange
service_resources.zip24GBUnchanged
model_b_application.zip3.6GBUnchanged
model_b_config_v1.1.json128KBModified

All validation devices

150

Fixed file target

30

Fixed file delivered and revalidated

The fixed configuration file goes to the 30 validation devices of that model.

Validation is run again, and the nationwide distribution can now be prepared.

Changed files go again only to the devices that need them.

AT THIS MOMENTNow

D-12

Fixed file validation

Until launch

12 days

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Fixed file validation

  • Target devices30
  • Files delivered30

Launch readiness

  • Field validationDone
  • Common resources deliveredPlanned
  • Final files deliveredPlanned
  • Service startPlanned

Next

Approve nationwide distribution

Next story, D-7Large resources sent early →

Nationwide preparation D-7Scene 5 / 14

There is no need to wait for every file until the day before launch.

The new service needs 24GB of common resources on each device.

Instead of sending every file just before launch, the company starts distributing the large resources that will not change a week ahead.

Files for the nationwide launch

Files that can go out before launch day

Common service resources

24GB, all devices

From D-7
Content and service data

Model program files

2.8~4.2GB, by model

From D-5

Final configuration and content changes

About 80MB, final version

On D-1

A rough picture to show the difference in file sizes.

Depending on each site’s network and devices, direct, regional and grid distribution are used.

The files wait in the new service’s path while the existing service keeps running.

The launch date and the date files are delivered do not have to be the same.

AT THIS MOMENTNow

D-7

Common resource pre-distribution

Until launch

7 days

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Common resource pre-distribution

  • Target devices6,000
  • Files delivered4,200

Launch readiness

  • Field validationDone
  • Common resources deliveredPlanned
  • Final files deliveredPlanned
  • Service startPlanned

Next

Prepare programs and final files

Next story, D-5Every site distributes differently →

Nationwide preparation D-5Scene 6 / 14

Every site distributes differently.

Not every site has the same network.

Some sites can take large files straight from headquarters. Other regions can use a regional server, and some sites have devices suited to grid distribution.

Common service resource distribution

The same files, delivered different ways.

Direct distribution

Sites that receive files straight from headquarters

80 sites, 2,000 devices

Source server

Delivered directly

Regional distribution

Files go to a regional hub, then to each site’s devices

100 sites, 2,500 devices

Hub A

Hub B

Hub C

Hub D

Grid distribution

A distributed environment spreads the load of a large distribution

60 sites, 1,500 devices

Distributed environment

Nationwide target

240 sites, 6,000 devices

The way files arrive differs by region, but the service and launch date the company must prepare are the same.

The distribution team uses the method that suits each environment and checks the overall result.

AT THIS MOMENTNow

D-5

Common resource pre-distribution

Until launch

5 days

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Common resource pre-distribution

  • Target devices6,000
  • Files delivered5,400

Launch readiness

  • Field validationDone
  • Common resources deliveredPlanned
  • Final files deliveredPlanned
  • Service startPlanned

Next

Prepare programs and final files

Next story, D-3Pre-distribution complete →

Nationwide preparation D-3Scene 7 / 14

144TB of data is already on the devices nationwide.

The common resources have reached all 6,000 devices.

The logical volume of common resources delivered across all devices is 144TB.

3 days before launch

Common service resources delivered nationwide

Files delivered

6,000 / 6,000

Delivered to every device

Logical data delivered to all devices

144TB

24GB × 6,000 devices

RegionTarget devicesCompleted
Capital region2,2502,250
Central region1,2501,250
Southern region2,0002,000
Other regions500500

Files the launch needs

Common service resources
Delivered
Model program files
Delivered
Final configuration and changes
Due on D-1

Each model’s program files are delivered in advance as well. Only the final configuration files, not yet fixed, remain.

The amount of data to deliver the day before launch is now far smaller.

The data that could go before launch day is already in place.

AT THIS MOMENTNow

D-3

Common resources delivered

Until launch

3 days

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Common resources delivered

  • Target devices6,000
  • Files delivered6,000

Launch readiness

  • Field validationDone
  • Common resources deliveredDone
  • Final files deliveredPlanned
  • Service startPlanned

Next

Fix the final version

Next story, 18:00Final service files fixed →

THE DAY BEFORE LAUNCH

And then it was the day before launch.

The new service, 29 days in the making.

Time until launch

15 : 00 : 00

Now 18:00, service starts 09:00 tomorrow

Common service resources

24GB

Pre-distributed

Model programs

2.8~4.2GB

Pre-distributed

Final changes

About 80MB

Distribution starting

Now the last files go to every device in the country.

INNORIX in practiceScene 1 / 14

Launch preparationD-30

The day before 18:00Scene 8 / 14

The final files for 9 a.m. tomorrow are ready.

The development and service operations teams have fixed the final service version.

The distribution team starts delivering each device’s final configuration and content files.

Service development team17:50

The final version of tomorrow’s service is fixed.

The common resources and program files delivered earlier have not changed. Please deliver only the final configuration files and the changed content.

Final release package

Final configuration and changed content, About 80MB per device

Distribution team18:00

Confirmed. We are starting the final distribution to every device.

Release distribution

Final release files

Dataset
Final release
Target devices
6,000
Destination
/service/next/
Started
18:00

Running

The large resources and program files delivered earlier stay as they are.

The existing service keeps running nationwide as usual.

What goes out today is not the whole service, only the files fixed last.

AT THIS MOMENTNow

18:00

Final distribution starts

Until launch

15 hours

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Final distribution starts

  • Target devices6,000
  • Files delivered0

Launch readiness

  • Field validationDone
  • Common resources deliveredDone
  • Final files deliveredPlanned
  • Service startPlanned

Next

Check the final delivery results

Next story, 23:40Devices not ready yet →

The day before 23:40Scene 9 / 14

9 hours 20 minutes until launch. 38 devices are still left.

5,962 devices nationwide have received the final files.

Current time

23:40

Until launch

9 h 20 min

Final service file distribution

5,962 / 6,000

99.37% complete

38 remaining

Devices to check

StatusDevicesResponse
Disconnected24Waiting for the connection
Low storage10Site team responding
Powered off4Waiting for power-on
Transferring0None in progress

The 38 initially unfinished devices

  • Disconnected, low storage
  • Powered off
  • Transferring
  • Delivered
  1. 18:00
  2. 23:40
  3. 02:10
  4. 04:50
  5. 05:30
  6. 06:12

5,962 devices have their files. 38 devices still need attention.

Of the 38 remaining, 24 are disconnected, 10 are low on storage and 4 are powered off.

The distribution team separates the 5,962 finished devices from the other 38.

The site support team finds the sites and devices that need help and starts responding.

Instead of restarting the whole distribution, the team focuses on the devices not yet finished.

AT THIS MOMENTNow

23:40

Unfinished devices

Until launch

9 h 20 min

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Unfinished devices

  • Target devices6,000
  • Files delivered5,962
  • Unfinished38

Launch readiness

  • Field validationDone
  • Common resources deliveredDone
  • Final files deliveredPlanned
  • Service startPlanned

Next

Restore connections and storage

Next story, 05:30The last 4 devices →

Launch day 05:30Scene 10 / 14

The last remaining devices are connected.

As connections came back and storage was freed, 34 more transfers finished.

A site manager also switched on the 4 devices that had been powered off.

Launch day before dawn, site operations log

  1. 02:10

    Disconnected devices finish their transfers

    Where the network came back, the interrupted transfers resumed and 24 devices received their files.

    Remaining, 14 devices

  2. 03:30

    Devices low on storage checked

    Site support team

    We checked the devices low on storage. We will clear unneeded temporary files and free the space they need.

  3. 04:50

    Storage freed, transfers complete

    The 10 devices whose storage problem was solved received their files.

    Remaining, 4 devices

  4. 05:30

    The last devices switched on

    Site manager

    We switched on the 4 devices that were due to start this morning.

    The devices connect and the remaining file deliveries go ahead.

The last devices start receiving their files.

The team checks each device’s final result and gets ready to hand it to the service operations team.

A distribution to thousands of devices ends only when the last device’s result is confirmed.

AT THIS MOMENTNow

05:30

Last device transfers

Until launch

3 h 30 min

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Last device transfers

  • Target devices6,000
  • Files delivered5,996
  • Transferring4

Launch readiness

  • Field validationDone
  • Common resources deliveredDone
  • Final files deliveredPlanned
  • Service startPlanned

Next

Check the final transfer results

Next story, 07:00All files delivered →

Launch day 07:00Scene 11 / 14

All 6,000 devices nationwide have the files they need.

The distribution team checks each device’s delivery records for common resources, programs and final configuration files.

The final distribution result is shared with the service operations team.

LAUNCH DAY, 07:00

Every device nationwide has all the files it needs.

Final release distribution

Started yesterday 18:00

Completed

Target devices

6,000

Completed

6,000

Incomplete

0

DatasetDevicesStatus
Common resources6,000Completed
Model A application3,000Completed
Model B application1,800Completed
Model C application1,200Completed
Final release files6,000Completed

Distribution team, 07:05

Every file the launch needs has been delivered. We also checked the delivery records of all 6,000 devices.

Service operations team, 07:10

Confirmed. We will apply the service and run the final operating checks.

The service operations team runs its own deployment and service validation steps.

File delivery and readiness to run the service are each confirmed.

The service operations team now does its final work with every file it needs in place.

AT THIS MOMENTNow

07:00

Final distribution complete

Until launch

2 hours

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Final distribution complete

  • Target devices6,000
  • Files delivered6,000

Launch readiness

  • Field validationDone
  • Common resources deliveredDone
  • Final files deliveredDone
  • Service startPlanned

Next

Apply the service and run final checks

Next story, 08:55Every site ready for launch →

Launch day 08:55Scene 12 / 14

Every site is ready for launch.

5 minutes until the service starts.

A month of work by different teams comes to an end.

08:55

Each team’s last preparation

5 min until launch

TeamTaskResult
DevelopmentFinal service version fixedDone
DistributionFiles delivered to all devicesDone
Field operationsSite devices checkedDone
Service operationsService applied and validatedDone
Service planningLaunch readiness confirmedDone

Service start

09:00

240 sites nationwide

AT THIS MOMENTNow

08:55

Each team’s last preparation

Until launch

5 min

Launch, D-DAY 09:00

Sites

240

All devices

6,000

Current distribution

Each team’s last preparation

  • Target devices6,000
  • Files delivered6,000

Launch readiness

  • Field validationDone
  • Common resources deliveredDone
  • Final files deliveredDone
  • Service startPlanned

Next

Service start

Next story, 09:00Service starts nationwide →

Launch day 09:00

A new service starts nationwide.

9 a.m. The company’s service systems start the new service on schedule.

LAUNCH DAY

09:00

The new service has started
nationwide.

Capital region90 sites, 2,250 devices

Central region50 sites, 1,250 devices

Southern region80 sites, 2,000 devices

Other regions20 sites, 500 devices

240

Sites

6,000

Devices

09:00

Service start

Site staff at 240 sites check that the service is in use.

Customers anywhere in the country can now use the newly prepared service.

The file distribution of the last 30 days was the preparation for one business date.

For customers, it is an ordinary morning on which a new service began.

After launch D+30

A major project a month ago is everyday work now.

Since launch, new content and some program files keep changing.

30 DAYS AFTER LAUNCH

After launch, the files keep changing.

DistributionTarget devicesSchedule
Common service content6,000Daily
Regional content changesRegional devicesWeekly
Model A program changes3,000On change
Model B configuration1,800On change
New service feature filesSelected devicesBy validation schedule

Service planning team

We are adding a new feature next month. Please prepare it so a few sites can validate it first.

Distribution team

We will use the validation sites and distribution environment we already set up.

The company runs its later file distribution with the device groups, datasets and distribution tasks it set up for the launch.

Delivering changed files only to the devices that need them, and checking the result, has become routine.

Teams preparing the next service launch can use the existing distribution environment too.

A service launches once, but running it goes on.

ONE LAUNCH. THIRTY DAYS OF PREPARATION.

For customers, it was an ordinary morning.

9 a.m., when the new service became available nationwide.

For that moment, the company prepared countless files and devices over 30 days.

  1. D-30

    Launch decided

    Devices and work checked nationwide

  2. D-14

    Field validation

    Checked in the real environment

  3. D-7

    Pre-distribution

    Large resources sent early

  4. D-DAY

    Service starts

    New work at the planned time

Not every file was sent the day before launch.

Not every device got the same distribution environment.

A problem on a few devices did not restart the whole distribution.

When the service has to start, the files it needs are already there.

INNORIX

The technology behind this story

Fleet-Scale Distribution

INNORIX delivers and manages the files thousands of devices need, on the schedule of a company’s important business dates.

View the product