INNORIX in Practice, Fleet-Scale Distribution

An error was found in a file already delivered to thousands of devices.

The delivery was done, but the work was not. Follow the story of a day when something nobody expected went wrong.

10:12Error found

TUESDAY MORNING

10:12

An urgent message arrives from the development team.

Development team, 10:12

We found an error in the Model B configuration file distributed yesterday. Please check where that file has gone.

configuration_v4.2.json

Configuration file distributed last night

Devices in service nationwide

8,000 devices

Capital area3,000

Central1,800

South2,400

Other regions800

The reach of the faulty file has to be confirmed.

A story set in a fictional company, showing how operations respond to a problem found after a large-scale file distribution.

INNORIX in practiceScene 1 / 15

The incidentNight before 22:00

The incident, Night before, 22:00Scene 1 / 15

Last night, new files went out as usual.

The company runs 8,000 devices across 320 sites nationwide.

Last night the scheduled distribution of a new program and configuration file began for the 4,000 Model B devices.

THE NIGHT BEFORE

22:00

Scheduled distribution starts

Model A

2,400 devices

Service as usual

Model B

4,000 devices

In this scheduled distribution

Model C

1,600 devices

Service as usual

Files in this scheduled distribution

FileSizeVersion
application_v4.2.zip3.6GB4.2
configuration_v4.2.json128KB4.2
service_resources.zip24GBExisting resource kept

The 24GB shared resources were already delivered by an earlier distribution.

Files were reaching devices one after another, as each device's schedule and connection allowed.

AT THIS MOMENTNow

Night before 22:00

Scheduled distribution running

All devices in service

8,000 devices

NextDevices start work

Next story, 10:12An urgent message →

The incident, 10:12Scene 2 / 15

An error is found in a file already distributed.

At 10:12 in the morning, the development team finds an error in a configuration file.

Development team10:12

We found an error in the Model B configuration file distributed yesterday.

Some business conditions may not be processed correctly. Please stop distributing this file and check which devices already have it.

configuration_v4.2.json

128KB, Version 4.2

Error confirmed

Distribution team10:14

We're checking the distribution that includes this file. We'll stop any further runs and check the delivery result for each device.

All devices in service

8,000 devices

Targets of this file

4,000 devices

At this point nobody yet knows how many devices received the file, or how many were affected.

On some devices that applied the file, certain business tasks may not be processed correctly.

The team confirms the exact version of the faulty file and the distribution that carried it.

AT THIS MOMENTNow

10:12

Investigating the scope

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices

NextStop the distribution

Next story, 10:18Stopping the distribution →

Confirming the scope, 10:18Scene 3 / 15

The faulty file must go no further.

The existing Model B distribution and its repeating runs are stopped.

Existing distribution

Model B release 4.2

Started yesterday, 22:00

Stopped

Target devices

4,000

Completed

3,840

Incomplete

160

Current run
Stopped 10:18
Scheduled runs
Disabled

Stopping the run and disabling the schedule are confirmed separately. Stopping does not recall files already delivered.

Devices that already received the file do not go back to their previous state because the distribution stops.

The team checks the result of the stop and the last transfer records.

AT THIS MOMENTNow

10:18

Distribution under control

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices

NextSearch the delivery history

Next story, 10:25Devices that got the file →

Confirming the scope, 10:25Scene 4 / 15

There was no need to check all 8,000 devices.

The team checks each device's transfer result by the faulty file and the run that delivered it.

Of the 4,000 Model B devices, 3,840 completed the distribution. The other 160 have not received the complete file yet.

All devices in operation, 8,000

  • Received the faulty file3,840
  • Earlier distribution incomplete160
  • Not affected4,000

File delivery history

Search

configuration_v4.2.json

Run, Model B release 4.2

Since 22:00 yesterday

3,840 devices

Delivery records for this file

DeviceLocationDelivered
Device-B-0012Capital area22:14
Device-B-0258Central23:32
Device-B-1092South02:18
Device-B-2840Other regions07:25
Device-B-3812Capital area10:05

3,840 completed deliveries, sample of the transfer history

Whether the program is actually applied is checked in a separate service operations system.

AT THIS MOMENTNow

10:25

Scope confirmed

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices

NextPrepare the fix

Next story, 10:35One standard nationwide →

10:35Scene 5 / 15

One standard for the whole country.

The devices that received the file are spread across the country.

Instead of collecting different results region by region, the team shares the confirmed device list with the field teams.

Scope by region

RegionAll devicesModel BFaulty fileIncomplete
Capital area3,0001,5001,45050
Central1,80090086535
South2,4001,2001,15050
Other regions80040037525
Total8,0004,0003,840160

Distribution team, 10:35

We have the full list of devices that received the faulty file. Sharing the targets by region.

Service team, 10:38

We'll check whether those devices applied the file and what it affected. Let us know when the fix is ready.

The development team now knows which file to fix, and the distribution team knows which devices to send it to.

Field teams can work on the devices that need attention, not on every site.

AT THIS MOMENTNow

10:35

Sharing the scope

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices

NextPrepare the fix

Next story, 10:45Finding the cause →

Preparing the fix, 10:45Scene 6 / 15

The problem was a 128KB configuration file.

The development team confirms the cause is a single configuration file, not the program as a whole.

The 3.6GB program and the 24GB shared resources do not need to change.

Preparing the fix

There was no need to send the 3.6GB program again.

Shared service resources

service_resources.zip

24GB

Existing file kept

Program file

application_v4.2.zip

3.6GB

Configuration file

configuration_v4.2.1.json

128KB

A rough picture of the difference in file sizes.

Previous file

configuration_v4.2.json

Version 4.2

Error confirmed, no redistribution

Corrected file

configuration_v4.2.1.json

Version 4.2.1

Field verification next

The development team prepares the corrected configuration file.

AT THIS MOMENTNow

10:45

Developing the fix

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices

NextVerify on real devices

Next story, 11:00Verifying the fix on site →

Preparing the fix, 11:00Scene 7 / 15

This time, check before going nationwide.

From the devices that received the faulty file, 40 at different sites are selected.

After the corrected file is delivered, field staff verify it in the real business environment.

Pre-verifying the fix

All targets of the fix

4,000 devices

Verified first

40 devices

Capital area

10 devices

Central

10 devices

South

10 devices

Other regions

10 devices

TaskTeamResult
Deliver the fixDistribution team40 done
Apply the fixService team40 done
Recheck the affected taskField teamNormal

Service team, 11:15

The 40 devices with the corrected file are processing work normally. Please go ahead nationwide.

Once verification is done, the company decides to deliver the fix to every Model B device in the country.

AT THIS MOMENTNow

11:00

Verifying the fix

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices
Corrected file delivery

40 / 4,000

Not yet delivered
3,960 devices
Back to normal
40 devices

NextApprove urgent nationwide distribution

Next story, 11:20Distributing the fix →

READY FOR CORRECTION

The corrected file is ready.

Now the right file has to reach devices across the country.

Up to here, the story was about confirming the reach of the problem and preparing the fix.

From here, it is about delivering the right file to devices nationwide and getting back to normal.

Devices that received the faulty file

3,840 devices

Fix verified on 40

Devices that never fully received it

160 devices

Send the right file first

All targets of the corrected file

4,000 devices

40 verified, 3,960 still to send

INNORIX in practiceScene 1 / 15

The incidentNight before 22:00

Urgent distribution, 11:20Scene 8 / 15

The same file, a different purpose for each device.

The 3,840 devices that received the faulty file get the corrected one, to prepare the fix.

Urgent distribution plan

Received the faulty file

3,840 devices

Fix pre-verified

40 done

Fix still to send

3,800 devices

Earlier distribution incomplete

160 devices

Keep the faulty version out

Right configuration file

160 devices

Program delivery checked separately

New distribution

Model B configuration correction

Dataset
Configuration v4.2.1
Target devices
3,960
File
configuration_v4.2.1.json
Size per device
128KB
Destination
/service/config/incoming/
Started
11:20

Running

The 160 that never received the faulty file get the right configuration file from the start.

The 40 already verified need nothing again, so the run targets the remaining 3,960.

AT THIS MOMENTNow

11:20

Urgent distribution of the fix

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices
Corrected file delivery

40 / 4,000

Not yet delivered
3,960 devices
Back to normal
40 devices

NextCheck delivery per device

Next story, 12:00The fix reaches devices →

Urgent distribution, 12:00Scene 9 / 15

The reach of the faulty file is shrinking.

Most devices now have the corrected file.

Status at noon

Business recovers on the devices that have the file first.

Back to normal

3,400

Delivered, being applied

320

Not yet delivered

280

Results over time

  • Fix delivered
  • Back to normal

A fictional record: delivery results come from INNORIX, back-to-normal results from the company's service operations system.

The service team applies the corrected file and checks business status, starting with the devices that have it.

Devices with the file delivered and devices confirmed back to normal are tracked separately.

AT THIS MOMENTNow

12:00

Delivering and applying the fix

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices
Corrected file delivery

3,720 / 4,000

Not yet delivered
280 devices
Back to normal
3,400 devices

NextHandle the remaining devices

Next story, 13:10The last 48 devices →

Urgent distribution, 13:10Scene 10 / 15

The 48 left matter more than the thousands done.

3,952 of the 4,000 devices have the corrected file.

Time now

13:10

Not yet delivered

48 devices

Corrected file delivery

3,952 / 4,000

Delivery rate 98.80%

StatusDevicesResponse
Disconnected28 devicesResume after reconnecting
Storage full12 devicesFree space, then send
Powered off8 devicesPower on, wait to connect
Transferring0 devicesDelivery in progress

State of the 48 remaining devices at 13:10

48 devices are left, split into disconnected, storage full and powered off.

Back to normal

3,870 / 4,000 devices

48 devices that are disconnected, short of storage or powered off need further work.

The team does not add finished devices to the run again, and focuses on the unfinished ones.

AT THIS MOMENTNow

13:10

Remaining devices

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices
Corrected file delivery

3,952 / 4,000

Not yet delivered
48 devices
Back to normal
3,870 devices

NextRestore connections and devices

Next story, 14:00Each device its own way →

14:00Scene 11 / 15

Each problem on site is solved its own way.

The 48 devices did not all have the same problem.

Each one gets the response its situation needs.

Field response this afternoon

  1. 14:00

    28 devices back online

    Interrupted transfers resume on devices whose network is back.

    • Device-B-0248Completed
    • Device-B-1092Completed
    • Device-B-2840Completed

    Devices still waiting, 20

  2. 14:40

    12 devices short of storage

    Field team

    We found the storage problem and cleared unneeded temporary files to free the space.

    The corrected file is delivered to the devices with space freed.

    Devices still waiting, 8

  3. 15:05

    The last 8, powered off

    Field engineer

    We switched on the 8 devices that were not in use.

    The devices reconnect and the corrected file is on its way.

The company did not send field staff across the country to check every device.

Transfers that could recover on their own were left to do so, and field support went only where it was really needed.

AT THIS MOMENTNow

14:00

Remaining devices

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices
Corrected file delivery

3,952 / 4,000

Not yet delivered
48 devices
Back to normal
3,870 devices

NextRestore connections and devices

Next story, 15:20The last device →

Back to normal, 15:20Scene 12 / 15

Every target device has the right file.

The last devices' connection and storage problems are solved and the corrected file is delivered.

15:20, Final delivery result

The faulty file history and the fix result, side by side.

RunTargetsDoneFinal status
Release v4.24,0003,840Stopped
Correction pilot4040Completed
Correction v4.2.13,9603,960Completed

All corrected-file deliveries, both runs combined, per device

4,000 / 4,000

Fix delivered

First received the faulty file

3,840 / 3,840

Fix delivered

Earlier distribution incomplete

160 / 160

Right configuration delivered

Devices that received the wrong file and devices that had not received it yet now all hold the right file.

The service team does a final check of devices whose application result is not confirmed yet.

AT THIS MOMENTNow

15:20

Fix delivered

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices
Corrected file delivery

4,000 / 4,000

Not yet delivered
0 devices
Back to normal
3,976 devices

NextFinal field checks

Next story, 16:00One device's day →

16:00Scene 13 / 15

One device's day.

Pick one of the devices that received the faulty file.

Beyond the company-wide result, you can see which file reached a given device, and when.

File delivery history per device

Device: Device-B-0248

Device-B-0248

Capital area, Site 028

INNORIX transfer history

  • Night before 23:14

    configuration_v4.2.json

    Previous file delivered

  • 13:48

    configuration_v4.2.1.json

    Corrected file delivered

Company service records

  • 14:05

    Fix applied and business checked

    Normal

Network dropped after the faulty file arrived; the fix was delivered once the connection came back.

Some devices received the faulty file and then the fix; others received the right file from the start.

The totals can match while each device took its own path.

AT THIS MOMENTNow

16:00

Checking one device's record

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices
Corrected file delivery

4,000 / 4,000

Not yet delivered
0 devices
Back to normal
3,976 devices

NextFinal field checks

Next story, 17:10The incident closes →

Back to normal, 17:10

The company is back to normal operations.

The service team confirms the right configuration is applied on all 4,000 devices.

TUESDAY AFTERNOON

17:10

Normal operations are back.

Devices across the country are doing their work again.

4,000 devices

Fix delivered

4,000 devices

Back to normal

0 devices

Still open

The distribution team puts together the first distribution record, the stop record and the fix delivery results.

The company makes the corrected version the baseline for future scheduled distributions.

AT THIS MOMENTNow

17:10

Incident response closed

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices
Corrected file delivery

4,000 / 4,000

Not yet delivered
0 devices
Back to normal
4,000 devices

NextReview the records

18:00

The company records what happened today.

Some work remains after the response is over.

Why it happened, which file was delivered, which device got the fix and when, and what to improve if it happens again: all of it is written down.

INCIDENT REVIEW

Model B, Configuration v4.2

Record of the distribution error and the fix

Error found

10:12

Fix delivered

15:20

Response closed

17:10

ItemResult
Faulty fileconfiguration_v4.2.json
Original targets4,000
Received the faulty file3,840
Earlier distribution incomplete160
Corrected fileconfiguration_v4.2.1.json
Verified first40
Fix distributed afterwards3,960
Fix delivered4,000
Back to normal4,000
Business impactReviewed in service records

Records to keep

  • First run and its stop
  • Faulty file delivery per device
  • Fix verification and nationwide results
  • Service application and business checks

Today's record can be used in the next distribution.

The faulty version is excluded from future scheduled distributions, and the corrected version becomes the new baseline.

Afterwards, one week later

The company changed how it prepares the next distribution.

The team reviewed the distribution history and the response from this incident.

File versioning, field verification targets and the approval for nationwide distribution are set out more clearly.

ONE WEEK LATER

The next distribution is prepared a little differently.

Before

  1. Prepare the new files

    Checked in development

  2. Nationwide scheduled distribution

    All Model B devices

  3. Check the results

    Delivery confirmed per device

Now

  1. Prepare the new version

    Files and target devices separated

  2. Field targets first

    Verified in different environments

  3. Review and approve

    The company's own procedure

  4. Nationwide scheduled distribution

    Approved files and device groups

  5. Check each device

    Transfer and application records kept

Distribution jobs from now on

FlowDevicesRuns
Model B pilot distribution40On verification
Model B nationwide release4,000After verification and approval
Daily business data8,000Daily
Regional contentBy regionExisting schedule

AT THIS MOMENTNow

One week later

Procedure improved

All devices in service

8,000 devices

Scope of this incident
All response targets
4,000 devices
Received the faulty file
3,840 devices
Faulty file incomplete
160 devices
Corrected file delivery

4,000 / 4,000

Not yet delivered
0 devices
Back to normal
4,000 devices

NextNext scheduled distribution

The next scheduled distribution still uses the existing device groups, datasets and run management.

WHEN THINGS DON'T GO AS PLANNED

Not every day goes to plan.

A problem was found in a file already on thousands of devices.

The company confirmed how far the file had gone, and sent a corrected file to the devices that needed it.

10:25, Scope confirmed

3,840 devices

Received the faulty file

15:20, Fix delivered

4,000 devices

Right file delivered

17:10, Back to normal

4,000 devices

Business confirmed normal

It did not check 8,000 devices one by one.

It did not send files again to thousands of devices that had no problem.

It did not lose the record of the distribution already done.

Even after a file is delivered, the company can see where it went.

INNORIX

The technology behind this story

Fleet-Scale Distribution

INNORIX delivers the files thousands of devices need, and keeps track of everything from files already delivered to the results of the next distribution.

View the product