The incident, Night before, 22:00Scene 1 / 15
Last night, new files went out as usual.
The company runs 8,000 devices across 320 sites nationwide.
Last night the scheduled distribution of a new program and configuration file began for the 4,000 Model B devices.
Scheduled distribution starts
Model A
2,400 devices
Service as usual
Model B
4,000 devices
In this scheduled distribution
Model C
1,600 devices
Service as usual
Files in this scheduled distribution
| File | Size | Version |
|---|
| application_v4.2.zip | 3.6GB | 4.2 |
| configuration_v4.2.json | 128KB | 4.2 |
| service_resources.zip | 24GB | Existing resource kept |
The 24GB shared resources were already delivered by an earlier distribution.
Files were reaching devices one after another, as each device's schedule and connection allowed.
AT THIS MOMENTNow
Night before 22:00
Scheduled distribution running
All devices in service8,000 devices
NextDevices start work
Next story, 10:12An urgent message →The incident, 10:12Scene 2 / 15
An error is found in a file already distributed.
At 10:12 in the morning, the development team finds an error in a configuration file.
Development team10:12
We found an error in the Model B configuration file distributed yesterday.
Some business conditions may not be processed correctly. Please stop distributing this file and check which devices already have it.
configuration_v4.2.json
128KB, Version 4.2
Error confirmed
Distribution team10:14
We're checking the distribution that includes this file. We'll stop any further runs and check the delivery result for each device.
All devices in service
8,000 devices
Targets of this file
4,000 devices
At this point nobody yet knows how many devices received the file, or how many were affected.
On some devices that applied the file, certain business tasks may not be processed correctly.
The team confirms the exact version of the faulty file and the distribution that carried it.
AT THIS MOMENTNow
10:12
Investigating the scope
All devices in service8,000 devices
Scope of this incident- All response targets
- 4,000 devices
- Received the faulty file
- 3,840 devices
- Faulty file incomplete
- 160 devices
NextStop the distribution
Next story, 10:18Stopping the distribution →Confirming the scope, 10:18Scene 3 / 15
The faulty file must go no further.
The existing Model B distribution and its repeating runs are stopped.
Model B release 4.2
Started yesterday, 22:00
Stopped
- Current run
- Stopped 10:18
- Scheduled runs
- Disabled
Stopping the run and disabling the schedule are confirmed separately. Stopping does not recall files already delivered.
Devices that already received the file do not go back to their previous state because the distribution stops.
The team checks the result of the stop and the last transfer records.
AT THIS MOMENTNow
10:18
Distribution under control
All devices in service8,000 devices
Scope of this incident- All response targets
- 4,000 devices
- Received the faulty file
- 3,840 devices
- Faulty file incomplete
- 160 devices
NextSearch the delivery history
Next story, 10:25Devices that got the file →Confirming the scope, 10:25Scene 4 / 15
There was no need to check all 8,000 devices.
The team checks each device's transfer result by the faulty file and the run that delivered it.
Of the 4,000 Model B devices, 3,840 completed the distribution. The other 160 have not received the complete file yet.
All devices in operation, 8,000
- Received the faulty file3,840
- Earlier distribution incomplete160
- Not affected4,000
File delivery history
Search
configuration_v4.2.json
Run, Model B release 4.2
Since 22:00 yesterday
3,840 devices
Delivery records for this file
| Device | Location | Delivered |
|---|
| Device-B-0012 | Capital area | 22:14 |
| Device-B-0258 | Central | 23:32 |
| Device-B-1092 | South | 02:18 |
| Device-B-2840 | Other regions | 07:25 |
| Device-B-3812 | Capital area | 10:05 |
3,840 completed deliveries, sample of the transfer history
Whether the program is actually applied is checked in a separate service operations system.
AT THIS MOMENTNow
All devices in service8,000 devices
Scope of this incident- All response targets
- 4,000 devices
- Received the faulty file
- 3,840 devices
- Faulty file incomplete
- 160 devices
NextPrepare the fix
Next story, 10:35One standard nationwide →10:35Scene 5 / 15
One standard for the whole country.
The devices that received the file are spread across the country.
Instead of collecting different results region by region, the team shares the confirmed device list with the field teams.
| Region | All devices | Model B | Faulty file | Incomplete |
|---|
| Capital area | 3,000 | 1,500 | 1,450 | 50 |
| Central | 1,800 | 900 | 865 | 35 |
| South | 2,400 | 1,200 | 1,150 | 50 |
| Other regions | 800 | 400 | 375 | 25 |
| Total | 8,000 | 4,000 | 3,840 | 160 |
Distribution team, 10:35
We have the full list of devices that received the faulty file. Sharing the targets by region.
Service team, 10:38
We'll check whether those devices applied the file and what it affected. Let us know when the fix is ready.
The development team now knows which file to fix, and the distribution team knows which devices to send it to.
Field teams can work on the devices that need attention, not on every site.
AT THIS MOMENTNow
All devices in service8,000 devices
Scope of this incident- All response targets
- 4,000 devices
- Received the faulty file
- 3,840 devices
- Faulty file incomplete
- 160 devices
NextPrepare the fix
Next story, 10:45Finding the cause →Preparing the fix, 10:45Scene 6 / 15
The problem was a 128KB configuration file.
The development team confirms the cause is a single configuration file, not the program as a whole.
The 3.6GB program and the 24GB shared resources do not need to change.
Preparing the fix
There was no need to send the 3.6GB program again.
Shared service resources
service_resources.zip
24GB
Existing file kept
Program file
application_v4.2.zip
3.6GB
Configuration file
configuration_v4.2.1.json
128KB
A rough picture of the difference in file sizes.
Previous file
configuration_v4.2.json
Version 4.2
Error confirmed, no redistribution
Corrected file
configuration_v4.2.1.json
Version 4.2.1
Field verification next
The development team prepares the corrected configuration file.
AT THIS MOMENTNow
All devices in service8,000 devices
Scope of this incident- All response targets
- 4,000 devices
- Received the faulty file
- 3,840 devices
- Faulty file incomplete
- 160 devices
NextVerify on real devices
Next story, 11:00Verifying the fix on site →Preparing the fix, 11:00Scene 7 / 15
This time, check before going nationwide.
From the devices that received the faulty file, 40 at different sites are selected.
After the corrected file is delivered, field staff verify it in the real business environment.
All targets of the fix
4,000 devices
| Task | Team | Result |
|---|
| Deliver the fix | Distribution team | 40 done |
| Apply the fix | Service team | 40 done |
| Recheck the affected task | Field team | Normal |
Service team, 11:15
The 40 devices with the corrected file are processing work normally. Please go ahead nationwide.
Once verification is done, the company decides to deliver the fix to every Model B device in the country.
AT THIS MOMENTNow
All devices in service8,000 devices
Scope of this incident- All response targets
- 4,000 devices
- Received the faulty file
- 3,840 devices
- Faulty file incomplete
- 160 devices
Corrected file delivery40 / 4,000
- Not yet delivered
- 3,960 devices
- Back to normal
- 40 devices
NextApprove urgent nationwide distribution
Next story, 11:20Distributing the fix →