Scene preview, 10:12, FIRST INCIDENTOne device stopped.
The device came back online, but the engineering team still had a problem to investigate.
INNORIX in Practice
Day 1, The first incident10:12
THE FIRST INCIDENT, Day 1, 10:12
A single device stops unexpectedly.
Sites across the country were running business as usual.
At 10:12 a.m., a device in operation at one site suddenly stopped.
The on-site operator checked the device and restarted it. It went back to work normally.
But when the report reached the engineering team at headquarters, they felt there was not yet enough information to write it off as a temporary glitch.
First, they needed to know what happened just before the device stopped.
DAY 1, 10:12
A device running as usual suddenly stopped.
Example of the environment of the affected industrial device
Device-0842
Site
Site-024
Deployed
June 2023
Operating system
Embedded Linux
Software
v4.8.2
Incident
10:12:04
Restarted
10:16:32
Current status: Normal
DEVICE ACTIVITY
09:5010:1210:1610:25
Example reconstruction of device activity before and after the shutdown
The device is running again.But the company cannot yet say the problem is over.
Live contextNow
10:12Day 1
Investigation stage
First incident found
Affected devices
1
WHAT WE KNOW
Unexpected shutdown on one device
WHAT WE NEED TO KNOW
Why did the device stop?
Day 1, 10:25
The device is running again, but the problem is not solved.
The engineering team began securing operation logs, error records and software configuration from the affected device.
They needed records from before the problem and from after the restart.
Because the device was already connected to INNORIX, they could collect what they needed without anyone on site having to find, compress and send the files to headquarters.
DAY 1, 10:25
Securing records from the affected device.
Device-0842
Source files
Quick collection
File
Size
Status
runtime.log
184 MB
Collected
system_error.log
42 MB
Collected
device_config.json
128 KB
Collected
software_version.json
12 KB
Collected
diagnostic_snapshot.zip
860 MB
Collected
Destination
Investigation storage / Device-0842
Total collected
1.09 GB
ENGINEERING REVIEW
Anomalies found in the records just before the shutdown.
10:11:42Resource usage increased
10:11:53Process response delayed
10:12:04PROCESS_TERMINATED
10:12:08Service restart requested
Example of the company's engineering team reviewing the collected logs in an external analysis environment.
Do these records exist only on this device?
Live contextNow
10:25Day 1
Investigation stage
First incident found
Affected devices
1
Records secured
1.09 GB
WHAT WE KNOW
Anomalies found in the records just before the shutdown
WHAT WE NEED TO KNOW
Do these records exist only on this device?
THE INVESTIGATION EXPANDS, Day 1, 10:45
960 devices run the same software version.
The engineering team first identified the devices running the same software version as the affected device.
960 devices on the same version were in operation at 64 sites nationwide.
If the problem stems from a specific software version, similar records may turn up on other devices.
But devices on the same version were not all running in the same environment.
DAY 1, 10:45
Devices nationwide running the same software as the affected device
Affected device
1
Devices on the same version
960
Sites on the same version
64
SOFTWARE VERSIONS ACROSS THE FLEET
v4.8.2960
v4.8.1640
v4.7.3580
v4.7.2420
v4.6.1300
v4.5.0180
v4.4.8120
Operating environment of same-version devices
Operating system
Devices
Embedded Linux
480
Windows IoT
260
Ubuntu
140
Android
80
Software versions are classified from the company's device management data or previously collected version files.
Live contextNow
10:45Day 1
Investigation stage
Investigation expands
Devices on the same version
960
Sites running the same version
64
WHAT WE KNOW
960 devices are running the same version
WHAT WE NEED TO KNOW
Why did the device stop?
Day 1, 11:00
The team decides to investigate every device nationwide.
Securing data only from devices on one version makes it hard to tell whether that version is the cause.
The team needs to compare whether the same behavior appears on devices running other versions.
Beyond the software version, the device's operating system, age, workload and environmental conditions may also have played a part.
The engineering team decided to expand the investigation to all 3,200 devices nationwide.
DAY 1, 11:00
The investigation now covers every device nationwide.
First found
1
Same-version check
960
Fleet-wide comparison
3,200
960 on the same version
2,240 for comparison
Each cell in the graphic represents 4 devices.
INVESTIGATION REQUEST
Fleet-wide anomaly comparison
Purpose: check whether the same error type occurs and how it correlates with each operating environment
Scope
3,200 devices at 86 sites nationwide
Period
Operation records from the last 7 days
Priority data
Operation logs, error records, configuration
Destination
Dedicated engineering investigation storage
A problem that started on one devicehas grown into a review of the company's entire operating environment.
Live contextNow
11:00Day 1
Investigation stage
Investigation expands
Devices in scope
3,200
WHAT WE KNOW
960 devices are running the same version
WHAT WE NEED TO KNOW
Did the same problem occur on other versions?
Day 1, 11:20
Not every file on every device is needed.
The 3,200 devices nationwide held a substantial amount of data.
Each device had months of operation records, business data, inspection results and system files.
But this investigation did not need all of it.
The engineering team decided to secure only the last 7 days of operation logs and error records, plus each device's current configuration.
Devices that turn out to need more detail will get an additional collection later, based on the findings.
DAY 1, 11:20
Specifying the files and time range the investigation needs.
ALL STORED DATA
186 TB
All data stored on devices nationwide
/business/Business data
/inspection/Inspection results
/logs/Operation records
/diagnostics/Diagnostic files
/config/Configuration
/archive/Archived data
REQUIRED FOR INVESTIGATION
2.4 TB
Estimated data needed for this investigation
runtime_*.logLast 7 days
error_*.logLast 7 days
system_*.logLast 7 days
device_config.jsonCurrent config
software_version.jsonCurrent version
This collection versus all stored data
About 1.3% to collect186 TB in total
Collection configuration
Source devices
All investigation devices, 3,200
Source paths
/logs/, /config/
Include
*.log, device_config.json, software_version.json
File period
Last 7 days, applied to operation logs
Exclude
Archived logs and unrelated files
Destination
Investigation storage
Save rule
Investigation ID / Site ID / Device ID
The investigation covers every device nationwide,but the collection is limited to the data the investigation needs.
Live contextNow
11:20Day 1
Investigation stage
Investigation expands
Devices in scope
3,200
Expected data to collect
2.4 TB
WHAT WE KNOW
This collection covers about 1.3% of all stored data
WHAT WE NEED TO KNOW
Did the same problem occur on other versions?
Day 1, 12:30
Field operations cannot stop for the investigation.
Data collection from devices nationwide was under way.
Meanwhile, the devices kept doing their jobs at each site.
Some were processing production data, others were running inspections, and some were delivering field services.
The engineering team had to secure the files it needed while keeping the impact on field operations to a minimum.
Collection runs within bandwidth and storage capacity, and only files that have finished being written are collected.
DAY 1, 12:30
Field operations and the engineering investigation run side by side.
FIELD OPERATIONS
Day-to-day work at sites nationwide
Site-008ProductionProduction in progress
Site-024General siteField service in progress
Site-046InspectionQuality inspection in progress
Site-071LogisticsShipping and receiving in progress
INVESTIGATION COLLECTION
Engineering investigation data collection
2,180 / 3,200
Devices with required data secured
Collected
2,180
In progress
760
Queued
188
Awaiting reconnection
72
Runs
Fleet investigation
Source device
Size
Progress
Status
Device-0024
684 MB
Completed
Device-0168
426 MB
Running
Device-0842
812 MB
Completed
Device-1246
1.2 GB
Running
Device-2480
924 MB
Waiting
Device-3018
620 MB
Disconnected
The field carries on with its usual work,while headquarters secures the investigation data it needs.
Live contextNow
12:30Day 1
Investigation stage
Collecting records
Data secured
2,180 / 3,200
Awaiting reconnection
72
WHAT WE KNOW
Field operations and the engineering investigation run side by side
WHAT WE NEED TO KNOW
Did the same problem occur on other versions?
WHAT THE DATA REVEALS, Day 1, 14:20
Similar records turn up on 38 devices.
At 2:20 p.m., collection from the full fleet was not yet complete, but most of the investigation data had reached the company.
The engineering team began analyzing the operation records already secured.
They found similar anomalies on 38 devices, including the one where the problem first appeared.
26 of them were running the software version originally suspected.
But the other 12 were running different versions.
It was no longer easy to call this a problem with one specific version.
DAY 1, 14:20
Findings nobody expected at the start.
Data secured
3,060
Similar records found
38
Software versions affected
3
PRELIMINARY ENGINEERING ANALYSIS
Similar anomalies by software version
v4.8.226
v4.8.18
v4.7.34
v4.7.20
v4.6.10
v4.5.00
v4.4.80
Counts from the records secured so far, not the final result for the whole fleet.
The initial hypothesis and what we know now
Initial hypothesis
A problem specific to one software version
New finding
Similar records on other software versions too
Next steps
Compare device configuration, memory state, throughput, uptime and shared modules
As more data comes in,the scope of the problem starts to look different from what the company first thought.This analysis is performed by the company's engineering team and an external log analysis system.
Live contextNow
14:20Day 1
Investigation stage
Interim analysis
Similar records found
38 devices
WHAT WE KNOW
Similar records found across 3 software versions
WHAT WE NEED TO KNOW
What do these devices have in common?
THE INVESTIGATION CHANGES
From investigating more devicesto investigating specific devices more deeply.
The first investigation secured common data from every device nationwide. Now, based on its findings, a different kind of investigation begins: securing more detailed data from specific devices.
When new facts come to light, the data you need changes too.
INNORIX in Practice
Day 1, The first incident10:12
Day 1, 14:45
Now specific devices need more detailed records.
The engineering team selected the 38 devices with similar anomalies for further investigation.
They also selected 12 devices running normally on the same versions in similar environments as a reference group.
The first investigation secured only common operation records. This time they need the detailed diagnostics, resource usage records and advanced configuration stored on the devices.
The existing collection keeps running while a new collection is added.
DAY 1, 14:45
Building a new collection target around the investigation goal.
AFFECTED RECORDS
38 devices
Devices with similar anomalies
Device-0842v4.8.2
Device-0914v4.8.2
Device-1628v4.8.1
Device-2186v4.7.3
Devices selected by external analysis
REFERENCE DEVICES
12 devices
Healthy devices for comparison
Device-0856v4.8.2
Device-0932v4.8.2
Device-1644v4.8.1
Device-2208v4.7.3
Reference devices in similar environments
Detailed investigation dataset
Files to collect
Purpose
diagnostic_*.zip
Detailed device diagnostics
resource_usage_*.csv
Memory and CPU usage over time
process_history_*.log
Process execution history
module_versions.json
Internal software module versions
advanced_config.json
Advanced device settings
PARALLEL COLLECTION RUNS
Initial investigation3,200 devices, common operation records
An investigation that started with every device nationwidenow selects only the devices it needs and digs deeper.
Live contextNow
14:45Day 1
Investigation stage
Detailed investigation
Devices for detailed collection
5038 affected, 12 reference
WHAT WE KNOW
Diagnostics beyond the existing data are needed
WHAT WE NEED TO KNOW
Which operating conditions contributed to the problem?
Day 1, 17:30
Some devices have not been investigated yet.
By 5:30 p.m., the engineering team had secured a substantial amount of investigation data.
All the detailed records needed from the 50 devices in the second investigation were in.
But the first, fleet-wide investigation still lacked the required records from 72 devices.
Some devices connect to the network only during business hours, and others were cut off by network problems on site.
A device where no problem was found is not the same as a device that has not been checked yet.
The engineering team does not treat devices without data as healthy.
DAY 1, 17:30
72 devices nobody can draw conclusions about yet
Devices in scope
3,200
Data secured
3,128
No data yet
72
INVESTIGATION COVERAGE
Secured 97.75%Missing 2.25%
Reason
Devices
Follow-up
Disconnected after business hours
38
Collect on next connection
Site network outage
21
Await reconnection
Device powered off
13
Collect after restart
Remaining collection targets
Device
Site
Last seen
Status
Device-0186
Site-006
13:42
Offline
Device-0872
Site-024
14:16
Disconnected
Device-1642
Site-041
12:38
Offline
Device-2286
Site-058
15:22
Disconnected
Device-3018
Site-082
11:48
Offline
Having no problemand not having checked yet mean entirely different things.
Live contextNow
17:30Day 1
Investigation stage
Checking missing data
Devices without data
72
WHAT WE KNOW
Data secured from 97.75% of all devices
WHAT WE NEED TO KNOW
Do the remaining devices hold the same records?
BETWEEN DAY 1 AND DAY 2
The engineering investigation continues,and records from offline devices arrive one by one.
Day 1, 17:30
3,128
Devices with data secured
Day 2, 00:00
3,154
Devices with data secured
Day 2, 08:40
3,194
Devices with data secured
RECOVERY AND COLLECTION ACTIVITY
18:42:16Device-0872Connection restored
18:48:28Device-0872Collection completed
21:06:04Device-2286Transfer resumed
00:12:36Device-1642Collection completed
06:48:21Device-0186Device connected
07:26:04Device-3018Collection completed
INNORIX in Practice
Day 1, The first incident10:12
Day 2, 08:40
Records arrive from devices that came back online.
The next morning, the engineering team checked on the devices it could not reach the day before.
Additional data had been secured from 66 of the 72 devices.
Some had their network connection restored overnight, while others reconnected as the morning shift began.
Only 6 devices are now without data.
The team checked those sites and their current connection status, and asked on-site staff to check the devices where needed.
DAY 2, 08:40
Devices not yet investigated drop from 72 to 6.
7246186
17:3000:0006:0008:40
Devices without data over time
Devices that still need checking
Device
Site
Remaining status
Device-0284
Site-009
Offline
Device-0628
Site-018
Offline
Device-1126
Site-031
Disconnected
Device-1862
Site-048
Offline
Device-2640
Site-069
Disconnected
Device-3108
Site-084
Offline
Without re-requesting the thousands of records it already has,the company focuses on the devices it has not reached yet.
Live contextNow
08:40Day 2
Investigation stage
Closing the investigation
Devices without data
6
WHAT WE KNOW
Additional data secured from 66 of 72 devices
WHAT WE NEED TO KNOW
Do the remaining devices hold the same records?
Day 2, 11:30
Investigation data from every device nationwide is secured.
At 11:30 a.m., as the site connection came back, data from the very last device reached the company.
Common investigation data from all 3,200 devices and detailed diagnostics from 50 devices were now fully secured.
Everything collected for the investigation was stored in a structure that keeps each site and device distinct.
The engineering team could see which file came from which device and find the original records again whenever the investigation needed them.
DAY 2, 11:30
Investigation data from devices nationwide is complete.
Initial investigation
3,200 / 3,200
Detailed investigation
50 / 50
Total data collected
3.02 TB
Collection results
Collection
Devices
Data
Status
Initial investigation
3,200
2.40 TB
Completed
Affected device diagnostics
38
480 GB
Completed
Reference device diagnostics
12
140 GB
Completed
INVESTIGATION STORAGE
INV-2026-014
Initial-investigation/
Site-024/
Device-0842/
Device-0856/
Device-0872/
Site-025/
Detailed-investigation/
Device-0842
runtime_20261109.log
runtime_20261110.log
system_error.log
device_config.json
software_version.json
Source device and collection record available
Checking the collection record of an original file
Source
Device-0842, Site-024
File
system_error.log
Investigation
INV-2026-014
Collected
Day 1, 11:24:18
Transfer result
Completed
Files gathered from thousands of devicesbecome evidence for investigating each one.
Live contextNow
11:30Day 2
Investigation stage
Closing the investigation
Data secured
3,200 / 3,200
Detailed diagnostics
50 / 50
Total data collected
3.02 TB
WHAT WE KNOW
Common investigation data secured from all 3,200 devices nationwide
WHAT WE NEED TO KNOW
Which operating conditions contributed to the problem?
FROM EVIDENCE TO DECISIONS, Day 3, 10:00
The company pins down the conditions behind the problem.
The engineering team compared the operation records secured from all 3,200 devices nationwide.
Similar anomalies also turned up in the records from the last devices to report in.
In the end, related records were confirmed on 41 devices in total.
Comparing the detailed diagnostics, the team concluded the problem was not confined to one software version: it most likely stemmed from an internal module shared across several versions, under specific operating conditions.
Based on the findings, the team separated the devices needing an immediate response from those needing further verification.
DAY 3, 10:00
From the first device to related records on 41 devices nationwide.
ENGINEERING INVESTIGATION REPORT
Fleet-wide comparison
3,200
Related records found
41
No matching records
3,159
The graphic is a schematic of findings across devices nationwide.
Comparing records from affected devices
Software version
Interim
Final
v4.8.2
26
28
v4.8.1
8
9
v4.7.3
4
4
Total
38
41
ENGINEERING FINDINGS
A module shared across versions, under specific operating conditions
Shared software module
Same module confirmed on related devices
Long continuous operation
Common condition among devices in the detailed investigation
Specific error records
Repeated response delays in the related process
Follow-up verification
Reproduction tests and fix verification required
A fictional analysis by the company's engineering team. INNORIX does not diagnose the cause of device errors itself.
The company no longer needs to apply the same fix to every device nationwide.
RESPONSE PLAN
Response targets are set by the findings.
27 immediate response14 further verification
27Immediate responseDevices meeting conditions with a high risk of shutdown
14Further verificationDevices with similar records that need further diagnostics
3,159Business as usualDevices with no matching anomaly in the records for the period investigated
Instead of guessing,the company can now respond based on records secured nationwide.
Live contextNow
10:00Day 3
Investigation stage
Final findings
Related records confirmed
41 devices
WHAT WE KNOW
Data from all 3,200 devices secured and compared
NEXT ACTION
Respond to 27 devices first, verify 14 more
ONE MONTH LATER, 1 month later
The company already holds the records from before a problem occurs.
With the investigation complete, the engineering team reviewed how its devices were being operated.
If records are secured only after a problem occurs, older files on the device may already have been deleted or overwritten.
The device may not even be connected to the network when the problem happens.
The company decided to collect and retain key operation records and error files from its devices on a regular schedule.
Detailed diagnostic files are now pulled only when a problem occurs or a further investigation needs them.
There is no need to keep collecting every file from every device.
From a company that looks for data after a problem,to one that secures the records it needs in advance.
BEFORE
Records secured after the problem
Records kept on site
Incident
When a problem occurs, the team starts by securing whatever records remain on that device.
AFTER
Operation records secured continuously
Records held by the company
Incident
The company can check records from before the problem and collect more data if needed.
The company’s new data collection workflows
Collection flow
Target
Trigger
Daily operation records
All 3,200 devices
Daily
Error records
All 3,200 devices
When an error file is created
Device configuration
All 3,200 devices
Weekly
Detailed diagnostics
Only the devices needed
On investigation request
INNORIX in Practice
Day 1, The first incident10:12
THE NEXT INCIDENT
Then one day, an anomaly appears on a different device.
It was a different site and a different device from a month earlier.
This time, the engineering team did not start by requesting every record from the device again.
They checked the recent operation records already secured in company storage.
They also compared them with records from problems that had occurred on other devices in the past.
Then they collected only the latest detailed diagnostic file they did not yet have.
THE NEXT INCIDENT
When a new problem occurs, the records are already there to check.
Device-2418
File history
Collected
File
Source
7 days ago
runtime_01.log
Device-2418
5 days ago
runtime_03.log
Device-2418
3 days ago
runtime_05.log
Device-2418
Yesterday
runtime_07.log
Device-2418
Today 09:20
system_error.log
Device-2418
ADDITIONAL COLLECTION REQUEST
Collecting
diagnostic_snapshot.zip
Detailed diagnostics from the device right now
Every time a new problem occurs,the company no longer has to start its investigation from scratch.
FROM A SINGLE INCIDENT TO FLEET-WIDE ANSWERS
The problem was found on one device,but the answer lay in the records left on devices nationwide.
THE INVESTIGATION
First incident found1 device
Fleet-wide investigation3,200 devices
Detailed diagnostics50 devices
Immediate response27 devices
The more devices a company runs,the more it matters to know what a problem found in one place means for the entire operation.
Wherever a problem occurs,a company should be able to secure the records it needs and move on to what comes next.
THE TECHNOLOGY BEHIND THIS STORY
Fleet-Scale Collection
Collect the files you need from devices and industrial systems running across the country. Gather common data from the entire fleet, or select devices that meet specific conditions to secure the detailed data you need.
From scheduled data collection to urgent collection when the unexpected happens, it supports the full range of your data operations.