INNORIX in Practice, Fleet-Scale Collection

A problem on one deviceleads to an investigation of devices nationwide.

An investigation that starts with a single anomaly.The company needs to know how many devices have the same problem.

Scenario
A fictional business scenario about an urgent technical investigation of devices operating nationwide.

Scope expandsDevices in scope

MONDAY, 10:12

Device-0842

Softwarev4.8.2

Unexpected device shutdown

Records found on site

10:12:04 PROCESS_TERMINATED

10:12:08 SERVICE_RESTART

INVESTIGATION SCOPE

From 1 device to 3,200

Devices running at 86 sites nationwide

Sites
86
Software versions
7
Devices in scope
3,200

Is the same problem happening on other devices?

  1. Incident
  2. Scope expands
  3. Collecting records
  4. Root-cause analysis

The full scope of the investigation and the questions still unanswered appear.

INNORIX in Practice

Day 1, The first incident10:12

THE FIRST INCIDENT, Day 1, 10:12

A single device stops unexpectedly.

Sites across the country were running business as usual.

At 10:12 a.m., a device in operation at one site suddenly stopped.

The on-site operator checked the device and restarted it. It went back to work normally.

But when the report reached the engineering team at headquarters, they felt there was not yet enough information to write it off as a temporary glitch.

First, they needed to know what happened just before the device stopped.

DAY 1, 10:12

A device running as usual suddenly stopped.

An industrial computer mounted on a DIN rail inside an electrical cabinet, with network cables connected
Example of the environment of the affected industrial device

Device-0842

Site
Site-024
Deployed
June 2023
Operating system
Embedded Linux
Software
v4.8.2
Incident
10:12:04
Restarted
10:16:32

Current status: Normal

DEVICE ACTIVITY

Example reconstruction of device activity before and after the shutdown

The device is running again.But the company cannot yet say the problem is over.

Live contextNow

10:12Day 1

Investigation stage
First incident found
Affected devices
1

WHAT WE KNOW

Unexpected shutdown on one device

WHAT WE NEED TO KNOW

Why did the device stop?

Day 1, 10:25

The device is running again, but the problem is not solved.

The engineering team began securing operation logs, error records and software configuration from the affected device.

They needed records from before the problem and from after the restart.

Because the device was already connected to INNORIX, they could collect what they needed without anyone on site having to find, compress and send the files to headquarters.

DAY 1, 10:25

Securing records from the affected device.

Device-0842

Source files

Quick collection
FileSizeStatus
runtime.log184 MBCollected
system_error.log42 MBCollected
device_config.json128 KBCollected
software_version.json12 KBCollected
diagnostic_snapshot.zip860 MBCollected
Destination
Investigation storage / Device-0842
Total collected
1.09 GB

ENGINEERING REVIEW

Anomalies found in the records just before the shutdown.

  1. 10:11:42Resource usage increased
  2. 10:11:53Process response delayed
  3. 10:12:04PROCESS_TERMINATED
  4. 10:12:08Service restart requested

Example of the company's engineering team reviewing the collected logs in an external analysis environment.

Do these records exist only on this device?

Live contextNow

10:25Day 1

Investigation stage
First incident found
Affected devices
1
Records secured
1.09 GB

WHAT WE KNOW

Anomalies found in the records just before the shutdown

WHAT WE NEED TO KNOW

Do these records exist only on this device?

THE INVESTIGATION EXPANDS, Day 1, 10:45

960 devices run the same software version.

The engineering team first identified the devices running the same software version as the affected device.

960 devices on the same version were in operation at 64 sites nationwide.

If the problem stems from a specific software version, similar records may turn up on other devices.

But devices on the same version were not all running in the same environment.

DAY 1, 10:45

Devices nationwide running the same software as the affected device

Affected device
1
Devices on the same version
960
Sites on the same version
64

SOFTWARE VERSIONS ACROSS THE FLEET

Operating environment of same-version devices

Operating systemDevices
Embedded Linux480
Windows IoT260
Ubuntu140
Android80

Software versions are classified from the company's device management data or previously collected version files.

Live contextNow

10:45Day 1

Investigation stage
Investigation expands
Devices on the same version
960
Sites running the same version
64

WHAT WE KNOW

960 devices are running the same version

WHAT WE NEED TO KNOW

Why did the device stop?

Day 1, 11:00

The team decides to investigate every device nationwide.

Securing data only from devices on one version makes it hard to tell whether that version is the cause.

The team needs to compare whether the same behavior appears on devices running other versions.

Beyond the software version, the device's operating system, age, workload and environmental conditions may also have played a part.

The engineering team decided to expand the investigation to all 3,200 devices nationwide.

DAY 1, 11:00

The investigation now covers every device nationwide.

First found
1
Same-version check
960
Fleet-wide comparison
3,200
  • 960 on the same version
  • 2,240 for comparison

Each cell in the graphic represents 4 devices.

INVESTIGATION REQUEST

Fleet-wide anomaly comparison

Purpose: check whether the same error type occurs and how it correlates with each operating environment

Scope
3,200 devices at 86 sites nationwide
Period
Operation records from the last 7 days
Priority data
Operation logs, error records, configuration
Destination
Dedicated engineering investigation storage
A problem that started on one devicehas grown into a review of the company's entire operating environment.

Live contextNow

11:00Day 1

Investigation stage
Investigation expands
Devices in scope
3,200

WHAT WE KNOW

960 devices are running the same version

WHAT WE NEED TO KNOW

Did the same problem occur on other versions?

Day 1, 11:20

Not every file on every device is needed.

The 3,200 devices nationwide held a substantial amount of data.

Each device had months of operation records, business data, inspection results and system files.

But this investigation did not need all of it.

The engineering team decided to secure only the last 7 days of operation logs and error records, plus each device's current configuration.

Devices that turn out to need more detail will get an additional collection later, based on the findings.

DAY 1, 11:20

Specifying the files and time range the investigation needs.

ALL STORED DATA

186 TB

All data stored on devices nationwide

  • /business/Business data
  • /inspection/Inspection results
  • /logs/Operation records
  • /diagnostics/Diagnostic files
  • /config/Configuration
  • /archive/Archived data

REQUIRED FOR INVESTIGATION

2.4 TB

Estimated data needed for this investigation

  • runtime_*.logLast 7 days
  • error_*.logLast 7 days
  • system_*.logLast 7 days
  • device_config.jsonCurrent config
  • software_version.jsonCurrent version

This collection versus all stored data

About 1.3% to collect186 TB in total

Collection configuration

Source devices
All investigation devices, 3,200
Source paths
/logs/, /config/
Include
*.log, device_config.json, software_version.json
File period
Last 7 days, applied to operation logs
Exclude
Archived logs and unrelated files
Destination
Investigation storage
Save rule
Investigation ID / Site ID / Device ID
The investigation covers every device nationwide,but the collection is limited to the data the investigation needs.

Live contextNow

11:20Day 1

Investigation stage
Investigation expands
Devices in scope
3,200
Expected data to collect
2.4 TB

WHAT WE KNOW

This collection covers about 1.3% of all stored data

WHAT WE NEED TO KNOW

Did the same problem occur on other versions?

Day 1, 12:30

Field operations cannot stop for the investigation.

Data collection from devices nationwide was under way.

Meanwhile, the devices kept doing their jobs at each site.

Some were processing production data, others were running inspections, and some were delivering field services.

The engineering team had to secure the files it needed while keeping the impact on field operations to a minimum.

Collection runs within bandwidth and storage capacity, and only files that have finished being written are collected.

DAY 1, 12:30

Field operations and the engineering investigation run side by side.

FIELD OPERATIONS

Day-to-day work at sites nationwide

  • Site-008ProductionProduction in progress
  • Site-024General siteField service in progress
  • Site-046InspectionQuality inspection in progress
  • Site-071LogisticsShipping and receiving in progress

INVESTIGATION COLLECTION

Engineering investigation data collection

2,180 / 3,200

Devices with required data secured

Collected
2,180
In progress
760
Queued
188
Awaiting reconnection
72

Runs

Fleet investigation
Source deviceSizeProgressStatus
Device-0024684 MBCompleted
Device-0168426 MBRunning
Device-0842812 MBCompleted
Device-12461.2 GBRunning
Device-2480924 MBWaiting
Device-3018620 MBDisconnected
The field carries on with its usual work,while headquarters secures the investigation data it needs.

Live contextNow

12:30Day 1

Investigation stage
Collecting records
Data secured
2,180 / 3,200
Awaiting reconnection
72

WHAT WE KNOW

Field operations and the engineering investigation run side by side

WHAT WE NEED TO KNOW

Did the same problem occur on other versions?

WHAT THE DATA REVEALS, Day 1, 14:20

Similar records turn up on 38 devices.

At 2:20 p.m., collection from the full fleet was not yet complete, but most of the investigation data had reached the company.

The engineering team began analyzing the operation records already secured.

They found similar anomalies on 38 devices, including the one where the problem first appeared.

26 of them were running the software version originally suspected.

But the other 12 were running different versions.

It was no longer easy to call this a problem with one specific version.

DAY 1, 14:20

Findings nobody expected at the start.

Data secured
3,060
Similar records found
38
Software versions affected
3

PRELIMINARY ENGINEERING ANALYSIS

Similar anomalies by software version

Counts from the records secured so far, not the final result for the whole fleet.

The initial hypothesis and what we know now

Initial hypothesis
A problem specific to one software version
New finding
Similar records on other software versions too
Next steps
Compare device configuration, memory state, throughput, uptime and shared modules
As more data comes in,the scope of the problem starts to look different from what the company first thought.This analysis is performed by the company's engineering team and an external log analysis system.

Live contextNow

14:20Day 1

Investigation stage
Interim analysis
Similar records found
38 devices

WHAT WE KNOW

Similar records found across 3 software versions

WHAT WE NEED TO KNOW

What do these devices have in common?

THE INVESTIGATION CHANGES

From investigating more devicesto investigating specific devices more deeply.

The first investigation secured common data from every device nationwide. Now, based on its findings, a different kind of investigation begins: securing more detailed data from specific devices.

INITIAL INVESTIGATION

3,200 devices

Common operation records from devices nationwide

Operation logs, error records, basic configuration

DETAILED INVESTIGATION

50 devices

Devices with anomalies and healthy reference devices

Detailed diagnostics, resource status, advanced settings

When new facts come to light, the data you need changes too.

INNORIX in Practice

Day 1, The first incident10:12

Day 1, 14:45

Now specific devices need more detailed records.

The engineering team selected the 38 devices with similar anomalies for further investigation.

They also selected 12 devices running normally on the same versions in similar environments as a reference group.

The first investigation secured only common operation records. This time they need the detailed diagnostics, resource usage records and advanced configuration stored on the devices.

The existing collection keeps running while a new collection is added.

DAY 1, 14:45

Building a new collection target around the investigation goal.

AFFECTED RECORDS

38 devices

Devices with similar anomalies

  • Device-0842v4.8.2
  • Device-0914v4.8.2
  • Device-1628v4.8.1
  • Device-2186v4.7.3

Devices selected by external analysis

REFERENCE DEVICES

12 devices

Healthy devices for comparison

  • Device-0856v4.8.2
  • Device-0932v4.8.2
  • Device-1644v4.8.1
  • Device-2208v4.7.3

Reference devices in similar environments

Detailed investigation dataset

Files to collectPurpose
diagnostic_*.zipDetailed device diagnostics
resource_usage_*.csvMemory and CPU usage over time
process_history_*.logProcess execution history
module_versions.jsonInternal software module versions
advanced_config.jsonAdvanced device settings

PARALLEL COLLECTION RUNS

  • Initial investigation3,200 devices, common operation records
    Running
  • Detailed investigation50 devices, detailed diagnostics
    Started
An investigation that started with every device nationwidenow selects only the devices it needs and digs deeper.

Live contextNow

14:45Day 1

Investigation stage
Detailed investigation
Devices for detailed collection
5038 affected, 12 reference

WHAT WE KNOW

Diagnostics beyond the existing data are needed

WHAT WE NEED TO KNOW

Which operating conditions contributed to the problem?

Day 1, 17:30

Some devices have not been investigated yet.

By 5:30 p.m., the engineering team had secured a substantial amount of investigation data.

All the detailed records needed from the 50 devices in the second investigation were in.

But the first, fleet-wide investigation still lacked the required records from 72 devices.

Some devices connect to the network only during business hours, and others were cut off by network problems on site.

A device where no problem was found is not the same as a device that has not been checked yet.

The engineering team does not treat devices without data as healthy.

DAY 1, 17:30

72 devices nobody can draw conclusions about yet

Devices in scope
3,200
Data secured
3,128
No data yet
72

INVESTIGATION COVERAGE

Secured 97.75%Missing 2.25%

ReasonDevicesFollow-up
Disconnected after business hours38Collect on next connection
Site network outage21Await reconnection
Device powered off13Collect after restart

Remaining collection targets

DeviceSiteLast seenStatus
Device-0186Site-00613:42Offline
Device-0872Site-02414:16Disconnected
Device-1642Site-04112:38Offline
Device-2286Site-05815:22Disconnected
Device-3018Site-08211:48Offline
Having no problemand not having checked yet mean entirely different things.

Live contextNow

17:30Day 1

Investigation stage
Checking missing data
Devices without data
72

WHAT WE KNOW

Data secured from 97.75% of all devices

WHAT WE NEED TO KNOW

Do the remaining devices hold the same records?

BETWEEN DAY 1 AND DAY 2

The engineering investigation continues,and records from offline devices arrive one by one.

Day 1, 17:30
3,128
Devices with data secured
Day 2, 00:00
3,154
Devices with data secured
Day 2, 08:40
3,194
Devices with data secured

RECOVERY AND COLLECTION ACTIVITY

  1. 18:42:16Device-0872Connection restored
  2. 18:48:28Device-0872Collection completed
  3. 21:06:04Device-2286Transfer resumed
  4. 00:12:36Device-1642Collection completed
  5. 06:48:21Device-0186Device connected
  6. 07:26:04Device-3018Collection completed

INNORIX in Practice

Day 1, The first incident10:12

Day 2, 08:40

Records arrive from devices that came back online.

The next morning, the engineering team checked on the devices it could not reach the day before.

Additional data had been secured from 66 of the 72 devices.

Some had their network connection restored overnight, while others reconnected as the morning shift began.

Only 6 devices are now without data.

The team checked those sites and their current connection status, and asked on-site staff to check the devices where needed.

DAY 2, 08:40

Devices not yet investigated drop from 72 to 6.

Devices without data over time

Devices that still need checking

DeviceSiteRemaining status
Device-0284Site-009Offline
Device-0628Site-018Offline
Device-1126Site-031Disconnected
Device-1862Site-048Offline
Device-2640Site-069Disconnected
Device-3108Site-084Offline
Without re-requesting the thousands of records it already has,the company focuses on the devices it has not reached yet.

Live contextNow

08:40Day 2

Investigation stage
Closing the investigation
Devices without data
6

WHAT WE KNOW

Additional data secured from 66 of 72 devices

WHAT WE NEED TO KNOW

Do the remaining devices hold the same records?

Day 2, 11:30

Investigation data from every device nationwide is secured.

At 11:30 a.m., as the site connection came back, data from the very last device reached the company.

Common investigation data from all 3,200 devices and detailed diagnostics from 50 devices were now fully secured.

Everything collected for the investigation was stored in a structure that keeps each site and device distinct.

The engineering team could see which file came from which device and find the original records again whenever the investigation needed them.

DAY 2, 11:30

Investigation data from devices nationwide is complete.

Initial investigation
3,200 / 3,200
Detailed investigation
50 / 50
Total data collected
3.02 TB

Collection results

CollectionDevicesDataStatus
Initial investigation3,2002.40 TBCompleted
Affected device diagnostics38480 GBCompleted
Reference device diagnostics12140 GBCompleted

INVESTIGATION STORAGE

  • INV-2026-014
  • Initial-investigation/
  • Site-024/
  • Device-0842/
  • Device-0856/
  • Device-0872/
  • Site-025/
  • Detailed-investigation/

Device-0842

  • runtime_20261109.log
  • runtime_20261110.log
  • system_error.log
  • device_config.json
  • software_version.json

Source device and collection record available

Checking the collection record of an original file

Source
Device-0842, Site-024
File
system_error.log
Investigation
INV-2026-014
Collected
Day 1, 11:24:18
Transfer result
Completed
Files gathered from thousands of devicesbecome evidence for investigating each one.

Live contextNow

11:30Day 2

Investigation stage
Closing the investigation
Data secured
3,200 / 3,200
Detailed diagnostics
50 / 50
Total data collected
3.02 TB

WHAT WE KNOW

Common investigation data secured from all 3,200 devices nationwide

WHAT WE NEED TO KNOW

Which operating conditions contributed to the problem?

FROM EVIDENCE TO DECISIONS, Day 3, 10:00

The company pins down the conditions behind the problem.

The engineering team compared the operation records secured from all 3,200 devices nationwide.

Similar anomalies also turned up in the records from the last devices to report in.

In the end, related records were confirmed on 41 devices in total.

Comparing the detailed diagnostics, the team concluded the problem was not confined to one software version: it most likely stemmed from an internal module shared across several versions, under specific operating conditions.

Based on the findings, the team separated the devices needing an immediate response from those needing further verification.

DAY 3, 10:00

From the first device to related records on 41 devices nationwide.

ENGINEERING INVESTIGATION REPORT

Fleet-wide comparison
3,200
Related records found
41
No matching records
3,159

The graphic is a schematic of findings across devices nationwide.

Comparing records from affected devices

Software versionInterimFinal
v4.8.22628
v4.8.189
v4.7.344
Total3841

ENGINEERING FINDINGS

A module shared across versions, under specific operating conditions

Shared software module
Same module confirmed on related devices
Long continuous operation
Common condition among devices in the detailed investigation
Specific error records
Repeated response delays in the related process
Follow-up verification
Reproduction tests and fix verification required

A fictional analysis by the company's engineering team. INNORIX does not diagnose the cause of device errors itself.

The company no longer needs to apply the same fix to every device nationwide.

RESPONSE PLAN

Response targets are set by the findings.

27 immediate response14 further verification

  • 27Immediate responseDevices meeting conditions with a high risk of shutdown
  • 14Further verificationDevices with similar records that need further diagnostics
  • 3,159Business as usualDevices with no matching anomaly in the records for the period investigated
Instead of guessing,the company can now respond based on records secured nationwide.

Live contextNow

10:00Day 3

Investigation stage
Final findings
Related records confirmed
41 devices

WHAT WE KNOW

Data from all 3,200 devices secured and compared

NEXT ACTION

Respond to 27 devices first, verify 14 more

ONE MONTH LATER, 1 month later

The company already holds the records from before a problem occurs.

With the investigation complete, the engineering team reviewed how its devices were being operated.

If records are secured only after a problem occurs, older files on the device may already have been deleted or overwritten.

The device may not even be connected to the network when the problem happens.

The company decided to collect and retain key operation records and error files from its devices on a regular schedule.

Detailed diagnostic files are now pulled only when a problem occurs or a further investigation needs them.

There is no need to keep collecting every file from every device.

From a company that looks for data after a problem,to one that secures the records it needs in advance.

BEFORE

Records secured after the problem

When a problem occurs, the team starts by securing whatever records remain on that device.

AFTER

Operation records secured continuously

The company can check records from before the problem and collect more data if needed.

The company’s new data collection workflows

Collection flowTargetTrigger
Daily operation recordsAll 3,200 devicesDaily
Error recordsAll 3,200 devicesWhen an error file is created
Device configurationAll 3,200 devicesWeekly
Detailed diagnosticsOnly the devices neededOn investigation request

INNORIX in Practice

Day 1, The first incident10:12

THE NEXT INCIDENT

Then one day, an anomaly appears on a different device.

It was a different site and a different device from a month earlier.

This time, the engineering team did not start by requesting every record from the device again.

They checked the recent operation records already secured in company storage.

They also compared them with records from problems that had occurred on other devices in the past.

Then they collected only the latest detailed diagnostic file they did not yet have.

THE NEXT INCIDENT

When a new problem occurs, the records are already there to check.

Device-2418

File history

CollectedFileSource
7 days agoruntime_01.logDevice-2418
5 days agoruntime_03.logDevice-2418
3 days agoruntime_05.logDevice-2418
Yesterdayruntime_07.logDevice-2418
Today 09:20system_error.logDevice-2418

ADDITIONAL COLLECTION REQUEST

Collecting

diagnostic_snapshot.zip

Detailed diagnostics from the device right now

Every time a new problem occurs,the company no longer has to start its investigation from scratch.

FROM A SINGLE INCIDENT TO FLEET-WIDE ANSWERS

The problem was found on one device,but the answer lay in the records left on devices nationwide.

THE INVESTIGATION

  1. First incident found1 device
  2. Fleet-wide investigation3,200 devices
  3. Detailed diagnostics50 devices
  4. Immediate response27 devices

The more devices a company runs,the more it matters to know what a problem found in one place means for the entire operation.

Wherever a problem occurs,a company should be able to secure the records it needs and move on to what comes next.

THE TECHNOLOGY BEHIND THIS STORY

Fleet-Scale Collection

Collect the files you need from devices and industrial systems running across the country. Gather common data from the entire fleet, or select devices that meet specific conditions to secure the detailed data you need.

From scheduled data collection to urgent collection when the unexpected happens, it supports the full range of your data operations.

Explore the product