Picking Which Files Go, from Folders to Datasets

How to decide what leaves a folder of hundreds of thousands of files: picking a folder, the three Item filters, sending only what changed, and Datasets.

IT Engineers

What this solves#

The first thing that stalls a transfer is what to send. A source folder does not hold only the files you want. Work-in-progress temp files, design sources, last month's logs and pre-compression originals all sit in the same place. There are too many to pick one by one, and sending the folder whole sends things that should never leave.

This document covers the four things you can do at that point.

  1. Pick a folder, not files one at a time.
  2. Item filters - narrow by type, by a regular expression, or by size.
  3. Send only what changed instead of everything, every time.
  4. Datasets - name the same path across many devices once and reuse it.

This document covers the source side only. How files are written at the destination (Save path, File conflict) is a different step, and reading the result has its own guides - see related.

Who this is for#

Situation What to use
An inspection machine piles up tens of thousands of images a day, but only the result files should go Types in Item filters
A log folder holds months of history and only new files should be collected Regular exp. on the date (why)
Only the large originals should go to the archive Minimum size in Item filters
The same path on 100 plants is being picked by hand every time A Dataset
Files are only told apart by their names Regular exp.

Before you start#

  • The sending device must be connected. A device that is not connected cannot be picked.
  • You need permission to browse the path, or the folder will not open.
  • To create a Dataset you need at least one connected device - without one the screen says "Connect a device first to add a data set."

Where you pick#

  1. Flows → pick the Flow → Edit flow.
  2. On the canvas, select the source endpoint. Its row appears in the panel on the right.
  3. With nothing picked yet, the screen says so - "Nothing is sent until you choose the folders or files to send." Saving in that state sends nothing.
  4. Press the path field to open the browser. Once something is picked the hint becomes "Click to browse or change selected files".
  5. Pick a folder. You do not have to pick files one at a time. The panel then shows the path and its size.

Picking files and picking a folder are not the same. Name a file and that file goes. Name a folder and whatever is in it when the flow runs goes. For a folder that gains files every day, pick the folder.

The three Item filters#

Press the filter chip on the endpoint row and the Item filters dialog opens. The chip's own wording tells you the state.

The chip reads Meaning
Add filters This endpoint has no filters yet
Item filters This endpoint has filters
Use default With several sources, this one follows the shared filters
Override With several sources, this one has its own

The dialog has three tabs - Types, Name and Size. Press Apply when they are set, or Cancel to drop them.

Types - by extension#

Choose Allow or Deny and enter extensions.

  • Allow - only what you list goes. Safer when the kinds you want are known (mp4, json).
  • Deny - only what you list is left out. Use it when what to exclude is known (tmp, bak).

Each entry can be removed on its own (Remove {{type}}).

Start from Allow. Deny means "send everything except these", so a new extension appearing in the folder later goes out without a word.

Name - by file name#

This tab holds three.

Option What it does
Include only files whose name contains what you type
Exclude leaves out files whose name contains it
Regular exp. matches with a regular expression

Do not reach for a regular expression where Include or Exclude will do - "only the ones with _OK" is an Include. Regular exp. is for names an extension or a single word cannot tell apart, such as LINE3_20261115_OK.csv, where each part of the name means something.

Size - by size#

There is Minimum size and Maximum size, both in MB, and you may set only one of them. The screen says so:

Use either a minimum or a maximum size – one limit per flow.
  • Minimum size - leaves out the small ones: empty files, header-only files, truncated files.
  • Maximum size - leaves out the large ones, for when a single original would eat the whole night window.

Zero, a negative or a non-number is refused with "Enter a size above 0, e.g. 10 or 1.5." Decimals are allowed (1.5).

Filters you set show up as chips#

Once you press Apply, a summary rides on the endpoint row - a name condition as Name <operator> '<value>', a size condition as Size <symbol> <value> <unit>. You can read what is set without opening the dialog.

"Only what is new" cannot be chosen yet#

Sending the same folder again and again, you will want "only what is new". That option cannot be chosen on screen today. Here is what we found, as we found it.

What Where it stands
A Flow's Start when Date/time, After transfer, URL request - three options. Nothing watches a folder. The first step shows only two - After transfer has nothing before it to wait for, so it is left out entirely rather than greyed
Sync on with New items / Modified items It only appears on a Flow whose trigger is the folder watch. That trigger cannot be chosen, so the field never opens

So the way to narrow what goes is Item filters. Types, a regular expression and size keep files you do not need from leaving at all. Where the date is in the file name, Regular exp. for that day's files is the closest thing to "only what is new".

If resending everything is the worry, handle it at the destination. Set File conflict to Skip and a file that is already there is passed over - it is still offered, but nothing is overwritten.

Grouping many devices - Datasets#

Picking /var/log/inspection on 100 plants, 100 times, does not survive contact with operations. Naming that group once is what a Dataset is for.

Creating one#

  1. Datasets → Create dataset.
  2. Give it a Dataset name.
  3. Under Connection settings, add resources with Add resource. One resource is a Device and a Path. Browse path picks the path; Refresh path reads it from the device again.
  4. As resources build up the table shows Device, Paths, Files, Size, Status and Action. Remove resource takes one out.
  5. With nothing added yet you see "No resources yet. Add a resource to get started."
  6. Keep it with Save draft or publish it with Save & publish. Discarding asks "Discard this draft?" - and only the draft is discarded; the files and folders on the device are untouched.

Reading one#

Open a group and you see Sources (how many places), Contents (what is in them) and Files & paths. The list itself has the columns Dataset, Devices, Path, Files, Size, Last updated, Last run and Action.

Read Last updated first - if files changed on the device and the group has not been read since, it no longer matches reality.

Using one in a flow#

Drag the Dataset onto a step. Three cases are refused, each with its reason on screen.

The screen says Meaning
This dataset is already in the flow. The same group was added twice
Add a device to this dataset before using it in a flow. The group has no resources
Every device in this dataset must be connected before using it in a flow. Some device in the group is not connected

The last one is what operations hits most. One powered-off site in a group of 100 blocks the whole group, so keeping the flaky sites in a smaller separate group is the practical answer.

Checking that the filters caught the right thing#

A run tells you immediately.

  1. After it runs, open that run in Runs.
  2. On Transferred files, look at what actually went. An extension you meant to exclude showing up usually means Types is set to Deny.
  3. Per device, Devices N of M completed says how many of how many.
  4. If nothing went at all, check in this order - (1) was a path picked, (2) is the Allow list too narrow, (3) does the date in Regular exp. match today.

Errors and exceptions#

Symptom Cause What to do
Saved, but nothing goes No source was picked Look for "Nothing is sent until you choose the folders or files to send." on the endpoint
Both size limits cannot be set One limit per flow Split the flow if you need both
The size field is refused Zero or not a number "Enter a size above 0, e.g. 10 or 1.5."
Filters differ per endpoint An Override is in place Check whether the chip reads Use default or Override
A Dataset cannot be placed No resources, or a device is offline See the three messages under using one in a flow
The dataset list looks empty It failed to load "Could not load datasets." - retry and check the connection

Running this in production#

  • Filters are per endpoint. Fixing one endpoint in a multi-source flow leaves the others as they were. The Use default / Override chip is how you tell.
  • A Deny list leaks over time. A new extension in the folder goes out with everything else. For long-lived flows, Allow is the safer shape.
  • A size limit is a hidden cause of "it never arrived". With Maximum size set, large originals drop out quietly. Write down the limits you set.
  • Put a Dataset's Last updated on your operations checklist. A group that has drifted from the devices turns into "we sent it, but that file is not there".
  • Narrowing the selection is the cheapest performance work there is. A file that is not sent costs neither time nor bandwidth.

By industry#

Industry How it is set
Semiconductor, manufacturing Pick the inspection folder whole, set Types to Allow and list csv and json. Image originals drop out by themselves
Broadcast, media Set a Minimum size on the originals folder to leave thumbnails and proxies behind
Finance, public sector Names carry the date and branch code, so Regular exp. takes only that day's files
Retail Group the store content folders as a Dataset and set File conflict to Skip so content already there is passed over
Healthcare, research Result file names carry the date, so Regular exp. takes only that day's - past studies are not sent again
Energy, infrastructure Keep remote sites as a Dataset, and split the unreliable ones into a smaller group so one site cannot block the rest