What this solves#
The first thing that stalls a transfer is what to send. A source folder does not hold only the files you want. Work-in-progress temp files, design sources, last month's logs and pre-compression originals all sit in the same place. There are too many to pick one by one, and sending the folder whole sends things that should never leave.
This document covers the four things you can do at that point.
- Pick a folder, not files one at a time.
Item filters- narrow by type, by a regular expression, or by size.- Send only what changed instead of everything, every time.
- Datasets - name the same path across many devices once and reuse it.
This document covers the source side only. How files are written at the destination (
Save path,File conflict) is a different step, and reading the result has its own guides - see related.
Who this is for#
| Situation | What to use |
|---|---|
| An inspection machine piles up tens of thousands of images a day, but only the result files should go | Types in Item filters |
| A log folder holds months of history and only new files should be collected | Regular exp. on the date (why) |
| Only the large originals should go to the archive | Minimum size in Item filters |
| The same path on 100 plants is being picked by hand every time | A Dataset |
| Files are only told apart by their names | Regular exp. |
Before you start#
- The sending device must be connected. A device that is not connected cannot be picked.
- You need permission to browse the path, or the folder will not open.
- To create a Dataset you need at least one connected device - without one the screen says "Connect a device first to add a data set."
Where you pick#
- Flows → pick the Flow → Edit flow.
- On the canvas, select the source endpoint. Its row appears in the panel on the right.
- With nothing picked yet, the screen says so - "Nothing is sent until you choose the folders or files to send." Saving in that state sends nothing.
- Press the path field to open the browser. Once something is picked the hint becomes "Click to browse or change selected files".
- Pick a folder. You do not have to pick files one at a time. The panel then shows the path and its size.
Picking files and picking a folder are not the same. Name a file and that file goes. Name a folder and whatever is in it when the flow runs goes. For a folder that gains files every day, pick the folder.
The three Item filters#
Press the filter chip on the endpoint row and the Item filters dialog opens. The chip's own wording tells you the state.
| The chip reads | Meaning |
|---|---|
Add filters |
This endpoint has no filters yet |
Item filters |
This endpoint has filters |
Use default |
With several sources, this one follows the shared filters |
Override |
With several sources, this one has its own |
The dialog has three tabs - Types, Name and Size. Press Apply when they are set, or Cancel to drop them.
Types - by extension#
Choose Allow or Deny and enter extensions.
Allow- only what you list goes. Safer when the kinds you want are known (mp4,json).Deny- only what you list is left out. Use it when what to exclude is known (tmp,bak).
Each entry can be removed on its own (Remove {{type}}).
Start from
Allow.Denymeans "send everything except these", so a new extension appearing in the folder later goes out without a word.
Name - by file name#
This tab holds three.
| Option | What it does |
|---|---|
Include |
only files whose name contains what you type |
Exclude |
leaves out files whose name contains it |
Regular exp. |
matches with a regular expression |
Do not reach for a regular expression where Include or Exclude will do - "only the ones with _OK" is an Include. Regular exp. is for names an extension or a single word cannot tell apart, such as LINE3_20261115_OK.csv, where each part of the name means something.
Size - by size#
There is Minimum size and Maximum size, both in MB, and you may set only one of them. The screen says so:
Use either a minimum or a maximum size – one limit per flow.Minimum size- leaves out the small ones: empty files, header-only files, truncated files.Maximum size- leaves out the large ones, for when a single original would eat the whole night window.
Zero, a negative or a non-number is refused with "Enter a size above 0, e.g. 10 or 1.5." Decimals are allowed (1.5).
Filters you set show up as chips#
Once you press Apply, a summary rides on the endpoint row - a name condition as Name <operator> '<value>', a size condition as Size <symbol> <value> <unit>. You can read what is set without opening the dialog.
"Only what is new" cannot be chosen yet#
Sending the same folder again and again, you will want "only what is new". That option cannot be chosen on screen today. Here is what we found, as we found it.
| What | Where it stands |
|---|---|
A Flow's Start when |
Date/time, After transfer, URL request - three options. Nothing watches a folder. The first step shows only two - After transfer has nothing before it to wait for, so it is left out entirely rather than greyed |
Sync on with New items / Modified items |
It only appears on a Flow whose trigger is the folder watch. That trigger cannot be chosen, so the field never opens |
So the way to narrow what goes is Item filters. Types, a regular expression and size keep files you do not need from leaving at all. Where the date is in the file name, Regular exp. for that day's files is the closest thing to "only what is new".
If resending everything is the worry, handle it at the destination. Set
File conflicttoSkipand a file that is already there is passed over - it is still offered, but nothing is overwritten.
Grouping many devices - Datasets#
Picking /var/log/inspection on 100 plants, 100 times, does not survive contact with operations. Naming that group once is what a Dataset is for.
Creating one#
- Datasets → Create dataset.
- Give it a
Dataset name. - Under
Connection settings, add resources withAdd resource. One resource is aDeviceand aPath.Browse pathpicks the path;Refresh pathreads it from the device again. - As resources build up the table shows
Device,Paths,Files,Size,StatusandAction.Remove resourcetakes one out. - With nothing added yet you see "No resources yet. Add a resource to get started."
- Keep it with
Save draftor publish it withSave & publish. Discarding asks "Discard this draft?" - and only the draft is discarded; the files and folders on the device are untouched.
Reading one#
Open a group and you see Sources (how many places), Contents (what is in them) and Files & paths. The list itself has the columns Dataset, Devices, Path, Files, Size, Last updated, Last run and Action.
Read Last updated first - if files changed on the device and the group has not been read since, it no longer matches reality.
Using one in a flow#
Drag the Dataset onto a step. Three cases are refused, each with its reason on screen.
| The screen says | Meaning |
|---|---|
This dataset is already in the flow. |
The same group was added twice |
Add a device to this dataset before using it in a flow. |
The group has no resources |
Every device in this dataset must be connected before using it in a flow. |
Some device in the group is not connected |
The last one is what operations hits most. One powered-off site in a group of 100 blocks the whole group, so keeping the flaky sites in a smaller separate group is the practical answer.
Checking that the filters caught the right thing#
A run tells you immediately.
- After it runs, open that run in Runs.
- On
Transferred files, look at what actually went. An extension you meant to exclude showing up usually meansTypesis set toDeny. - Per device,
Devices N of M completedsays how many of how many. - If nothing went at all, check in this order - (1) was a path picked, (2) is the
Allowlist too narrow, (3) does the date inRegular exp.match today.
Errors and exceptions#
| Symptom | Cause | What to do |
|---|---|---|
| Saved, but nothing goes | No source was picked | Look for "Nothing is sent until you choose the folders or files to send." on the endpoint |
| Both size limits cannot be set | One limit per flow | Split the flow if you need both |
| The size field is refused | Zero or not a number | "Enter a size above 0, e.g. 10 or 1.5." |
| Filters differ per endpoint | An Override is in place |
Check whether the chip reads Use default or Override |
| A Dataset cannot be placed | No resources, or a device is offline | See the three messages under using one in a flow |
| The dataset list looks empty | It failed to load | "Could not load datasets." - retry and check the connection |
Running this in production#
- Filters are per endpoint. Fixing one endpoint in a multi-source flow leaves the others as they were. The
Use default/Overridechip is how you tell. - A
Denylist leaks over time. A new extension in the folder goes out with everything else. For long-lived flows,Allowis the safer shape. - A size limit is a hidden cause of "it never arrived". With
Maximum sizeset, large originals drop out quietly. Write down the limits you set. - Put a Dataset's
Last updatedon your operations checklist. A group that has drifted from the devices turns into "we sent it, but that file is not there". - Narrowing the selection is the cheapest performance work there is. A file that is not sent costs neither time nor bandwidth.
By industry#
| Industry | How it is set |
|---|---|
| Semiconductor, manufacturing | Pick the inspection folder whole, set Types to Allow and list csv and json. Image originals drop out by themselves |
| Broadcast, media | Set a Minimum size on the originals folder to leave thumbnails and proxies behind |
| Finance, public sector | Names carry the date and branch code, so Regular exp. takes only that day's files |
| Retail | Group the store content folders as a Dataset and set File conflict to Skip so content already there is passed over |
| Healthcare, research | Result file names carry the date, so Regular exp. takes only that day's - past studies are not sent again |
| Energy, infrastructure | Keep remote sites as a Dataset, and split the unreliable ones into a smaller group so one site cannot block the rest |