Filters

Filter is the stage that decides which rows survive. It runs after the
transform and before the mapping, so it
judges finished values, including anything a calculation derived, and everything it
refuses is produced, judged, and then thrown away without reaching the destination.

The choices

Choice	What it does
None	Every row the task produces is written
Conditions	Keep only the rows for which a Python condition holds
TimeFrame filter	Marked coming later in the dialog. Not available yet.

What a filter does to a run

A dropped row is gone for good. It never reaches the destination, the hold-back, the
checkpoint or a JSON document, and it does not advance the written-row index. The run log
and every preview carry filter kept [k] of [n] rows.

A window in which the filter drops every row still advances the watermark, so an
outage is not re-read forever.

To see what a filter refused, run Preview and tick Show filtered:
the dropped rows appear among the kept ones, on a light red ground.

Where a filter is available

The filter feeds Table and JSON outputs. Choosing a filter against a write-back
moves the output to Table, and the console says why.

What's Next

• Conditions: the filter available today
• Preview: the rows a destination would receive, and the ones refused
• Mapping: the columns the surviving rows become

Related

• Tasks: the whole pipeline this is one stage of
• Transformations: the stage that runs before it
• Object properties: every property a condition can read