Task Output

The Output tab is the last decision: where the result goes, and what happens to what
is already there.

For a Table or JSON task that means an endpoint and a format. For a
Write-back task there is no endpoint to choose: the results go back to the
channels the calculation names, and one decision remains, which is the one below.

Write-back mode

On a write-back task the Output tab holds a single control: what happens when the
task writes a value at a timestamp the archive already holds one for.

Mode	What lands
Insert (no compression)	The new value is added beside the existing one, bypassing the archive's compression. The default, and what every task did before this setting existed
Insert (compressed)	Added beside the existing value, through compression
Replace	The new value replaces the one at that timestamp
Keep existing (no replace)	Written only where the timestamp holds nothing yet

The default is right for a task moving steadily forward: it never revisits a
timestamp, so nothing collides.

Reprocessing needs Replace

The default adds rather than overwrites. So a task that re-reads a window, after a
Data start override: writes a second value at
every timestamp it had already written, and the tag ends up with two events at one
instant. If this task reprocesses windows, choose Replace.

Changing the mode affects the next writes only; nothing already written is touched.

Endpoint

Choose one of the destination endpoints configured on the
Endpoints page. Only destinations appear here: a PI System endpoint
is a source and is not offered.

Underneath the dropdown, Stream tells you how the chosen endpoint will interpret the
table name from the Mapping tab, for a file endpoint, that each
/ becomes a folder, one sub-folder per resolved value. Read it to confirm the
partitioning will produce the layout you intended.

Endpoint options

What appears here depends on the endpoint type. A file destination has the most to
decide.

Format

CSV and JSON are available. Parquet and Avro appear in the list and
are planned: their writers are not built yet, and the tab says so.

On each run

This is the setting to get right, because it decides what happens to the data from
the last run.

Mode	What happens
Append	The previous run's file is kept and extended
Replace	The previous content is overwritten

Append is right for a history you are accumulating: every run adds its readings
to a growing record.

Replace is right for a snapshot: a file that should always hold the current
state and nothing older.

JSON does not accept Append

The two settings are not independent: JSON output refuses Append, and the task is
rejected when you save it. A JSON file is a single array, so it is only well-formed
once, at the end. Extending it in place would mean concatenating arrays into something
no parser reads.

Use Replace, and put a time token in the map's name on the
Mapping tab, [scope.gmt.Format=yyyyMMdd] and the like, so each
period gets its own file. That is almost always what someone reaching for Append on a
JSON target actually wants.

Append and a fixed target together grow without limit

Append into a single unpartitioned target means one file that grows forever. A
task running every fifteen minutes writes about 35,000 batches a year into it. Pair
Append with a dated partition on the Mapping tab, so each day
or month starts a new target.

Header row

Write column names first puts a header row at the top of a CSV.

The header is written only when a file is started. Appending to a file that already exists adds rows and never a second
header. So you get one header per file, wherever your partitioning happens to start
new files: which is what a reader expects, and what you would otherwise have to
work around.

Field delimiter

The character between values. A comma by default. Change it to a semicolon or a tab
where a comma appears in your data, or where the system reading the file expects
something else.

Row delimiter

The character between rows. Leave it empty to use the service's own newline, which is
the right choice unless a downstream system specifically requires the other
convention.

SQL destinations

A SQL endpoint has fewer choices: format and delimiters do not apply. What matters
instead is on the Mapping tab: the table name, the column types, and which columns
are indexed.

Two things to check before the first run:

• The account can create or write the table. Stream reports this as a write
  failure in Logs, which reads like a task problem and is a permissions
  problem.
• The column types are compatible with an existing table, if you are writing into
  one that already exists.

Databricks and Snowflake destinations

These two land Parquet files and can then load them into a table, so the Output tab
shows a different set of controls: an Ingest mode, and: depending on the mode
: a table override, a Create table checkbox, a statement, and Partition
folders.

Above them, the tab resolves two things for you as you type: which table the rows
will land in, and the folder the files go to. Both are worth a glance before you
save; they answer the question a name expression usually leaves open.

Because there is a lot to say about the modes, they have their own page:
Loading a Lake Table →.

The short version: Copy Into loads a real table and is the sensible default,
View makes the files queryable without copying them (Databricks only), None
lands files for something else to pick up, and Custom runs SQL you write.

Before you save

Run the task once by hand from the Tasks page, then look at the destination and
confirm three things:

1. The output arrived where you expected: the right table, or the right folder.
2. The header and delimiters are as the consuming system needs.
3. A second run behaves correctly: appending where you meant to append, replacing
   where you meant to replace. This is the one people discover late.

What's Next

The task is complete. Give it a schedule so it runs unattended: continue to
Task Scheduler →.

Related

• Endpoints: configuring the destination this tab selects
• Loading a Lake Table: the ingest modes for Databricks and Snowflake
• Task Mapping: the table name, partitioning, and column types
• Logs: diagnosing a write that failed