DeduplicateItems

yaml
type: "io.kestra.plugin.core.storage.DeduplicateItems"

Deduplicate a file by retaining only the latest item for each extracted key.

The Deduplicate task involves reading the input file twice, rather than loading the entire file into memory. The first iteration is used to build a deduplication map in memory containing the last lines observed for each key. The second iteration is used to rewrite the file without the duplicates. The task must be used with this in mind.

Examples

yaml
id: "deduplicateitems"
type: "io.kestra.plugin.core.storage.DeduplicateItems"
tasks:
   - id: deduplicate
     type: io.kestra.plugin.core.storage.DeduplicateItems
     from: "{{ inputs.uri }}"
     expr: "{{ key }}"