TransformItems
Parse arbitrary text and structure it using Grok expressions.
The TransformItems task is similar to the famous Logstash Grok filter from the ELK stack.
It is particularly useful for transforming unstructured data such as logs into a structured, indexable, and queryable data structure.
The TransformItems ships with all the default patterns as defined You can find them here: https://github.com/kestra-io/plugin-transform/tree/main/plugin-transform-grok/src/main/resources/patterns.
type: "io.kestra.plugin.transform.grok.TransformItems"Consume, parse, and structure logs events from Kafka topic.
id: grok_transform_items
namespace: company.team
tasks:
- id: transform_items
type: io.kestra.plugin.transform.grok.TransformItems
pattern: "%{TIMESTAMP_ISO8601:logdate} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"
from: "{{ trigger.uri }}"
triggers:
- id: trigger
type: io.kestra.plugin.kafka.Trigger
topic: test_kestra
properties:
bootstrap.servers: localhost:9092
serdeProperties:
schema.registry.url: http://localhost:8085
keyDeserializer: STRING
valueDeserializer: STRING
groupId: kafkaConsumerGroupId
interval: PT30S
maxRecords: 5
The file to be transformed.
Must be a kestra:// internal storage URI.
The Grok pattern to match.
Custom pattern definitions.
A map of pattern-name and pattern pairs defining custom patterns to be used by the current tasks. Patterns matching existing names will override the pre-existing definition.
The list of Grok patterns to match.
List of user-defined pattern directories.
Directories must be paths relative to the working directory.