PluginsSingerTasksTargets

DatamillCoPostgres​Datamill​Co​Postgres

yaml
type: "io.kestra.plugin.singer.targets.DatamillCoPostgres"

A Singer target loads data into a Postgres database.

Full documentation can be found here

Properties

from

  • Type: string
  • Dynamic: ❌
  • Required: ✔️

The raw data from a tap.

host

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Min length: 1

The database hostname.

port

  • Type: integer
  • Dynamic: ❌
  • Required: ✔️

The database port

stateName

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Default: singer-state

The name of Singer state file stored in KV Store.

username

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Min length: 1

The database user.

addUpsertIndexes

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌
  • Default: true

Whether the Target should create column indexes on the important columns used during data loading.

These indexes will make data loading slightly slower but the deduplication phase much faster. Defaults to on for better baseline performance.

afterRunSql

  • Type: string
  • Dynamic: ❌
  • Required: ❌

Raw SQL statement(s) to before closing the connection to Postgres.

batchDetectionThreshold

  • Type: integer
  • Dynamic: ❌
  • Required: ❌

How often, in rows received, to count the buffered rows and bytes to check if a flush is necessary.

There's a slight performance penalty to checking the buffered records count or bytesize, so this controls how often this is polled in order to mitigate the penalty. This value is usually not necessary to set as the default is dynamically adjusted to check reasonably often.

Default is 5000, or 1/40th maxBatchRows

beforeRunSql

  • Type: string
  • Dynamic: ❌
  • Required: ❌

Raw SQL statement(s) to execute as soon as the connection to Postgres is opened by the target.

Useful for setup like SET ROLE or other connection state that is important.

command

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Override default singer command.

containerImage

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: python:3.10.12

The task runner container image, only used if the task runner is container-based.

dbName

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The database name.

docker

Deprecated, use 'taskRunner' instead

invalidRecordsDetect

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌
  • Default: true

Crash on invalid records.

invalidRecordsThreshold

  • Type: integer
  • Dynamic: ❌
  • Required: ❌
  • Default: 0

Include a positive value n in your config to allow to encounter at most n invalid records per stream before giving up.

loggingLevel

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: INFO

The level for logging.

Set to DEBUG to get things like queries executed, timing of those queries, etc. See Python's Logger Levels for information about valid values.

maxBatchRows

  • Type: integer
  • Dynamic: ❌
  • Required: ❌
  • Default: 200000

The maximum number of rows to buffer in memory before writing to the destination table in Postgres.

maxBufferSize

  • Type: integer
  • Dynamic: ❌
  • Required: ❌
  • Default: 104857600

The maximum number of bytes to buffer in memory before writing to the destination table in Postgres.

password

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The database user's password.

persistEmptyTables

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌
  • Default: false

Whether the Target should create tables which have no records present in Remote.

pipPackages

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

Override default pip packages to use a specific version.

schema

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: public

The database schema.

sslMode

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: prefer

Refer to the libpq docs for more information about SSL.

taskRunner

  • Type: TaskRunner
  • Dynamic: ❌
  • Required: ❌
  • Default: { "type": "io.kestra.plugin.scripts.runner.docker.Docker" }

The task runner to use.

Task runners are provided by plugins, each have their own properties.

Outputs

stateKey

  • Type: string
  • Required: ❌

Definitions

io.kestra.plugin.scripts.runner.docker.Cpu

  • cpus
    • Type: integer
    • Dynamic: ❌
    • Required: ❌

io.kestra.core.models.tasks.runners.TaskRunner

  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Validation RegExp: \p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*(\.\p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*)*
    • Min length: 1

io.kestra.plugin.scripts.runner.docker.Memory

  • kernelMemory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memoryReservation
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwap
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwappiness
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • oomKillDisable
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌

io.kestra.plugin.scripts.exec.scripts.models.DockerOptions

  • image
    • Type: string
    • Dynamic: ✔️
    • Required: ✔️
    • Min length: 1
  • config
    • Type:
      • string
      • object
    • Dynamic: ✔️
    • Required: ❌
  • cpu
    • Type: Cpu
    • Dynamic: ❌
    • Required: ❌
  • credentials
  • deviceRequests
  • entryPoint
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • extraHosts
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • host
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: Memory
    • Dynamic: ❌
    • Required: ❌
  • networkMode
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • pullPolicy
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: ALWAYS
    • Possible Values:
      • IF_NOT_PRESENT
      • ALWAYS
      • NEVER
  • shmSize
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • user
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • volumes
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.Credentials

  • auth
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • identityToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • password
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registry
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registryToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • username
    • Type: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.DeviceRequest

  • capabilities
    • Type: array
    • SubType: array
    • Dynamic: ❌
    • Required: ❌
  • count
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
  • deviceIds
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • driver
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • options
    • Type: object
    • SubType: string
    • Dynamic: ❌
    • Required: ❌