PluginsSingerTasksTargets

AdswerveBigQuery​Adswerve​Big​Query

yaml
type: "io.kestra.plugin.singer.targets.AdswerveBigQuery"

A Singer target loads data into a BigQuery.

Full documentation can be found here

Properties

datasetId

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Min length: 1

The BigQuery dataset.

from

  • Type: string
  • Dynamic: ❌
  • Required: ✔️

The raw data from a tap.

projectId

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Min length: 1

The BigQuery project.

stateName

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Default: singer-state

The name of Singer state file stored in KV Store.

validateRecords

  • Type: boolean
  • Dynamic: ❌
  • Required: ✔️
  • Default: false

Validate every single record message to the corresponding JSON schema.

This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation..

addMetadataColumns

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌
  • Default: false

Add singer Metadata columns.

Add _time_extracted and _time_loaded metadata columns.

command

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Override default singer command.

containerImage

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: python:3.10.12

The task runner container image, only used if the task runner is container-based.

docker

Deprecated, use 'taskRunner' instead

location

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The Dataset location.

maxCache

  • Type: integer
  • Dynamic: ❌
  • Required: ❌
  • Default: 50

Maximum cache size in MB.

mergeStateMessages

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌
  • Default: false

Enable control state flush.

default: merges multiple state messages from the tap into the state file, if true : uses the last state message as the state file.

pipPackages

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

Override default pip packages to use a specific version.

replicationMethod

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Default: append
  • Possible Values:
    • append
    • truncate

The replication method, append or truncate.

serviceAccount

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The JSON service account key as string.

tableConfigs

  • Type: object
  • Dynamic: ❌
  • Required: ❌

Table configs.

tablePrefix

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Add prefix to table name.

tableSuffix

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Add suffix to table name.

taskRunner

  • Type: TaskRunner
  • Dynamic: ❌
  • Required: ❌
  • Default: { "type": "io.kestra.plugin.scripts.runner.docker.Docker" }

The task runner to use.

Task runners are provided by plugins, each have their own properties.

Outputs

stateKey

  • Type: string
  • Required: ❌

Definitions

io.kestra.plugin.scripts.runner.docker.Cpu

  • cpus
    • Type: integer
    • Dynamic: ❌
    • Required: ❌

io.kestra.core.models.tasks.runners.TaskRunner

  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Validation RegExp: \p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*(\.\p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*)*
    • Min length: 1

io.kestra.plugin.scripts.runner.docker.Memory

  • kernelMemory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memoryReservation
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwap
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwappiness
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • oomKillDisable
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌

io.kestra.plugin.scripts.exec.scripts.models.DockerOptions

  • image
    • Type: string
    • Dynamic: ✔️
    • Required: ✔️
    • Min length: 1
  • config
    • Type:
      • string
      • object
    • Dynamic: ✔️
    • Required: ❌
  • cpu
    • Type: Cpu
    • Dynamic: ❌
    • Required: ❌
  • credentials
  • deviceRequests
  • entryPoint
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • extraHosts
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • host
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: Memory
    • Dynamic: ❌
    • Required: ❌
  • networkMode
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • pullPolicy
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: ALWAYS
    • Possible Values:
      • IF_NOT_PRESENT
      • ALWAYS
      • NEVER
  • shmSize
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • user
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • volumes
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.Credentials

  • auth
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • identityToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • password
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registry
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registryToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • username
    • Type: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.DeviceRequest

  • capabilities
    • Type: array
    • SubType: array
    • Dynamic: ❌
    • Required: ❌
  • count
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
  • deviceIds
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • driver
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • options
    • Type: object
    • SubType: string
    • Dynamic: ❌
    • Required: ❌