PluginsSingerTasksTaps

BigQuery​Big​Query

yaml
type: "io.kestra.plugin.singer.taps.BigQuery"

A Singer tap to fetch data from a BigQuery.

Full documentation can be found here

Properties

limit

  • Type: integer
  • Dynamic: ❌
  • Required: ✔️

Limits the number of records returned in each stream, applied as a limit in the query.

startAlwaysInclusive

  • Type: boolean
  • Dynamic: ❌
  • Required: ✔️
  • Default: true

When replicating incrementally, disable to only select records whose datetime_key is greater than the maximum value replicated in the last run, by excluding records whose timestamps match exactly.

This could cause records to be missed that were created after the last run finished, but during the same second and with the same timestamp.

startDateTime

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Format: date-time

Determines how much historical data will be extracted.

Please be aware that the larger the time period and amount of data, the longer the initial extraction can be expected to take.

stateName

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Default: singer-state

The name of Singer state file stored in KV Store.

streams

  • Type: array
  • SubType: BigQuery-Stream
  • Dynamic: ❌
  • Required: ✔️
  • Min items: 1

Array holding objects describing streams (tables) to extract, with name, table, columns, datetime_key, and filters keys.

streamsConfigurations

The list of stream configurations

command

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Override default singer command.

containerImage

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: python:3.10.12

The task runner container image, only used if the task runner is container-based.

docker

Deprecated, use 'taskRunner' instead

endDateTime

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Format: date-time

Date up to when historical data will be extracted.

pipPackages

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

Override default pip packages to use a specific version.

serviceAccount

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The JSON service account key as string.

taskRunner

  • Type: TaskRunner
  • Dynamic: ❌
  • Required: ❌
  • Default: { "type": "io.kestra.plugin.scripts.runner.docker.Docker" }

The task runner to use.

Task runners are provided by plugins, each have their own properties.

Outputs

count

  • Type: integer
  • Required: ❌

raw

  • Type: string
  • SubType: string
  • Required: ❌
  • Format: uri

Definitions

io.kestra.plugin.scripts.runner.docker.Cpu

  • cpus
    • Type: integer
    • Dynamic: ❌
    • Required: ❌

io.kestra.core.models.tasks.runners.TaskRunner

  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Validation RegExp: \p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*(\.\p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*)*
    • Min length: 1

io.kestra.plugin.scripts.runner.docker.Memory

  • kernelMemory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memoryReservation
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwap
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwappiness
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • oomKillDisable
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌

io.kestra.plugin.scripts.exec.scripts.models.DockerOptions

  • image
    • Type: string
    • Dynamic: ✔️
    • Required: ✔️
    • Min length: 1
  • config
    • Type:
      • string
      • object
    • Dynamic: ✔️
    • Required: ❌
  • cpu
    • Type: Cpu
    • Dynamic: ❌
    • Required: ❌
  • credentials
  • deviceRequests
  • entryPoint
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • extraHosts
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • host
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: Memory
    • Dynamic: ❌
    • Required: ❌
  • networkMode
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • pullPolicy
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: ALWAYS
    • Possible Values:
      • IF_NOT_PRESENT
      • ALWAYS
      • NEVER
  • shmSize
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • user
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • volumes
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.singer.taps.BigQuery-Stream

  • columns
    • Type: array
    • SubType: string
    • Dynamic: ❌
    • Required: ❌
  • datetime_key
    • Type: string
    • Dynamic: ❌
    • Required: ❌
  • filters
    • Type: array
    • SubType: string
    • Dynamic: ❌
    • Required: ❌
  • name
    • Type: string
    • Dynamic: ❌
    • Required: ❌
  • table
    • Type: string
    • Dynamic: ❌
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.Credentials

  • auth
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • identityToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • password
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registry
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registryToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • username
    • Type: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.DeviceRequest

  • capabilities
    • Type: array
    • SubType: array
    • Dynamic: ❌
    • Required: ❌
  • count
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
  • deviceIds
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • driver
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • options
    • Type: object
    • SubType: string
    • Dynamic: ❌
    • Required: ❌

io.kestra.plugin.singer.models.StreamsConfiguration

  • propertiesPattern
    • Type: array
    • SubType: string
    • Dynamic: ❌
    • Required: ❌
  • replicationKeys
    • Type: string
    • Dynamic: ❌
    • Required: ❌
  • replicationMethod
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Possible Values:
      • FULL_TABLE
      • INCREMENTAL
      • LOG_BASED
  • selected
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
    • Default: true
  • stream
    • Type: string
    • Dynamic: ❌
    • Required: ❌