PluginsSingerTasksTaps

PipelinewiseSqlServer​Pipelinewise​Sql​Server

yaml
type: "io.kestra.plugin.singer.taps.PipelinewiseSqlServer"

A Singer tap to fetch data from a Microsoft SQL Server database.

Full documentation can be found here

Examples

yaml
id: "pipelinewisesqlserver"
type: "io.kestra.plugin.singer.taps.PipelinewiseSqlServer"
host: 127.0.0.1
username: SA
password: sqlserver_passwd
port: 57037
filterDbs: dbo
streamsConfigurations:
  - stream: Categories
    replicationMethod: INCREMENTAL
    replicationKeys: CategoryID
    selected: true
  - propertiesPattern:
      - Description
    selected: false

Properties

database

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Min length: 1

The database name.

host

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Min length: 1

The database hostname.

password

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Min length: 1

The database user's password.

port

  • Type: integer
  • Dynamic: ❌
  • Required: ✔️

The database port.

stateName

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Default: singer-state

The name of Singer state file stored in KV Store.

streamsConfigurations

The list of stream configurations

useDateDatatype

  • Type: boolean
  • Dynamic: ❌
  • Required: ✔️
  • Default: true

Emit Date datatypes as-is without converting them to datetime.

username

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Min length: 1

The database user.

characterSet

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The characterset for the database / source system. The default is utf8, however older databases might use a charactersets like cp1252 for the encoding.

command

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Override default singer command.

containerImage

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: python:3.10.12

The task runner container image, only used if the task runner is container-based.

cursorArraySize

  • Type: integer
  • Dynamic: ❌
  • Required: ❌

A numeric setting adjusting the internal buffersize for the tap (default 10000).

The common query tuning scenario is for SELECT statements that return a large number of rows over a slow network. Increasing arraysize can improve performance by reducing the number of round-trips to the database. However increasing this value increases the amount of memory required.

docker

Deprecated, use 'taskRunner' instead

filterDbs

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

The list of schemas to extract tables only from particular schemas and to improve data extraction performance.

pipPackages

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

Override default pip packages to use a specific version.

taskRunner

  • Type: TaskRunner
  • Dynamic: ❌
  • Required: ❌
  • Default: { "type": "io.kestra.plugin.scripts.runner.docker.Docker" }

The task runner to use.

Task runners are provided by plugins, each have their own properties.

tdsVersion

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

TDS version to use when communicating with SQL Server (default is 7.3).

useSingerDecimal

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌

Emit all numeric values as strings and treat floats as string data types for the target (default false).

When true, the resulting SCHEMA message will contain an attribute in additionalProperties containing the scale and precision of the discovered property.

Outputs

count

  • Type: integer
  • Required: ❌

raw

  • Type: string
  • SubType: string
  • Required: ❌
  • Format: uri

Definitions

io.kestra.plugin.scripts.runner.docker.Cpu

  • cpus
    • Type: integer
    • Dynamic: ❌
    • Required: ❌

io.kestra.core.models.tasks.runners.TaskRunner

  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Validation RegExp: \p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*(\.\p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*)*
    • Min length: 1

io.kestra.plugin.scripts.runner.docker.Memory

  • kernelMemory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memoryReservation
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwap
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwappiness
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • oomKillDisable
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌

io.kestra.plugin.scripts.exec.scripts.models.DockerOptions

  • image
    • Type: string
    • Dynamic: ✔️
    • Required: ✔️
    • Min length: 1
  • config
    • Type:
      • string
      • object
    • Dynamic: ✔️
    • Required: ❌
  • cpu
    • Type: Cpu
    • Dynamic: ❌
    • Required: ❌
  • credentials
  • deviceRequests
  • entryPoint
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • extraHosts
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • host
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: Memory
    • Dynamic: ❌
    • Required: ❌
  • networkMode
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • pullPolicy
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: ALWAYS
    • Possible Values:
      • IF_NOT_PRESENT
      • ALWAYS
      • NEVER
  • shmSize
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • user
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • volumes
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.Credentials

  • auth
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • identityToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • password
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registry
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registryToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • username
    • Type: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.DeviceRequest

  • capabilities
    • Type: array
    • SubType: array
    • Dynamic: ❌
    • Required: ❌
  • count
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
  • deviceIds
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • driver
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • options
    • Type: object
    • SubType: string
    • Dynamic: ❌
    • Required: ❌

io.kestra.plugin.singer.models.StreamsConfiguration

  • propertiesPattern
    • Type: array
    • SubType: string
    • Dynamic: ❌
    • Required: ❌
  • replicationKeys
    • Type: string
    • Dynamic: ❌
    • Required: ❌
  • replicationMethod
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Possible Values:
      • FULL_TABLE
      • INCREMENTAL
      • LOG_BASED
  • selected
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
    • Default: true
  • stream
    • Type: string
    • Dynamic: ❌
    • Required: ❌