PluginsGoogle CloudTasksBigquery

CopyPartitions​Copy​Partitions

yaml
type: "io.kestra.plugin.gcp.bigquery.CopyPartitions"

Copy partitions between interval to another table

Examples

yaml
id: gcp_bq_copy_partitions
namespace: company.team

tasks:
  - id: copy_partitions
    type: io.kestra.plugin.gcp.bigquery.CopyPartitions
    projectId: my-project
    dataset: my-dataset
    table: my-table
    destinationTable: my-dest-table
    partitionType: DAY
    from: "{{ now() | dateAdd(-30, 'DAYS') }}"
    to: "{{ now() | dateAdd(-7, 'DAYS') }}"

Properties

dataset

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The dataset's user-defined ID.

from

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The inclusive starting date or integer.

If the partition :

  • is a numeric range, must be a valid integer
  • is a date, must a valid datetime like

partitionType

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Possible Values:
    • DAY
    • HOUR
    • MONTH
    • YEAR
    • RANGE

The partition type of the table

table

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The table's user-defined ID.

to

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The inclusive ending date or integer.

If the partition :

  • is a numeric range, must be a valid integer
  • is a date, must a valid datetime like

createDisposition

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Possible Values:
    • CREATE_IF_NEEDED
    • CREATE_NEVER

Whether the job is allowed to create tables.

destinationTable

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The table where to put query results.

If not provided, a new table is created.

dryRun

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌
  • Default: false

Whether the job has to be dry run or not.

A valid query will mostly return an empty response with some processing statistics, while an invalid query will return the same error as it would if it were an actual run.

impersonatedServiceAccount

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP service account to impersonate.

jobTimeout

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Format: duration

Job timeout.

If this time limit is exceeded, BigQuery may attempt to terminate the job.

labels

  • Type: object
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

The labels associated with this job.

You can use these to organize and group your jobs. Label keys and values can be no longer than 63 characters, can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. Label values are optional. Label keys must start with a letter and each label in the list must have a different key.

location

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The geographic location where the dataset should reside.

This property is experimental and might be subject to change or removed.

See Dataset Location

projectId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP project ID.

retryAuto

  • Type:
  • Dynamic: ❌
  • Required: ❌

retryMessages

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [ "due to concurrent update", "Retrying the job may solve the problem" ]

The messages which would trigger an automatic retry.

Message is tested as a substring of the full message, and is case insensitive.

retryReasons

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [ "rateLimitExceeded", "jobBackendError", "internalError", "jobInternalError" ]

The reasons which would trigger an automatic retry.

scopes

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [ "https://www.googleapis.com/auth/cloud-platform" ]

The GCP scopes to be used.

serviceAccount

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP service account.

writeDisposition

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Possible Values:
    • WRITE_TRUNCATE
    • WRITE_APPEND
    • WRITE_EMPTY

The action that should occur if the destination table already exists.

Outputs

datasetId

  • Type: string
  • Required: ❌

jobId

  • Type: string
  • Required: ❌

partitions

  • Type: array
  • SubType: string
  • Required: ❌

projectId

  • Type: string
  • Required: ❌

table

  • Type: string
  • Required: ❌

Definitions

io.kestra.core.models.tasks.retrys.Constant

  • interval
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Format: duration
  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Default: constant
  • behavior
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: RETRY_FAILED_TASK
    • Possible Values:
      • RETRY_FAILED_TASK
      • CREATE_NEW_EXECUTION
  • maxAttempt
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
    • Minimum: ›= 1
  • maxDuration
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Format: duration
  • warningOnRetry
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
    • Default: false

io.kestra.core.models.tasks.retrys.Random

  • maxInterval
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Format: duration
  • minInterval
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Format: duration
  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Default: random
  • behavior
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: RETRY_FAILED_TASK
    • Possible Values:
      • RETRY_FAILED_TASK
      • CREATE_NEW_EXECUTION
  • maxAttempt
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
    • Minimum: ›= 1
  • maxDuration
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Format: duration
  • warningOnRetry
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
    • Default: false

io.kestra.core.models.tasks.retrys.Exponential

  • interval
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Format: duration
  • maxInterval
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Format: duration
  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Default: exponential
  • behavior
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: RETRY_FAILED_TASK
    • Possible Values:
      • RETRY_FAILED_TASK
      • CREATE_NEW_EXECUTION
  • delayFactor
    • Type: number
    • Dynamic: ❌
    • Required: ❌
  • maxAttempt
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
    • Minimum: ›= 1
  • maxDuration
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Format: duration
  • warningOnRetry
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
    • Default: false