PluginsGoogle CloudTasksBigquery

CreateTable​Create​Table

yaml
type: "io.kestra.plugin.gcp.bigquery.CreateTable"

Create a table

Examples

yaml
id: gcp_bq_create_table
namespace: company.team

tasks:
  - id: create_table
    type: io.kestra.plugin.gcp.bigquery.CreateTable
    projectId: my-project
    dataset: my-dataset
    table: my-table
    tableDefinition:
      type: TABLE
      schema:
        fields:
        - name: id
          type: INT64
        - name: name
          type: STRING
      standardTableDefinition:
        clustering:
        - id
        - name
    friendlyName: new_table

Properties

dataset

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The dataset's user-defined ID.

table

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The table's user-defined ID.

description

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The user-friendly description for the table.

encryptionConfiguration

The encryption configuration.

expirationDuration

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Format: duration

Sets the duration, since now, when this table expires.

If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed.

friendlyName

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The user-friendly name for the table.

impersonatedServiceAccount

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP service account to impersonate.

labels

  • Type: object
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

Return a map for labels applied to the table.

location

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The geographic location where the dataset should reside.

This property is experimental and might be subject to change or removed.

See Dataset Location

projectId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP project ID.

requirePartitionFilter

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌

Return true if a partition filter (that can be used for partition elimination) is required for queries over this table.

retryAuto

  • Type:
  • Dynamic: ❌
  • Required: ❌

retryMessages

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [ "due to concurrent update", "Retrying the job may solve the problem" ]

The messages which would trigger an automatic retry.

Message is tested as a substring of the full message, and is case insensitive.

retryReasons

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [ "rateLimitExceeded", "jobBackendError", "internalError", "jobInternalError" ]

The reasons which would trigger an automatic retry.

scopes

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [ "https://www.googleapis.com/auth/cloud-platform" ]

The GCP scopes to be used.

serviceAccount

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP service account.

tableDefinition

The table definition.

Outputs

creationTime

  • Type: string
  • Required: ❌
  • Format: date-time

datasetId

  • Type: string
  • Required: ❌

definition

description

  • Type: string
  • Required: ❌

encryptionConfiguration

etag

  • Type: string
  • Required: ❌

expirationTime

  • Type: string
  • Required: ❌
  • Format: date-time

friendlyName

  • Type: string
  • Required: ❌

generatedId

  • Type: string
  • Required: ❌

labels

  • Type: object
  • SubType: string
  • Required: ❌

lastModifiedTime

  • Type: string
  • Required: ❌
  • Format: date-time

numBytes

  • Type: integer
  • Required: ❌

numLongTermBytes

  • Type: integer
  • Required: ❌

numRows

  • Type: integer
  • Required: ❌

projectId

  • Type: string
  • Required: ❌

requirePartitionFilter

  • Type: boolean
  • Required: ❌
  • Type: string
  • Required: ❌

table

  • Type: string
  • Required: ❌

Definitions

io.kestra.core.models.tasks.retrys.Constant

  • interval
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Format: duration
  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Default: constant
  • behavior
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: RETRY_FAILED_TASK
    • Possible Values:
      • RETRY_FAILED_TASK
      • CREATE_NEW_EXECUTION
  • maxAttempt
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
    • Minimum: ›= 1
  • maxDuration
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Format: duration
  • warningOnRetry
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
    • Default: false

io.kestra.plugin.gcp.bigquery.models.MaterializedViewDefinition

  • enableRefresh
    • Type: boolean
    • Dynamic: ❌
    • Required: ❓
  • lastRefreshDate
    • Type: string
    • Dynamic: ❌
    • Required: ❓
    • Format: date-time
  • query
    • Type: string
    • Dynamic: ✔️
    • Required: ❓
  • refreshInterval
    • Type: string
    • Dynamic: ❌
    • Required: ❓
    • Format: duration

io.kestra.plugin.gcp.bigquery.models.TableDefinition

io.kestra.plugin.gcp.bigquery.models.TimePartitioning

  • expiration
    • Type: string
    • Dynamic: ❌
    • Required: ❓
    • Format: duration
  • field
    • Type: string
    • Dynamic: ✔️
    • Required: ❓
  • requirePartitionFilter
    • Type: boolean
    • Dynamic: ❌
    • Required: ❓
  • type
    • Type: string
    • Dynamic: ❌
    • Required: ❓
    • Possible Values:
      • DAY
      • HOUR
      • MONTH
      • YEAR

io.kestra.plugin.gcp.bigquery.models.StandardTableDefinition

io.kestra.core.models.tasks.retrys.Exponential

  • interval
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Format: duration
  • maxInterval
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Format: duration
  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Default: exponential
  • behavior
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: RETRY_FAILED_TASK
    • Possible Values:
      • RETRY_FAILED_TASK
      • CREATE_NEW_EXECUTION
  • delayFactor
    • Type: number
    • Dynamic: ❌
    • Required: ❌
  • maxAttempt
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
    • Minimum: ›= 1
  • maxDuration
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Format: duration
  • warningOnRetry
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
    • Default: false

io.kestra.plugin.gcp.bigquery.models.Field

  • description
    • Type: string
    • Dynamic: ✔️
    • Required: ❓
  • mode
    • Type: string
    • Dynamic: ✔️
    • Required: ❓
    • Possible Values:
      • NULLABLE
      • REQUIRED
      • REPEATED
  • name
    • Type: string
    • Dynamic: ✔️
    • Required: ❓
  • policyTags
  • subFields
    • Type: array
    • SubType: Field
    • Dynamic: ✔️
    • Required: ❓
  • type
    • Type: string
    • Dynamic: ❌
    • Required: ❓
    • Possible Values:
      • BOOL
      • INT64
      • FLOAT64
      • NUMERIC
      • BIGNUMERIC
      • STRING
      • BYTES
      • STRUCT
      • ARRAY
      • TIMESTAMP
      • DATE
      • TIME
      • DATETIME
      • GEOGRAPHY
      • JSON
      • INTERVAL
      • RANGE

io.kestra.plugin.gcp.bigquery.models.UserDefinedFunction

  • The type of user defined function.
    • Type: string
    • Dynamic: ❌
    • Required: ❓
    • Possible Values:
      • INLINE
      • FROM_URI
  • Type of UserDefinedFunction
    • Type: string
    • Dynamic: ✔️
    • Required: ❓

io.kestra.plugin.gcp.bigquery.models.ExternalTableDefinition

  • autodetect
    • Type: boolean
    • Dynamic: ❌
    • Required: ❓
  • compression
    • Type: string
    • Dynamic: ✔️
    • Required: ❓
  • formatType
    • Type: string
    • Dynamic: ❌
    • Required: ❓
    • Possible Values:
      • CSV
      • JSON
      • BIGTABLE
      • DATASTORE_BACKUP
      • AVRO
      • GOOGLE_SHEETS
      • PARQUET
      • ORC
  • ignoreUnknownValues
    • Type: boolean
    • Dynamic: ❌
    • Required: ❓
  • maxBadRecords
    • Type: integer
    • Dynamic: ❌
    • Required: ❓
  • sourceUris
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❓

com.google.cloud.bigquery.StandardTableDefinition-StreamingBuffer

  • estimatedBytes
    • Type: integer
    • Dynamic: ❌
    • Required: ❓
  • estimatedRows
    • Type: integer
    • Dynamic: ❌
    • Required: ❓
  • oldestEntryTime
    • Type: integer
    • Dynamic: ❌
    • Required: ❓

io.kestra.plugin.gcp.bigquery.models.RangePartitioning-Range

  • The end of range partitioning.
    • Type: integer
    • Dynamic: ❌
    • Required: ❓
  • The start of range partitioning.
    • Type: integer
    • Dynamic: ❌
    • Required: ❓
  • The width of each interval.
    • Type: integer
    • Dynamic: ❌
    • Required: ❓

io.kestra.core.models.tasks.retrys.Random

  • maxInterval
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Format: duration
  • minInterval
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Format: duration
  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Default: random
  • behavior
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: RETRY_FAILED_TASK
    • Possible Values:
      • RETRY_FAILED_TASK
      • CREATE_NEW_EXECUTION
  • maxAttempt
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
    • Minimum: ›= 1
  • maxDuration
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Format: duration
  • warningOnRetry
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
    • Default: false

io.kestra.plugin.gcp.bigquery.models.PolicyTags

  • The policy tags' names.
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❓

io.kestra.plugin.gcp.bigquery.models.RangePartitioning

  • The range of range partitioning.
  • The range partitioning field.
    • Type: string
    • Dynamic: ✔️
    • Required: ❓

io.kestra.plugin.gcp.bigquery.models.Schema

  • The fields associated with this schema.
    • Type: array
    • SubType: Field
    • Dynamic: ❌
    • Required: ❓

io.kestra.plugin.gcp.bigquery.models.ViewDefinition

  • query
    • Type: string
    • Dynamic: ✔️
    • Required: ❓
  • viewUserDefinedFunctions

io.kestra.plugin.gcp.bigquery.models.EncryptionConfiguration

  • The KMS key name.
    • Type: string
    • Dynamic: ✔️
    • Required: ❓