TableMetadata

type: "io.kestra.plugin.gcp.bigquery.TableMetadata"

Get table metadata.

Examples

id: gcp_bq_table_metadata
namespace: company.team

tasks:
  - id: table_metadata
    type: io.kestra.plugin.gcp.bigquery.TableMetadata
    projectId: my-project
    dataset: my-dataset
    table: my-table

Properties

dataset

• Type: string
• Dynamic: ✔️
• Required: ✔️

The dataset's user-defined ID.

table

• Type: string
• Dynamic: ✔️
• Required: ✔️

The table's user-defined ID.

ifNotExists

• Type: string
• Dynamic: ❌
• Required: ❌
• Default: ERROR
• Possible Values:
  • ERROR
  • SKIP

Policy to apply if a table don't exists.

If the policy is SKIP, the output will contain only null value, otherwise an error is raised.

location

• Type: string
• Dynamic: ✔️
• Required: ❌

The geographic location where the dataset should reside.

This property is experimental and might be subject to change or removed.

See Dataset Location

projectId

• Type: string
• Dynamic: ✔️
• Required: ❌

The GCP project ID.

retryAuto

• Type:
  • Constant
  • Exponential
  • Random
• Dynamic: ❌
• Required: ❌

retryMessages

• Type: array
• SubType: string
• Dynamic: ✔️
• Required: ❌
• Default: [due to concurrent update, Retrying the job may solve the problem]

The messages which would trigger an automatic retry.

Message is tested as a substring of the full message, and is case insensitive.

retryReasons

• Type: array
• SubType: string
• Dynamic: ✔️
• Required: ❌
• Default: [rateLimitExceeded, jobBackendError, internalError, jobInternalError]

The reasons which would trigger an automatic retry.

scopes

• Type: array
• SubType: string
• Dynamic: ✔️
• Required: ❌
• Default: [https://www.googleapis.com/auth/cloud-platform]

The GCP scopes to be used.

serviceAccount

• Type: string
• Dynamic: ✔️
• Required: ❌

The GCP service account key.

Outputs

creationTime

• Type: string
• Required: ❌
• Format: date-time

The time when this table was created.

datasetId

• Type: string
• Required: ❌

The dataset's ID.

definition

• Type: TableDefinition
• Required: ❌

The table definition.

description

• Type: string
• Required: ❌

The user-friendly description for the table.

encryptionConfiguration

• Type: EncryptionConfiguration
• Required: ❌

The encryption configuration.

etag

• Type: string
• Required: ❌

The hash of the table resource.

expirationTime

• Type: string
• Required: ❌
• Format: date-time

Returns the time when this table expires.

If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed.

friendlyName

• Type: string
• Required: ❌

The user-friendly name for the table.

generatedId

• Type: string
• Required: ❌

The service-generated id for the table.

labels

• Type: object
• SubType: string
• Required: ❌

Return a map for labels applied to the table.

lastModifiedTime

• Type: string
• Required: ❌
• Format: date-time

The time when this table was last modified.

numBytes

• Type: integer
• Required: ❌

The size of this table in bytes.

numLongTermBytes

• Type: integer
• Required: ❌

The number of bytes considered "long-term storage" for reduced billing purposes.

numRows

• Type: integer
• Required: ❌

The number of rows of data in this table.

projectId

• Type: string
• Required: ❌

The project's ID.

requirePartitionFilter

• Type: boolean
• Required: ❌

Return true if a partition filter (that can be used for partition elimination) is required for queries over this table.

selfLink

• Type: string
• Required: ❌

The URL that can be used to access the resource again. The returned URL can be used for get or update requests.

table

• Type: string
• Required: ❌

The table name.

Definitions

io.kestra.core.models.tasks.retrys.Constant

Properties

interval

• Type: string
• Dynamic: ❓
• Required: ✔️
• Format: duration

type

• Type: string
• Dynamic: ❓
• Required: ✔️
• Default: constant

behavior

• Type: string
• Dynamic: ❓
• Required: ❌
• Default: RETRY_FAILED_TASK
• Possible Values:
  • RETRY_FAILED_TASK
  • CREATE_NEW_EXECUTION

maxAttempt

• Type: integer
• Dynamic: ❓
• Required: ❌
• Minimum: >= 1

maxDuration

• Type: string
• Dynamic: ❓
• Required: ❌
• Format: duration

warningOnRetry

• Type: boolean
• Dynamic: ❓
• Required: ❌
• Default: false

io.kestra.plugin.gcp.bigquery.models.MaterializedViewDefinition

Properties

enableRefresh

• Type: boolean
• Dynamic: ❌
• Required: ❓

Whether automatic refresh is enabled for the materialized view when the base table is updated.

lastRefreshDate

• Type: string
• Dynamic: ❓
• Required: ❓
• Format: date-time

Date when this materialized view was last modified.

query

• Type: string
• Dynamic: ✔️
• Required: ❓

The query whose result is persisted.

refreshInterval

• Type: string
• Dynamic: ❌
• Required: ❓
• Format: duration

The maximum frequency at which this materialized view will be refreshed.

io.kestra.plugin.gcp.bigquery.models.TableDefinition

Properties

externalTableDefinition

• Type: ExternalTableDefinition
• Dynamic: ❓
• Required: ❓

The external table definition if the type is EXTERNAL.

materializedViewDefinition

• Type: MaterializedViewDefinition
• Dynamic: ❓
• Required: ❓

The materialized view definition if the type is MATERIALIZED_VIEW.

schema

• Type: Schema
• Dynamic: ❓
• Required: ❓

The table's schema.

standardTableDefinition

• Type: StandardTableDefinition
• Dynamic: ❓
• Required: ❓

The table definition if the type is TABLE.

type

• Type: string
• Dynamic: ❓
• Required: ❓
• Possible Values:
  • TABLE
  • VIEW
  • MATERIALIZED_VIEW
  • EXTERNAL
  • MODEL

The table's type.

viewDefinition

• Type: ViewDefinition
• Dynamic: ❓
• Required: ❓

The view definition if the type is VIEW.

io.kestra.plugin.gcp.bigquery.models.TimePartitioning

Properties

expiration

• Type: string
• Dynamic: ❌
• Required: ❓
• Format: duration

The number of milliseconds for which to keep the storage for a partition. When expired, the storage for the partition is reclaimed. If null, the partition does not expire.

field

• Type: string
• Dynamic: ✔️
• Required: ❓

If not set, the table is partitioned by pseudo column '_PARTITIONTIME'; if set, the table is partitioned by this field.

requirePartitionFilter

• Type: boolean
• Dynamic: ❌
• Required: ❓

If set to true, queries over this table require a partition filter (that can be used for partition elimination) to be specified.

type

• Type: string
• Dynamic: ❌
• Required: ❓
• Possible Values:
  • DAY
  • HOUR
  • MONTH
  • YEAR

The time partitioning type.

io.kestra.plugin.gcp.bigquery.models.StandardTableDefinition

Properties

clustering

• Type: array
• SubType: string
• Dynamic: ❓
• Required: ❓

Returns the clustering configuration for this table. If {@code null}, the table is not clustered.

rangePartitioning

• Type: RangePartitioning
• Dynamic: ❓
• Required: ❓

Returns the range partitioning configuration for this table. If {@code null}, the table is not range-partitioned.

streamingBuffer

• Type: StandardTableDefinition-StreamingBuffer
• Dynamic: ❓
• Required: ❓

Returns information on the table's streaming buffer, if exists. Returns {@code null} if no streaming buffer exists.

timePartitioning

• Type: TimePartitioning
• Dynamic: ❓
• Required: ❓

Returns the time partitioning configuration for this table. If {@code null}, the table is not time-partitioned.

io.kestra.core.models.tasks.retrys.Exponential

Properties

interval

• Type: string
• Dynamic: ❓
• Required: ✔️
• Format: duration

maxInterval

• Type: string
• Dynamic: ❓
• Required: ✔️
• Format: duration

type

• Type: string
• Dynamic: ❓
• Required: ✔️
• Default: exponential

behavior

• Type: string
• Dynamic: ❓
• Required: ❌
• Default: RETRY_FAILED_TASK
• Possible Values:
  • RETRY_FAILED_TASK
  • CREATE_NEW_EXECUTION

delayFactor

• Type: number
• Dynamic: ❓
• Required: ❌

maxAttempt

• Type: integer
• Dynamic: ❓
• Required: ❌
• Minimum: >= 1

maxDuration

• Type: string
• Dynamic: ❓
• Required: ❌
• Format: duration

warningOnRetry

• Type: boolean
• Dynamic: ❓
• Required: ❌
• Default: false

io.kestra.plugin.gcp.bigquery.models.Field

Properties

description

• Type: string
• Dynamic: ✔️
• Required: ❓

The field description.

mode

• Type: string
• Dynamic: ✔️
• Required: ❓
• Possible Values:
  • NULLABLE
  • REQUIRED
  • REPEATED

The field mode.

By default, Field.Mode.NULLABLE is used.

name

• Type: string
• Dynamic: ✔️
• Required: ❓

The field name.

policyTags

• Type: PolicyTags
• Dynamic: ✔️
• Required: ❓

The policy tags for the field.

subFields

• Type: array
• SubType: Field
• Dynamic: ✔️
• Required: ❓

The list of sub-fields if type is a LegacySQLType.RECORD. Returns null otherwise.

type

• Type: string
• Dynamic: ❌
• Required: ❓
• Possible Values:
  • BOOL
  • INT64
  • FLOAT64
  • NUMERIC
  • BIGNUMERIC
  • STRING
  • BYTES
  • STRUCT
  • ARRAY
  • TIMESTAMP
  • DATE
  • TIME
  • DATETIME
  • GEOGRAPHY
  • JSON
  • INTERVAL
  • RANGE

The field type.

io.kestra.plugin.gcp.bigquery.models.UserDefinedFunction

Properties

The type of user defined function.

• Type: string
• Dynamic: ❌
• Required: ❓
• Possible Values:
  • INLINE
  • FROM_URI

Type of UserDefinedFunction

• Type: string
• Dynamic: ✔️
• Required: ❓

If type is UserDefinedFunction.Type.INLINE, this method returns a code blob. If type is UserDefinedFunction.Type.FROM_URI, the method returns a Google Cloud Storage URI (e.g. gs://bucket/path).

io.kestra.plugin.gcp.bigquery.models.ExternalTableDefinition

Properties

autodetect

• Type: boolean
• Dynamic: ❓
• Required: ❓

Whether automatic detection of schema and format options should be performed.

compression

• Type: string
• Dynamic: ✔️
• Required: ❓

The compression type of the data source.

formatType

• Type: string
• Dynamic: ❓
• Required: ❓
• Possible Values:
  • CSV
  • JSON
  • BIGTABLE
  • DATASTORE_BACKUP
  • AVRO
  • GOOGLE_SHEETS
  • PARQUET
  • ORC

The source format, and possibly some parsing options, of the external data.

ignoreUnknownValues

• Type: boolean
• Dynamic: ❓
• Required: ❓

Whether BigQuery should allow extra values that are not represented in the table schema.

If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result.

maxBadRecords

• Type: integer
• Dynamic: ❓
• Required: ❓

The maximum number of bad records that BigQuery can ignore when reading data.

If the number of bad records exceeds this value, an invalid error is returned in the job result.

sourceUris

• Type: array
• SubType: string
• Dynamic: ✔️
• Required: ❓

The fully-qualified URIs that point to your data in Google Cloud Storage.

Each URI can

• contain one '*' wildcard character that must come after the bucket's name. Size limits related
• to load jobs apply to external data sources, plus an additional limit of 10 GB maximum size
• across all URIs.

com.google.cloud.bigquery.StandardTableDefinition-StreamingBuffer

Properties

estimatedBytes

• Type: integer
• Dynamic: ❓
• Required: ❓

estimatedRows

• Type: integer
• Dynamic: ❓
• Required: ❓

oldestEntryTime

• Type: integer
• Dynamic: ❓
• Required: ❓

io.kestra.plugin.gcp.bigquery.models.RangePartitioning-Range

Properties

The end of range partitioning.

• Type: integer
• Dynamic: ❌
• Required: ❓

The start of range partitioning.

• Type: integer
• Dynamic: ❌
• Required: ❓

The width of each interval.

• Type: integer
• Dynamic: ❌
• Required: ❓

io.kestra.core.models.tasks.retrys.Random

Properties

maxInterval

• Type: string
• Dynamic: ❓
• Required: ✔️
• Format: duration

minInterval

• Type: string
• Dynamic: ❓
• Required: ✔️
• Format: duration

type

• Type: string
• Dynamic: ❓
• Required: ✔️
• Default: random

behavior

• Type: string
• Dynamic: ❓
• Required: ❌
• Default: RETRY_FAILED_TASK
• Possible Values:
  • RETRY_FAILED_TASK
  • CREATE_NEW_EXECUTION

maxAttempt

• Type: integer
• Dynamic: ❓
• Required: ❌
• Minimum: >= 1

maxDuration

• Type: string
• Dynamic: ❓
• Required: ❌
• Format: duration

warningOnRetry

• Type: boolean
• Dynamic: ❓
• Required: ❌
• Default: false

io.kestra.plugin.gcp.bigquery.models.PolicyTags

Properties

The policy tags' names.

• Type: array
• SubType: string
• Dynamic: ✔️
• Required: ❓

io.kestra.plugin.gcp.bigquery.models.RangePartitioning

Properties

The range of range partitioning.

• Type: RangePartitioning-Range
• Dynamic: ❓
• Required: ❓

The range partitioning field.

• Type: string
• Dynamic: ✔️
• Required: ❓

io.kestra.plugin.gcp.bigquery.models.Schema

Properties

The fields associated with this schema.

• Type: array
• SubType: Field
• Dynamic: ❓
• Required: ❓

io.kestra.plugin.gcp.bigquery.models.ViewDefinition

Properties

query

• Type: string
• Dynamic: ✔️
• Required: ❓

The query whose result is persisted.

viewUserDefinedFunctions

• Type: array
• SubType: UserDefinedFunction
• Dynamic: ❓
• Required: ❓

User defined functions that can be used by query. Returns {@code null} if not set.

io.kestra.plugin.gcp.bigquery.models.EncryptionConfiguration

Properties

The KMS key name.

• Type: string
• Dynamic: ✔️
• Required: ❓