dbt Athena

dbt-athena (dbt-athena variant)

The dbt-athena utility is an adapter-specific dbt transformer for running SQL-based transformations on data stored in your warehouse. This utility plugin is meant to be used in favor of the dbt-athena transformer plugin type. Note that this plugin can only be run as part of an ELT pipeline with the meltano run command. If you are using meltano elt you should use the transformer plugins. We do recommend migrating to meltano run as the transformer plugin type will be deprecated in a future major Meltano release.

EDK Based Plugin

This utility is based on the Meltano Extension Developer Kit (EDK) which is the preferred way to build and add non-Singer plugins to Meltano Hub. For more information about the EDK, please read this section of the Meltano docs. If you have any feedback or suggestions, add them to the EDK repo.

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Installation and configuration

  1. Add the dbt-athena utility to your project using 
    meltano add
     : 
    2. meltano add utility dbt-athena
  2. Configure the dbt-athena settings using 
    meltano config
     : 
    3. meltano config dbt-athena set --interactive

Next steps

  1. If you're running dbt for the first time in a new environment:
# create a starter dbt_project.yml file, a profiles.yml file, and models directory
meltano invoke dbt-athena:initialize

If you run into any issues, learn how to get help.

Capabilities

This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!

Settings

Settings for dbt itself can be configured through dbt_project.yml as usual, which can be found at transform/dbt_project.yml in your Meltano project.

The dbt-athena settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

You can also list these settings using 

meltano config
with the list subcommand: 

meltano config dbt-athena list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

AWS Access Key ID (aws_access_key_id)

  • Environment variable: DBT_ATHENA_AWS_ACCESS_KEY_ID

Access key ID of the user performing requests. Ex. AKIAIOSFODNN7EXAMPLE


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set aws_access_key_id [value]

AWS Profile Name (aws_profile_name)

  • Environment variable: DBT_ATHENA_AWS_PROFILE_NAME

Profile to use from your AWS shared credentials file. Ex. my-profile


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set aws_profile_name [value]

AWS Secret Access Key (aws_secret_access_key)

  • Environment variable: DBT_ATHENA_AWS_SECRET_ACCESS_KEY

Secret access key of the user performing requests. Ex. wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set aws_secret_access_key [value]

Database (database)

  • Environment variable: DBT_ATHENA_DATABASE

Specify the database (Data catalog) to build models into (lowercase only). Ex. awsdatacatalog


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set database [value]

Debug Query State (debug_query_state)

  • Environment variable: DBT_ATHENA_DEBUG_QUERY_STATE

Flag if debug message with Athena query state is needed. Ex. false


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set debug_query_state [value]

LF Tags Database (lf_tags_database)

  • Environment variable: DBT_ATHENA_LF_TAGS_DATABASE

Default LF tags for new database if it's created by dbt. Ex. tag_key: tag_value


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set lf_tags_database [value]

Number of Boto3 Retries (num_boto3_retries)

  • Environment variable: DBT_ATHENA_NUM_BOTO3_RETRIES

Number of times to retry boto3 requests (e.g. deleting S3 files for materialized tables). Ex. 5


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set num_boto3_retries [value]

Work Group (num_retries)

  • Environment variable: DBT_ATHENA_NUM_RETRIES

Number of times to retry a failing query. Ex. 3


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set num_retries [value]

Poll Interval (poll_interval)

  • Environment variable: DBT_ATHENA_POLL_INTERVAL

Interval in seconds to use for polling the status of query results in Athena. Ex. 5


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set poll_interval [value]

Profiles Directory (profiles_dir)

  • Environment variable: DBT_ATHENA_PROFILES_DIR
  • Default Value: $MELTANO_PROJECT_ROOT/transform/profiles/athena
[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config dbt-athena set profiles_dir [value]

Project Directory (project_dir)

  • Environment variable: DBT_ATHENA_PROJECT_DIR
  • Default Value: $MELTANO_PROJECT_ROOT/transform
[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config dbt-athena set project_dir [value]

Region Name (region_name)

  • Environment variable: DBT_ATHENA_REGION_NAME

AWS region of your Athena instance. Ex. eu-west-1


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set region_name [value]

S3 Data Directory (s3_data_dir)

  • Environment variable: DBT_ATHENA_S3_DATA_DIR

Prefix for storing tables, if different from the connection's s3_staging_dir. Ex. s3://bucket2/dbt/


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set s3_data_dir [value]

S3 Data Naming (s3_data_naming)

  • Environment variable: DBT_ATHENA_S3_DATA_NAMING

How to generate table paths in s3_data_dir. Ex. schema_table_unique


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set s3_data_naming [value]

S3 Staging Directory (s3_staging_dir)

  • Environment variable: DBT_ATHENA_S3_STAGING_DIR

S3 location to store Athena query results and metadata. Ex. s3://bucket/dbt/


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set s3_staging_dir [value]

S3 Temporary Table Directory (s3_tmp_table_dir)

  • Environment variable: DBT_ATHENA_S3_TMP_TABLE_DIR

Prefix for storing temporary tables, if different from the connection's s3_data_dir. Ex. s3://bucket3/dbt/


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set s3_tmp_table_dir [value]

Schema (schema)

  • Environment variable: DBT_ATHENA_SCHEMA

Specify the schema (Athena database) to build models into (lowercase only). Ex. dbt


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set schema [value]

Seed S3 Upload Arguments (seed_s3_upload_args)

  • Environment variable: DBT_ATHENA_SEED_S3_UPLOAD_ARGS

Dictionary containing boto3 ExtraArgs when uploading to S3. Ex. {"ACL": "bucket-owner-full-control"}


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set seed_s3_upload_args [value]

Skip Pre-invoke (skip_pre_invoke)

  • Environment variable: DBT_ATHENA_SKIP_PRE_INVOKE
  • Default Value: false

Whether to skip pre-invoke hooks which automatically run dbt clean and deps


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set skip_pre_invoke [value]

Target Path (target_path)

  • Environment variable: DBT_ATHENA_TARGET_PATH
  • Default Value: $MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target
[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config dbt-athena set target_path [value]

dbt Profile type (type)

  • Environment variable: DBT_ATHENA_TYPE
  • Default Value: athena
[No description provided.]

Configure this setting directly using the following Meltano command:

meltano config dbt-athena set type [value]

Work Group (work_group)

  • Environment variable: DBT_ATHENA_WORK_GROUP

Identifier of Athena workgroup. Ex. my-custom-workgroup


Configure this setting directly using the following Meltano command:

meltano config dbt-athena set work_group [value]

Commands

The dbt-athena utility supports the following commands that can be used with 
meltano invoke
:

build

  • Equivalent to: build

Will run your models, tests, snapshots and seeds in DAG order.

meltano invoke dbt-athena:build [args...]

clean

  • Equivalent to: clean

Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)

meltano invoke dbt-athena:clean [args...]

compile

  • Equivalent to: compile

Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory.

meltano invoke dbt-athena:compile [args...]

debug

  • Equivalent to: debug

Debug your DBT project and warehouse connection.

meltano invoke dbt-athena:debug [args...]

deps

  • Equivalent to: deps

Pull the most recent version of the dependencies listed in packages.yml

meltano invoke dbt-athena:deps [args...]

describe

  • Equivalent to: describe

Describe the

meltano invoke dbt-athena:describe [args...]

docs-generate

  • Equivalent to: docs generate

Generate documentation for your project.

meltano invoke dbt-athena:docs-generate [args...]

docs-serve

  • Equivalent to: docs serve

Serve documentation for your project. Make sure you ran `docs-generate` first.

meltano invoke dbt-athena:docs-serve [args...]

freshness

  • Equivalent to: source freshness

Check the freshness of your source data.

meltano invoke dbt-athena:freshness [args...]

initialize

  • Equivalent to: initialize

Initialize a new dbt project. This will create a dbt_project.yml file, a profiles.yml file, and models directory. 

meltano invoke dbt-athena:initialize [args...]

run

  • Equivalent to: run

Compile SQL and execute against the current target database.

meltano invoke dbt-athena:run [args...]

seed

  • Equivalent to: seed

Load data from csv files into your data warehouse.

meltano invoke dbt-athena:seed [args...]

snapshot

  • Equivalent to: snapshot

Execute snapshots defined in your project.

meltano invoke dbt-athena:snapshot [args...]

test

  • Equivalent to: test

Runs tests on data in deployed models.

meltano invoke dbt-athena:test [args...]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the dbt-athena utility to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the 
#plugins-general
channel.

Install

meltano add utility dbt-athena

Maintenance Status

  • Maintenance Status

Repo

https://github.com/dbt-athena/dbt-athena
  • Stars
  • Forks
  • Last Commit Date
  • Open Issues
  • Open PRs
  • Contributors
  • License

EDK Extension Repo

https://github.com/meltano/dbt-ext

Maintainer

  • dbt Athena

Meltano Stats

  • Total Executions (Last 3 Months)
  • Projects (Last 3 Months)

Keywords

  • meltano_edk