Skip to main content

Quickstart for dbt Cloud and Amazon Athena

Updated
Amazon
Athena
dbt Cloud
Quickstart
Beginner
Menu
  • 1 Introduction​
  • 2 Getting started​
  • 3 Configure Amazon Athena​
  • 4 Set up security access to Athena​
  • 5 Configure the connection in dbt Cloud​
  • 6 Set up a dbt Cloud managed repository​
  • 7 Initialize your dbt project​ and start developing​
  • 8 Build your first model​
  • 9 Change the way your model is materialized​
  • 10 Delete the example models​
  • 11 Build models on top of other models​
  • 12 Add tests to your models​
  • 13 Document your models​
  • 14 Commit your changes​
  • 15 Deploy dbt​

Introduction​

In this quickstart guide, you'll learn how to use dbt Cloud with Amazon Athena. It will show you how to:

  • Create an S3 bucket for Athena query results.
  • Creat an Athena database.
  • Access sample data in a public dataset.
  • Connect dbt Cloud to Amazon Athena.
  • Take a sample query and turn it into a model in your dbt project. A model in dbt is a select statement.
  • Add tests to your models.
  • Document your models.
  • Schedule a job to run.
Videos for you

You can check out dbt Fundamentals for free if you're interested in course learning with videos.

Prerequisites​​

Related content​

BackNext

Getting started​

For the following guide you can use an existing S3 bucket or create a new one.

Download the following CSV files (the Jaffle Shop sample data) and upload them to your S3 bucket:

BackNext

Configure Amazon Athena​

  1. Log into your AWS account and navigate to the Athena console.
    • If this is your first time in the Athena console (in your current AWS Region), click Explore the query editor to open the query editor. Otherwise, Athena opens automatically in the query editor.
  2. Open Settings and find the Location of query result box field.
    1. Enter the path of the S3 bucket (prefix it with s3://).
    2. Navigate to Browse S3, select the S3 bucket you created, and click Choose.
  3. Save these settings.
  4. In the query editor, create a database by running create database YOUR_DATABASE_NAME.
  5. To make the database you created the one you write into, select it from the Database list on the left side menu.
  6. Access the Jaffle Shop data in the S3 bucket using one of these options:
    1. Manually create the tables.
    2. Create a glue crawler to recreate the data as external tables (recommended).
  7. Once the tables have been created, you will able to SELECT from them.
BackNext

Set up security access to Athena​

To setup the security access for Athena, determine which access method you want to use:

  • Obtain aws_access_key_id and aws_secret_access_key (recommended)
  • Obtain an AWS credentials file.

To obtain your aws_access_key_id and aws_secret_access_key:

  1. Open the AWS Console.
  2. Click on your username near the top right and click Security Credentials.
  3. Click on Users in the sidebar.
  4. Click on your username (or the name of the user for whom to create the key).
  5. Click on the Security Credentials tab.
  6. Click Create Access Key.
  7. Click Show User Security Credentials and

Save the aws_access_key_id and aws_secret_access_key for a future step.

AWS credentials file​

To obtain your AWS credentials file:

  1. Follow the instructions for configuring the credentials file usin the AWS CLI
  2. Locate the ~/.aws/credentials file on your computer
    1. Windows: %USERPROFILE%\.aws\credentials
    2. Mac/Linux: ~/.aws/credentials

Retrieve the aws_access_key_id and aws_secret_access_key from the ~/.aws/credentials file for a future step.

BackNext

Configure the connection in dbt Cloud​

To configure the Athena connection in dbt Cloud:

  1. Click your account name on the left-side menu and click Account settings.
  2. Click Connections and click New connection.
  3. Click Athena and fill out the required fields (and any optional fields).
    1. AWS region name — The AWS region of your environment.
    2. Database (catalog) — Enter the database name created in earlier steps (lowercase only).
    3. AWS S3 staging directory — Enter the S3 bucket created in earlier steps.
  4. Click Save

Configure your environment​

To configure the Athena credentials in your environment:

  1. Click Deploy on the left-side menu and click Environments.
  2. Click Create environment and fill out the General settings.
    • Your dbt version must be set to Versionless to use the Athena connection.
  3. Select the Athena connection from the Connection dropdown.
  4. Fill out the aws_access_key and aws_access_id recorded in previous steps, as well as the Schema to write to.
  5. Click Test connection and once it succeeds, Save the environment.

Repeat the process to create a development environment.

BackNext

Set up a dbt Cloud managed repository​

When you develop in dbt Cloud, you can leverage Git to version control your code.

To connect to a repository, you can either set up a dbt Cloud-hosted managed repository or directly connect to a supported git provider. Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and continuous integration.

To set up a managed repository:

  1. Under "Setup a repository", select Managed.
  2. Type a name for your repo such as bbaggins-dbt-quickstart
  3. Click Create. It will take a few seconds for your repository to be created and imported.
  4. Once you see the "Successfully imported repository," click Continue.
BackNext

Initialize your dbt project​ and start developing​

Now that you have a repository configured, you can initialize your project and start development in dbt Cloud:

  1. Click Start developing in the IDE. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse.
  2. Above the file tree to the left, click Initialize dbt project. This builds out your folder structure with example models.
  3. Make your initial commit by clicking Commit and sync. Use the commit message initial commit and click Commit. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code.
  4. You can now directly query data from your warehouse and execute dbt run. You can try this out now:
    • Click + Create new file, add this query to the new file, and click Save as to save the new file: 
      select * from jaffle_shop.customers
    • In the command line bar at the bottom, enter dbt run and click Enter. You should see a dbt run succeeded message.
BackNext

Build your first model​

You have two options for working with files in the dbt Cloud IDE:

  • Create a new branch (recommended) — Create a new branch to edit and commit your changes. Navigate to Version Control on the left sidebar and click Create branch.
  • Edit in the protected primary branch — If you prefer to edit, format, or lint files and execute dbt commands directly in your primary git branch. The dbt Cloud IDE prevents commits to the protected branch, so you will be prompted to commit your changes to a new branch.

Name the new branch add-customers-model.

  1. Click the ... next to the models directory, then select Create file.
  2. Name the file customers.sql, then click Create.
  3. Copy the following query into the file and click Save.
with customers as (

    select
        id as customer_id,
        first_name,
        last_name

    from jaffle_shop.customers

),

orders as (

    select
        id as order_id,
        user_id as customer_id,
        order_date,
        status

    from jaffle_shop.orders

),

customer_orders as (

    select
        customer_id,

        min(order_date) as first_order_date,
        max(order_date) as most_recent_order_date,
        count(order_id) as number_of_orders

    from orders

    group by 1

),

final as (

    select
        customers.customer_id,
        customers.first_name,
        customers.last_name,
        customer_orders.first_order_date,
        customer_orders.most_recent_order_date,
        coalesce(customer_orders.number_of_orders, 0) as number_of_orders

    from customers

    left join customer_orders using (customer_id)

)

select * from final
  1. Enter dbt run in the command prompt at the bottom of the screen. You should get a successful run and see the three models.

Later, you can connect your business intelligence (BI) tools to these views and tables so they only read cleaned up data rather than raw data in your BI tool.

FAQs​

How can I see the SQL that dbt is running?
How did dbt choose which schema to build my models in?
Do I need to create my target schema before running dbt?
If I rerun dbt, will there be any downtime as models are rebuilt?
What happens if the SQL in my query is bad or I get a database error?
BackNext

Change the way your model is materialized​

One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes.

By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization.

  1. Edit your dbt_project.yml file.

    • Update your project name to:

      dbt_project.yml
      name: 'jaffle_shop'

    • Configure jaffle_shop so everything in it will be materialized as a table; and configure example so everything in it will be materialized as a view. Update your models config block to:

      dbt_project.yml
      models:
        jaffle_shop:
          +materialized: table
          example:
            +materialized: view

    • Click Save.

  2. Enter the dbt run command. Your customers model should now be built as a table!

    info

    To do this, dbt had to first run a drop view statement (or API call on BigQuery), then a create table as statement.

  3. Edit models/customers.sql to override the dbt_project.yml for the customers model only by adding the following snippet to the top, and click Save:

    models/customers.sql
    {{
      config(
        materialized='view'
      )
    }}

    with customers as (

        select
            id as customer_id
            ...

    )

  4. Enter the dbt run command. Your model, customers, should now build as a view.

    • BigQuery users need to run dbt run --full-refresh instead of dbt run to full apply materialization changes.

  5. Enter the dbt run --full-refresh command for this to take effect in your warehouse.

FAQs​

What materializations are available in dbt?
Which materialization should I use for my model?
What model configurations exist?
BackNext

Delete the example models​

You can now delete the files that dbt created when you initialized the project:

  1. Delete the models/example/ directory.

  2. Delete the example: key from your dbt_project.yml file, and any configurations that are listed under it.

    dbt_project.yml
    # before
    models:
      jaffle_shop:
        +materialized: table
        example:
          +materialized: view
    dbt_project.yml
    # after
    models:
      jaffle_shop:
        +materialized: table

  3. Save your changes.

FAQs​

How do I remove deleted models from my data warehouse?
I got an "unused model configurations" error message, what does this mean?
BackNext

Build models on top of other models​

As a best practice in SQL, you should separate logic that cleans up your data from logic that transforms your data. You have already started doing this in the existing query by using common table expressions (CTEs).

Now you can experiment by separating the logic out into separate models and using the ref function to build models on top of other models:

The DAG we want for our dbt projectThe DAG we want for our dbt project

  1. Create a new SQL file, models/stg_customers.sql, with the SQL from the customers CTE in our original query.

  2. Create a second new SQL file, models/stg_orders.sql, with the SQL from the orders CTE in our original query.

    models/stg_customers.sql
    select
        id as customer_id,
        first_name,
        last_name

    from jaffle_shop.customers
    models/stg_orders.sql
    select
        id as order_id,
        user_id as customer_id,
        order_date,
        status

    from jaffle_shop.orders

  3. Edit the SQL in your models/customers.sql file as follows:

    models/customers.sql
    with customers as (

        select * from {{ ref('stg_customers') }}

    ),

    orders as (

        select * from {{ ref('stg_orders') }}

    ),

    customer_orders as (

        select
            customer_id,

            min(order_date) as first_order_date,
            max(order_date) as most_recent_order_date,
            count(order_id) as number_of_orders

        from orders

        group by 1

    ),

    final as (

        select
            customers.customer_id,
            customers.first_name,
            customers.last_name,
            customer_orders.first_order_date,
            customer_orders.most_recent_order_date,
            coalesce(customer_orders.number_of_orders, 0) as number_of_orders

        from customers

        left join customer_orders using (customer_id)

    )

    select * from final

  4. Execute dbt run.

    This time, when you performed a dbt run, separate views/tables were created for stg_customers, stg_orders and customers. dbt inferred the order to run these models. Because customers depends on stg_customers and stg_orders, dbt builds customers last. You do not need to explicitly define these dependencies.

FAQs​

How do I run one model at a time?
Do ref-able resource names need to be unique?
As I create more models, how should I keep my project organized? What should I name my models?
BackNext

Add tests to your models​

Adding tests to a project helps validate that your models are working correctly.

To add tests to your project:

  1. Create a new YAML file in the models directory, named models/schema.yml

  2. Add the following contents to the file:

    models/schema.yml
    version: 2

    models:
      - name: customers
        columns:
          - name: customer_id
            tests:
              - unique
              - not_null

      - name: stg_customers
        columns:
          - name: customer_id
            tests:
              - unique
              - not_null

      - name: stg_orders
        columns:
          - name: order_id
            tests:
              - unique
              - not_null
          - name: status
            tests:
              - accepted_values:
                  values: ['placed', 'shipped', 'completed', 'return_pending', 'returned']
          - name: customer_id
            tests:
              - not_null
              - relationships:
                  to: ref('stg_customers')
                  field: customer_id

  3. Run dbt test, and confirm that all your tests passed.

When you run dbt test, dbt iterates through your YAML files, and constructs a query for each test. Each query will return the number of records that fail the test. If this number is 0, then the test is successful.

FAQs​

What tests are available for me to use in dbt? Can I add my own custom tests?
How do I test one model at a time?
One of my tests failed, how can I debug it?
Does my test file need to be named `schema.yml`?
Why do model and source yml files always start with `version: 2`?
What tests should I add to my project?
When should I run my tests?
BackNext

Document your models​

Adding documentation to your project allows you to describe your models in rich detail, and share that information with your team. Here, we're going to add some basic documentation to our project.

  1. Update your models/schema.yml file to include some descriptions, such as those below.

    models/schema.yml
    version: 2

    models:
      - name: customers
        description: One record per customer
        columns:
          - name: customer_id
            description: Primary key
            tests:
              - unique
              - not_null
          - name: first_order_date
            description: NULL when a customer has not yet placed an order.

      - name: stg_customers
        description: This model cleans up customer data
        columns:
          - name: customer_id
            description: Primary key
            tests:
              - unique
              - not_null

      - name: stg_orders
        description: This model cleans up order data
        columns:
          - name: order_id
            description: Primary key
            tests:
              - unique
              - not_null
          - name: status
            tests:
              - accepted_values:
                  values: ['placed', 'shipped', 'completed', 'return_pending', 'returned']
          - name: customer_id
            tests:
              - not_null
              - relationships:
                  to: ref('stg_customers')
                  field: customer_id

  2. Run dbt docs generate to generate the documentation for your project. dbt introspects your project and your warehouse to generate a JSONJSON (JavaScript Object Notation) is a minimal format for semi-structured data used to capture relationships between fields and values. file with rich documentation about your project.

  1. Click the book icon in the Develop interface to launch documentation in a new tab.

FAQs​

How do I write long-form explanations in my descriptions?
How do I access documentation in dbt Explorer?
BackNext

Commit your changes​

Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code.

If you edited directly in the protected primary branch:

  1. Click the Commit and sync git button. This action prepares your changes for commit.
  2. A modal titled Commit to a new branch will appear.
  3. In the modal window, name your new branch add-customers-model. This branches off from your primary branch with your new changes.
  4. Add a commit message, such as "Add customers model, tests, docs" and and commit your changes.
  5. Click Merge this branch to main to add these changes to the main branch on your repo.

If you created a new branch before editing:

  1. Since you already branched out of the primary protected branch, go to Version Control on the left.
  2. Click Commit and sync to add a message.
  3. Add a commit message, such as "Add customers model, tests, docs."
  4. Click Merge this branch to main to add these changes to the main branch on your repo.
BackNext

Deploy dbt​

Use dbt Cloud's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps.

Create a deployment environment​

  1. In the upper left, select Deploy, then click Environments.
  2. Click Create Environment.
  3. In the Name field, write the name of your deployment environment. For example, "Production."
  4. In the dbt Version field, select the latest version from the dropdown.
  5. Under Deployment connection, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema".
  6. Click Save.

Create and run a job​

Jobs are a set of dbt commands that you want to run on a schedule. For example, dbt build.

As the jaffle_shop business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the customers model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job.

  1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select Deploy in the upper left, then click Jobs.
  2. Click Create one and provide a name, for example, "Production run", and link to the Environment you just created.
  3. Scroll down to the Execution Settings section.
  4. Under Commands, add this command as part of your job if you don't see it:
    • dbt build
  5. Select the Generate docs on run checkbox to automatically generate updated project docs each time your job runs.
  6. For this exercise, do not set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as deploying a project.
  7. Select Save, then click Run now to run your job.
  8. Click the run and watch its progress under "Run history."
  9. Once the run is complete, click View Documentation to see the docs for your project.

Congratulations 🎉! You've just deployed your first dbt project!

FAQs​

What happens if one of my runs fails?
BackNext
0