For every star on GitHub, we'll donate $2 to clean up our waterways. Star us now!
This guide will describe how to bring existing dbt code into the your Meltano project. Meltano uses some suggested patterns for organizing your dbt project so it integrates well with core features of Meltano like environments. You can organize your project whatever way you chose but this guide will describe how to import it so it matches the default transformer installation.
As always, we highly recommend git versioning your Meltano project prior to following this guide so you have the ability to roll back and not affect your existing Meltano project configurations.
Add your adapter-specific dbt variant (e.g. dbt-postgres) that can be found on MeltanoHub.
meltano add utility dbt-<adapter_name> # For example meltano add utility dbt-postgres
Next configure your transformer to include database names, connection credentials, etc. See the transform data guide for more details. Or use the interactive config flag to follow prompts.
meltano config dbt-snowflake set --interactive
Once you’ve configured your transformer you should be able to run the following command to test your connection and credentials.
meltano invoke dbt-postgres debug
Note that the
initialize command for a dbt transformer utility creates the expected scaffolding within your Meltano
/transform directory including a
If you have an existing dbt project you can skip running the
initialize because you will already have your own version of these files in your other repo, so we’ll describe how to merge what you have and what Meltano provides and expects.
Meltano expects dbt project files to exist in the default directories listed below.
You can either place your files in the appropriate directories or you can update the given
dbt_project.yml to follow the directory structure of your existing project, if thats preferred.
Meltano’s default dbt project scaffolding comes with a profiles.yml (see dbt profiles docs for details) that is configured to take advantage of the environments feature. This means that you configure dbt using the provided Meltano settings and they get automatically passed to dbt based on what Meltano environment is active. Meltano’s dbt installation comes with pre-configured dbt targets mapped to the default environment names (i.e. dev, staging, prod), avoiding the need to toggle credentials manually and allowing sharing of settings/credentials across plugins.
If you had any configurations in your dbt_project.yml such as definitions of how models are materialized, target databases, schemas, etc. you can directly copy them into your new Meltano dbt_project.yml file.
Again Meltano doesn’t require this structure, any valid dbt project will work, but this is the default recommended structure with some base configurations for a simple integration between Meltano and dbt.