Skip to main content

Meltano 2.0 Migration Guide

Note: This document is still a work in progress. Expect further changes, coming soon.

The following list includes all recommended migration tasks as well as breaking changes in Meltano version 2.0.

Migrate to an Adapter-Specific dbt Transformer​

If you previously used dbt or dbt-<adapter> (available adapters documentation) Transformer, we recommend migrating to an adapter specific utility plugin.

Install dbt​

This is easy to do! Following the instructions from above to discover and install your chosen adapter:

# install adapter-specific dbt, e.g. for snowflake
meltano add utility dbt-snowflake

Update your dbt_project.yml​

Installation of a new Transformer will introduce two important files to your transform/ directory:

  • A new profiles.yml file in transform/profiles/<adapter name>/profiles.yml
  • A new dbt_project.yml file in transform/dbt_project (<adapter name>).yml

The new profiles.yml will only be used by adapter-specific dbt executions (e.g. dbt-snowflake), and can be customized to meet your requirements. Your existing profiles.yml will remain in use by your existing dbt Transformer plugin (via elt and invoke).

It is likely that the new dbt_project (<adapter name>).yml will contain changes from your previous dbt_project.yml file, especially if you haven't already upgraded to dbt v1.0. To complete your migration, consolidate dbt_project.yml and dbt_project (<adapter name>).yml into a single file called dbt_project.yml. As this project file will be used by both dbt and dbt-<adapter> Transformer plugins by default, you must ensure you are running an up-to-date installation of plugin dbt if you intend to use both adapter-specific and legacy dbt installs together (not recommended).

If you make use of Transform plugins, these will continue to work as regular dbt packages. However adding new Transform plugins will currently (tracking at #3304) re-add the legacy dbt Transformer plugin. To avoid this we recommend adding Transforms as regular packages directly via dbt as per the dbt Packages documentation.

Remove the dbt Transformer plugin and associated files​

To remove the legacy dbt Transformer plugin, run:

# remove the transformer `dbt`
meltano remove transformer dbt

# remove the file bundle `dbt`
meltano remove files dbt

Removing a file bundle does not remove any files from your transform/ directory. Manually remove transform/profile/profiles.yml to complete clean-up (as adapter-specific installs come with their own profiles.yml in transform/profiles/<adapter name>/profiles.yml).

Migrate from orchestrators to utilities​

If you have been using Meltano orchestrators to schedule your ELT jobs, we recommend migrating to the new utilities:

Removed​

model and dashboard plugin types​

These plugin types provided very basic BI capabilities using Meltano UI. However there are already great 3rd party open source BI solutions in this space, such as the newly added Superset plugin. Meltano model and dashboard plugins have been removed in favour of existing and future 3rd party tools for the same purpose.

transform support in meltano elt​

Meltano 2.0 continues to support extract-load (EL) operations with meltano elt. However, for EL+T operations which also need to transform data, please use meltano run.

transform support in Meltano schedules​

Meltano 2.0 continues to support extract-load (EL) operations in schedules. However, for EL+T operations which also need to transform data, please use the new meltano job add command to create a job definition and then specify the new job name in your schedule.

env_aliases in Plugin config​

As part of our effort to streamline the configuration experience in Meltano, we are deprecating the env_aliases attribute of plugin definitions. Previously env_aliases provided two functions:

  1. Sourcing setting values from the terminal by a name other than the default environment variable (of the form <PLUGIN_NAME>_<SETTING_NAME>).
  2. Writing setting values into the plugins' runtime environment under an environment variable name other than the default.

For sourcing setting values we encourage users going forward to make use of the default environment variables (of the form <PLUGIN_NAME>_<SETTING_NAME>) for settings in most cases. These can be conveniently found for a given plugin by running meltano config <plugin> list. In cases where an environment variable of a name other than the default must be used to source a setting value, Meltano supports referencing in meltano.yml. For example:

plugins:
  extractors:
    - name: tap-gitlab
      config:
        ultimate_license: $GITLAB_API_ULTIMATE_LICENSE

This will take the value from the environment variable GITLAB_API_ULTIMATE_LICENSE and use it configure the tap-gitlab setting named ultimate_license.

For writing setting values into the plugins' runtime environment under an environment variable name other than the default, we support the env: key for setting definitions. These can be added or overridden in your meltano.yml file. For example:

plugins:
  extractors:
    - name: tap-gitlab
      settings:
        - name: ultimate_license
          env: GITLAB_API_ULTIMATE_LICENSE

This will create an environment variable called GITLAB_API_ULTIMATE_LICENSE in the plugins' runtime environment with the configured value of the setting ultimate_license.

Updating your Project​

Before v2.0 Meltano made use of env_aliases internally and in several common plugins. To ensure Meltano and those plugins continue to work as expected, references to the deprecated environment variables in your own project should be replaced. The table below contains details of the env_aliases that have been removed, and the corresponding default setting environment variable to use in each case. If for any reason you wish to keep sourcing or writing setting values to deprecated aliases, we recommend using the techniques detailed above to replace the functionality formally provided by env_aliases.

PluginVariantDeprecatedReplacement
meltanoMELTANO_LOG_LEVELMELTANO_CLI_LOG_LEVEL
MELTANO_LOG_CONFIGMELTANO_CLI_LOG_CONFIG
MELTANO_API_HOSTNAMEMELTANO_UI_BIND_HOST
MELTANO_API_PORTMELTANO_UI_BIND_PORT
PORTMELTANO_UI_BIND_PORT
WORKERSMELTANO_UI_WORKERS
WEB_CONCURRENCYMELTANO_UI_WORKERS
FORWARDED_ALLOW_IPSMELTANO_UI_FORWARDED_ALLOW_IPS
MELTANO_READONLYMELTANO_UI_READONLY
MELTANO_AUTHENTICATIONMELTANO_UI_AUTHENTICATION
MELTANO_NOTIFICATIONMELTANO_UI_NOTIFICATION
TAP_ADWORDS_OAUTH_CLIENT_IDMELTANO_OAUTH_SERVICE_GOOGLE_ADWORDS_CLIENT_ID
TAP_ADWORDS_OAUTH_CLIENT_SECRETMELTANO_OAUTH_SERVICE_GOOGLE_ADWORDS_CLIENT_SECRET
OAUTH_GITLAB_APPLICATION_IDMELTANO_OAUTH_GITLAB_CLIENT_ID
OAUTH_GITLAB_SECRETMELTANO_OAUTH_GITLAB_CLIENT_SECRET
MELTANO_CLI_TRACKING_IDMELTANO_TRACKING_IDS_CLI
MELTANO_UI_TRACKING_IDMELTANO_TRACKING_IDS_UI
MELTANO_EMBED_TRACKING_IDMELTANO_TRACKING_IDS_UI_EMBED
dbt-bigquerymeltanoDBT_PROFILES_DIRDBT_BIGQUERY_PROFILES_DIR
dbt-postgresmeltanoDBT_PROFILES_DIRDBT_POSTGRES_PROFILES_DIR
dbt-redshiftmeltanoDBT_PROFILES_DIRDBT_REDSHIFT_PROFILES_DIR
dbt-snowflakemeltanoDBT_PROFILES_DIRDBT_SNOWFLAKE_PROFILES_DIR
tap-adwordsmeltanoOAUTH_GOOGLE_ADWORDS_CLIENT_IDTAP_ADWORDS_OAUTH_CLIENT_ID
OAUTH_GOOGLE_ADWORDS_CLIENT_SECRETTAP_ADWORDS_OAUTH_CLIENT_SECRET
OAUTH_GOOGLE_ADWORDS_DEVELOPER_TOKENTAP_ADWORDS_DEVELOPER_TOKEN
singer-ioOAUTH_GOOGLE_ADWORDS_CLIENT_IDTAP_ADWORDS_OAUTH_CLIENT_ID
OAUTH_GOOGLE_ADWORDS_CLIENT_SECRETTAP_ADWORDS_OAUTH_CLIENT_SECRET
OAUTH_GOOGLE_ADWORDS_DEVELOPER_TOKENTAP_ADWORDS_DEVELOPER_TOKEN
tap-bigqueryanelendataGOOGLE_APPLICATION_CREDENTIALSTAP_BIGQUERY_CREDENTIALS_PATH
tap-bing-adssinger-ioOAUTH_BING_ADS_CLIENT_IDTAP_BING_ADS_OAUTH_CLIENT_ID
OAUTH_BING_ADS_CLIENT_SECRETTAP_BING_ADS_OAUTH_CLIENT_SECRET
OAUTH_BING_ADS_DEVELOPER_TOKENTAP_BING_ADS_DEVELOPER_TOKEN
tap-csvmeltanoTAP_CSV_FILES_DEFINITIONTAP_CSV_CSV_FILES_DEFINITION
meltanolabsTAP_CSV_FILES_DEFINITIONTAP_CSV_CSV_FILES_DEFINITION
tap-gitlabmeltanoGITLAB_API_GROUPSTAP_GITLAB_GROUPS
GITLAB_API_PROJECTSTAP_GITLAB_PROJECTS
GITLAB_API_START_DATETAP_GITLAB_START_DATE
GITLAB_API_TOKENTAP_GITLAB_PRIVATE_TOKEN
GITLAB_API_ULTIMATE_LICENSETAP_GITLAB_ULTIMATE_LICENSE
meltanolabsGITLAB_API_GROUPSTAP_GITLAB_GROUPS
GITLAB_API_PROJECTSTAP_GITLAB_PROJECTS
GITLAB_API_START_DATETAP_GITLAB_START_DATE
GITLAB_API_TOKENTAP_GITLAB_PRIVATE_TOKEN
GITLAB_API_ULTIMATE_LICENSETAP_GITLAB_ULTIMATE_LICENSE
tap-google-analyticsmeltanoGOOGLE_ANALYTICS_API_CLIENT_SECRETSTAP_GOOGLE_ANALYTICS_KEY_FILE_LOCATION
GOOGLE_ANALYTICS_API_END_DATETAP_GOOGLE_ANALYTICS_END_DATE
GOOGLE_ANALYTICS_API_OAUTH_ACCESS_TOKENTAP_GOOGLE_ANALYTICS_OAUTH_CREDENTIALS_ACCESS_TOKEN
GOOGLE_ANALYTICS_API_OAUTH_CLIENT_IDTAP_GOOGLE_ANALYTICS_OAUTH_CREDENTIALS_CLIENT_ID
GOOGLE_ANALYTICS_API_OAUTH_CLIENT_SECRETTAP_GOOGLE_ANALYTICS_OAUTH_CREDENTIALS_CLIENT_SECRET
GOOGLE_ANALYTICS_API_OAUTH_REFRESH_TOKENTAP_GOOGLE_ANALYTICS_OAUTH_CREDENTIALS_REFRESH_TOKEN
GOOGLE_ANALYTICS_API_REPORTSTAP_GOOGLE_ANALYTICS_REPORTS
GOOGLE_ANALYTICS_API_START_DATETAP_GOOGLE_ANALYTICS_START_DATE
GOOGLE_ANALYTICS_API_VIEW_IDTAP_GOOGLE_ANALYTICS_VIEW_ID
tap-pendosinger-ioTAP_PENDO_INTEGRATION_KEYTAP_PENDO_X_PENDO_INTEGRATION_KEY
tap-stripemeltanoSTRIPE_ACCOUNT_IDTAP_STRIPE_ACCOUNT_ID
STRIPE_API_KEYTAP_STRIPE_CLIENT_SECRET
prratekSTRIPE_API_KEYTAP_STRIPE_API_KEY
singer-ioSTRIPE_ACCOUNT_IDTAP_STRIPE_ACCOUNT_ID
STRIPE_API_KEYTAP_STRIPE_CLIENT_SECRET
target-bigqueryadswerveGOOGLE_APPLICATION_CREDENTIALSTARGET_BIGQUERY_CREDENTIALS_PATH
target-postgresdatamill-coPG_ADDRESSTARGET_POSTGRES_POSTGRES_HOST
PG_DATABASETARGET_POSTGRES_POSTGRES_DATABASE
PG_PASSWORDTARGET_POSTGRES_POSTGRES_PASSWORD
PG_PORTTARGET_POSTGRES_POSTGRES_PORT
PG_SCHEMATARGET_POSTGRES_POSTGRES_SCHEMA
PG_USERNAMETARGET_POSTGRES_POSTGRES_USERNAME
TARGET_POSTGRES_DATABASETARGET_POSTGRES_POSTGRES_DATABASE
TARGET_POSTGRES_HOSTTARGET_POSTGRES_POSTGRES_HOST
TARGET_POSTGRES_PASSWORDTARGET_POSTGRES_POSTGRES_PASSWORD
TARGET_POSTGRES_PORTTARGET_POSTGRES_POSTGRES_PORT
TARGET_POSTGRES_SCHEMATARGET_POSTGRES_POSTGRES_SCHEMA
TARGET_POSTGRES_SSLCERTTARGET_POSTGRES_POSTGRES_SSLCERT
TARGET_POSTGRES_SSLCRLTARGET_POSTGRES_POSTGRES_SSLCRL
TARGET_POSTGRES_SSLKEYTARGET_POSTGRES_POSTGRES_SSLKEY
TARGET_POSTGRES_SSLMODETARGET_POSTGRES_POSTGRES_SSLMODE
TARGET_POSTGRES_SSLROOTCERTTARGET_POSTGRES_POSTGRES_SSLROOTCERT
TARGET_POSTGRES_USERNAMETARGET_POSTGRES_POSTGRES_USERNAME
meltanoPG_ADDRESSTARGET_POSTGRES_HOST
PG_DATABASETARGET_POSTGRES_DBNAME
PG_PASSWORDTARGET_POSTGRES_PASSWORD
PG_PORTTARGET_POSTGRES_PORT
PG_SCHEMATARGET_POSTGRES_SCHEMA
PG_URLTARGET_POSTGRES_URL
PG_USERNAMETARGET_POSTGRES_USER
POSTGRES_DBNAMETARGET_POSTGRES_DBNAME
POSTGRES_HOSTTARGET_POSTGRES_HOST
POSTGRES_PASSWORDTARGET_POSTGRES_PASSWORD
POSTGRES_PORTTARGET_POSTGRES_PORT
POSTGRES_SCHEMATARGET_POSTGRES_SCHEMA
POSTGRES_URLTARGET_POSTGRES_URL
POSTGRES_USERTARGET_POSTGRES_USER
transferwisePG_ADDRESSTARGET_POSTGRES_HOST
PG_DATABASETARGET_POSTGRES_DBNAME
PG_PASSWORDTARGET_POSTGRES_PASSWORD
PG_PORTTARGET_POSTGRES_PORT
PG_SCHEMATARGET_POSTGRES_DEFAULT_TARGET_SCHEMA
PG_USERNAMETARGET_POSTGRES_USER
target-redshifttransferwiseAWS_ACCESS_KEY_IDTARGET_REDSHIFT_AWS_ACCESS_KEY_ID
AWS_PROFILETARGET_REDSHIFT_AWS_PROFILE
AWS_SECRET_ACCESS_KEYTARGET_REDSHIFT_AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKENTARGET_REDSHIFT_AWS_SESSION_TOKEN
TARGET_REDSHIFT_SCHEMATARGET_REDSHIFT_DEFAULT_TARGET_SCHEMA
target-snowflakedatamill-coSF_ACCOUNTTARGET_SNOWFLAKE_SNOWFLAKE_ACCOUNT
SF_DATABASETARGET_SNOWFLAKE_SNOWFLAKE_DATABASE
SF_PASSWORDTARGET_SNOWFLAKE_SNOWFLAKE_PASSWORD
SF_ROLETARGET_SNOWFLAKE_SNOWFLAKE_ROLE
SF_SCHEMATARGET_SNOWFLAKE_SNOWFLAKE_SCHEMA
SF_USERTARGET_SNOWFLAKE_SNOWFLAKE_USERNAME
SF_WAREHOUSETARGET_SNOWFLAKE_SNOWFLAKE_WAREHOUSE
TARGET_SNOWFLAKE_ACCOUNTTARGET_SNOWFLAKE_SNOWFLAKE_ACCOUNT
TARGET_SNOWFLAKE_DATABASETARGET_SNOWFLAKE_SNOWFLAKE_DATABASE
TARGET_SNOWFLAKE_PASSWORDTARGET_SNOWFLAKE_SNOWFLAKE_PASSWORD
TARGET_SNOWFLAKE_ROLETARGET_SNOWFLAKE_SNOWFLAKE_ROLE
TARGET_SNOWFLAKE_SCHEMATARGET_SNOWFLAKE_SNOWFLAKE_SCHEMA
TARGET_SNOWFLAKE_USERNAMETARGET_SNOWFLAKE_SNOWFLAKE_USERNAME
TARGET_SNOWFLAKE_USERNAMETARGET_SNOWFLAKE_SNOWFLAKE_USERNAME
TARGET_SNOWFLAKE_WAREHOUSETARGET_SNOWFLAKE_SNOWFLAKE_WAREHOUSE
meltanoSF_ACCOUNTTARGET_SNOWFLAKE_ACCOUNT
SF_DATABASETARGET_SNOWFLAKE_DATABASE
SF_PASSWORDTARGET_SNOWFLAKE_PASSWORD
SF_ROLETARGET_SNOWFLAKE_ROLE
SF_SCHEMATARGET_SNOWFLAKE_SCHEMA
SF_USERTARGET_SNOWFLAKE_USERNAME
SF_WAREHOUSETARGET_SNOWFLAKE_WAREHOUSE
SNOWFLAKE_ACCOUNTTARGET_SNOWFLAKE_ACCOUNT
SNOWFLAKE_DATABASETARGET_SNOWFLAKE_DATABASE
SNOWFLAKE_PASSWORDTARGET_SNOWFLAKE_PASSWORD
SNOWFLAKE_ROLETARGET_SNOWFLAKE_ROLE
SNOWFLAKE_SCHEMATARGET_SNOWFLAKE_SCHEMA
SNOWFLAKE_USERNAMETARGET_SNOWFLAKE_USERNAME
SNOWFLAKE_WAREHOUSETARGET_SNOWFLAKE_WAREHOUSE
transferwiseSF_ACCOUNTTARGET_SNOWFLAKE_ACCOUNT
SF_DATABASETARGET_SNOWFLAKE_DBNAME
SF_PASSWORDTARGET_SNOWFLAKE_PASSWORD
SF_ROLETARGET_SNOWFLAKE_SNOWFLAKE_ROLE
SF_SCHEMATARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA
SF_USERTARGET_SNOWFLAKE_USER
SF_WAREHOUSETARGET_SNOWFLAKE_WAREHOUSE
TARGET_SNOWFLAKE_DATABASETARGET_SNOWFLAKE_DBNAME
TARGET_SNOWFLAKE_SCHEMATARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA
TARGET_SNOWFLAKE_USERNAMETARGET_SNOWFLAKE_USER
target-sqlitemeltanoSQLITE_DATABASETARGET_SQLITE_DATABASE
meltanolabsSQLITE_DATABASETARGET_SQLITE_DATABASE

CLI and API Changes​

Use --state-id instead of --job_id​

In 2.0, many references to "Job ID" in our code and docs were changed to the more accurate name of "State ID".

If you are explicitly providing a --job_id option to any scripted or otherwise automated CLI workflows, these commands should be updated to use the --state-id option instead.

Schedule list format changes​

If you have custom orchestrator integrations based on the meltano schedule list command, you will need to make adjustments to handle a new output format.

With the addition of support for scheduled jobs in 2.0, the schema output of meltano schedule list --format=json has changed. It now includes a top level field schedules and two nested array fields job and elt which hold and describe their respective schedules.

ex:

{
  "schedules": {
    "job": [
      {
        "name": "daily-doit",
        "interval": "@daily",
        "cron_interval": "0 0 * * *",
        "env": {},
        "job": {
          "name": "simple-demo",
          "tasks": [
            "tap-gitlab hide-gitlab-secrets target-jsonl",
            "tap-gitlab target-csv"
          ]
        }
      }
    ],
    "elt": [
      {legacy elt schedule entry remains unchanged}, ...
    ]
  }
}
Github LogoEdit this page on Github