Skip to main content

Deployment in Production

Once you've set up a Meltano project and run some pipelines on your local machine, it'll be time to repeat this trick in production!

This page will help you figure out:

  1. how to get your Meltano project onto the production environment,
  2. how to install Meltano,
  3. how to install your Meltano project's plugins,
  4. where to store your pipeline state and other metadata,
  5. where to store your pipeline logs,
  6. how to manage your environment-specific and sensitive configuration, and finally
  7. how to run your pipelines.

If you're containerizing your Meltano project, you can skip steps 1 through 3 and refer primarily to the "Containerized Meltano project" subsections on this page.

Managed Hosting Options​

Meltano Cloud has been shut down in favor of Arch.

Consider running your Meltano pipelines using a managed Airflow service like Astronomer, Google Cloud Composer, or Amazon MWAA.

Your Meltano project​

Off of your local machine...​

Since a Meltano project is just a directory on your filesystem containing text-based files, you can treat it like any other software development project and benefit from DataOps best practices such as version control, code review, and continuous integration and deployment (CI/CD).

As such, getting your Meltano project onto the production environment starts with getting it off of your local machine, and onto a (self-)hosted Git repository platform like GitLab or GitHub.

By default, your Meltano project comes with a .gitignore file to ensure that environment-specific and potentially sensitive configuration stored inside the .meltano directory and .env file is not leaked accidentally. All other files are recommended to be checked into the repository and shared between all users and environments that may use the project.

... and onto the production environment​

Once your Meltano project is in version control, getting it to your production environment can take various shapes.

In general, we recommend setting up a CI/CD pipeline to run automatically whenever new changes are pushed to your repository's default branch, that will connect with the production environment and either directly push the project files, or trigger some kind of mechanism to pull the latest changes from the repository.

A simpler (temporary?) approach would be to manually connect to the production environment and pull the repository, right now while you're setting this up, and/or later whenever changes are made.

Containerized Meltano project​

If you're containerizing your Meltano project, your project-specific Docker image will already contain all of your project files.

Installing Meltano​

Just like on your local machine, the most straightforward way to install Meltano onto a production environment is to use pip to install the meltano package from PyPI.

If you add meltano (or meltano==<version>) to your project's requirements.txt file, you can choose to automatically run pip install -r requirements.txt on your production environment whenever your Meltano project is updated to ensure you're always on the latest (or requested) version.

Containerized Meltano project​

If you're containerizing your Meltano project, your project-specific Docker image will already contain a Meltano installation since it's built from the meltano/meltano base image.

Installing plugins​

Whenever you add a new plugin to a Meltano project, it will be installed into your project's .meltano directory automatically. However, since this directory is included in your project's .gitignore file by default, you'll need to explicitly run meltano install before any other meltano commands whenever you clone or pull an existing Meltano project from version control, to install (or update) all plugins specified in your meltano.yml project file.

Thus, it is strongly recommended that you automatically run meltano install on your production environment whenever your Meltano project is updated to ensure you're always using the correct versions of plugins.

Containerized Meltano project​

If you're containerizing your Meltano project, your project-specific Docker image will already contain all of your project's plugins since meltano install is a step in its build process.

Storing metadata​

Meltano stores various types of metadata in a project-specific system database, that takes the shape of a SQLite database stored inside the project at .meltano/meltano.db by default. Like all files stored in the .meltano directory (which you'll remember is included in your project's .gitignore file by default), the system database is also environment-specific.

While SQLite is great for use during local development and testing since it requires no external database to be set up, it has various limitations that make it inappropriate for use in production. Since it's a simple file, it only supports one concurrent connection, for example.

Thus, it is is strongly recommended that you use a PostgreSQL system database in production instead. You can configure Meltano to use it using the database_uri setting.

Containerized Meltano project​

If you're containerizing your Meltano project, you will definitely want to use an external system database, since changes to .meltano/meltano.db would not be persisted outside the container.

Storing logs​

Meltano stores all output generated by meltano run and meltano elt in .meltano/logs/elt/{state_id}/{run_id}/elt.log, where state_id refers to the value autogenerated by run, provided to elt via the --state_id flag, or the name of a scheduled pipeline, and run_id is an autogenerated UUID.

If you'd like to store these logs elsewhere, you can symlink the .meltano/logs or .meltano/logs/elt directory to a location of your choice.

Containerized Meltano project​

If you're containerizing your Meltano project, these logs will not be persisted outside the container running your pipelines unless you exfiltrate them by mounting a volume inside the container at /project/.meltano/logs/elt.

Managing configuration​

All of your Meltano project's configuration that is not environment-specific or sensitive should be stored in its meltano.yml project file and checked into version control.

Configuration that is environment-specific or sensitive is most appropriately managed using environment variables. Meltano Environments can be used to better manage configuration between different deployment environments. How these can be best administered will depend on your deployment strategy and destination.

If you'd like to store sensitive configuration in a secrets store, you can consider using the chamber CLI, which lets you store secrets in the AWS Systems Manager Parameter Store that can then be exported as environment variables when executing an arbitrary command like meltano.

Containerized Meltano project​

If you're containerizing your Meltano project, you will want to manage sensitive configuration using the mechanism provided by your container runner, e.g. Docker Secrets or Kubernetes Secrets.

Running pipelines​

meltano run​

If all of the above has been set up correctly, you should now be able to run a pipeline using meltano run, just like you did locally. Congratulations!

You can run the command using any mechanism capable of running executables, whether that's cron, Airflow's BashOperator, or any of dozens of other orchestration tools.

Airflow orchestrator​

If you've added Airflow to your Meltano project as an orchestrator utility, you can have it automatically run your project's scheduled pipelines by starting its scheduler using meltano invoke airflow scheduler.

Similarly, you can start its web interface using meltano invoke airflow webserver.

However, do take into account Airflow's own Deployment in Production Best Practices. Specifically, you will want to configure Airflow to:

  • use the LocalExecutor instead of the SequentialExecutor default by setting the core.executor setting (or AIRFLOW__CORE__EXECUTOR environment variable) to LocalExecutor:

    meltano config airflow set core.executor LocalExecutor

  • use a PostgreSQL metadata database instead of the SQLite default (sounds familiar?) by setting the core.sql_alchemy_conn setting (or AIRFLOW__CORE__SQL_ALCHEMY_CONN environment variable) to a postgresql:// URI:

    meltano config airflow set core.sql_alchemy_conn postgresql://<username>:<password>@<host>:<port>/<database>

    For this to work, the psycopg2 package will also need to be installed alongside apache-airflow, which you can realize by adding psycopg2 to airflow's pip_url in your meltano.yml project file (e.g. pip_url: psycopg2 apache-airflow) and running meltano install utility airflow.

Containerized Meltano project​

If you're containerizing your Meltano project, the built image's entrypoint will be the meltano command, meaning that you can provide meltano subcommands and arguments like elt ... and invoke airflow ... directly to docker run <image-name> ... as trailing arguments.

Github LogoEdit this page on Github