For every star on GitHub, we'll donate $2 to clean up our waterways. Star us now!
Once you’ve set up a Meltano project and run some pipelines on your local machine, it’ll be time to repeat this trick in production!
The Deployment in Production guide will walk you through getting Meltano, your project, and all of its plugins onto a new environment one-by-one, among other things, but you can greatly simplify this process (and prevent issues caused by inconsistencies between environments!) by wrapping them all up into a project-specific Docker container image: “a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings.”
This image can then be used on any environment running Docker
(or a compatible tool like Kubernetes) to directly
in the context of your project, without needing to separately manage the installation of
Meltano, your project’s plugins, or any of their dependencies.
If you’re storing your Meltano project in version control on a platform like GitLab or GitHub, you can set up a CI/CD pipeline to run every time a change is made to your project, which can automatically build a new version of the image and push it to a container registry. The image can then be pulled from that registry onto any local or cloud environment on which you’d like to run your project’s pipelines.
If you’d like to containerize your Meltano project, you can easily add the
.dockerignore files to your project by adding the
docker file bundle:
# For these examples to work, ensure that # Docker has been installed docker --version # Add Docker files to your project meltano add files files-docker # Build Docker image containing # Meltano, your project, and all of its plugins docker build --tag meltano-demo-project:dev .
Files added to your project include a
FROM the public
meltano/meltano:latest image available on Docker Hub.
This can be customized to use another public mirror, a private mirror (e.g.
your-company/meltano:latest), a specific version of Meltano (e.g.
meltano/meltano:v2.15-python3.10), or Python 3.9 or 3.10 (e.g.
meltano/meltano:v2.15-python3.10) by modifying the
Dockerfile or overriding the
--build-arg. We currently publish images to Docker Hub and Gitlab Registry. Using an alternative public mirror, or creating a private one, can avoid issues during your Docker build stage relating to registry rate limits.
The built image’s entrypoint
will be the
meaning that you can provide
meltano subcommands and arguments like
run ... and
invoke airflow ... directly to
docker run <image-name> ...
as trailing arguments:
# View Meltano version docker run meltano-demo-project:dev --version # Run gitlab-to-jsonl pipeline with # mounted volume to exfiltrate target-jsonl output docker run \ --volume $(pwd)/output:/project/output \ meltano-demo-project:dev \ run tap-gitlab target-jsonl
If you’d like to use Docker Compose to experiment with
a production-grade setup of your containerized project,
you can add the appropriate
docker-compose.prod.yml file to your project by adding the
docker-compose file bundle:
# For these examples to work, ensure that # Docker Compose has been installed docker-compose --version # Add Docker Compose files to your project meltano add files files-docker-compose # Start the `meltano-ui` and `meltano-system-db` services in the background docker-compose -f docker-compose.prod.yml up -d
For more details and instructions, refer to README contained in the file bundle.
If you’d like to use GitLab CI/CD to continuously
build your Meltano project’s Docker image and push it to GitLab’s built-in
you can add the appropriate
files to your project by adding the
gitlab-ci file bundle:
# For these examples to work, ensure that # you have an account on GitLab.com or # a self-hosted GitLab instance with # GitLab CI/CD and Container Registry enabled # Add GitLab CI/CD files to your project meltano add files files-gitlab-ci # Initialize Git repository, if you haven't already git init # Add and commit all files git add -A git commit -m "Set up Meltano project with Docker and GitLab CI" # Push to GitLab, which will automatically create # a new private project at the specified path NAMESPACE="<your-gitlab-username-or-group>" git push firstname.lastname@example.org:$NAMESPACE/meltano-demo-project.git master
GitLab CI/CD will now start building your Meltano project’s dedicated Docker image,
which will be available at
once the CI/CD pipeline completes.