For every star on GitHub, we'll donate $2 to clean up our waterways. Star us now!
This document is intended to give you enough technical understanding of Meltano to become excited about it and wanting to use it! It’s not going to teach you how to use it, we’ve got Tutorials & How To’s for that. When you’re ready to start your first Meltano project, we recommend you dive right into our Tutorial.
“For us it’s a better day at work when we can use Meltano.” - Nino Müller, Head of Technology at Substring
“I love Meltano because it’s so pleasant to use with its DevOps and Everything-as-Code style. It is easy to set up, flexible, and integrates with pretty much any orchestrator as well as dbt (data build tool)”. - Martin Morset
Welcome to your Open Source DataOps Infrastructure! With Meltano you can move your data with 10x the developer experience while also managing all of the data tools in your stack. With Meltano, you can collaboratively build and improve your ideal data platform like a software project; spinning up a service or tool (Singer connectors, Airflow, dbt, Great Expectations, Snowflake, etc) and easily configure, deploy, and manage it through a single control plane.
Waiting to see how Meltano works within 90 secs? We got you covered:
Meltano helps you to create your end-to-end data stack within minutes. The core workflow depends on your data stack, but it will usually involve:
Meltano allows you to do any combination of these steps inside your Meltano project, controlled by the Meltano CLI.
Here’s a complete walk-through pulling data from AWS S3 and dumping it into a PostgreSQL database within 60 secs.
Here’s a complete walk-through extending the extract & load to include more CSVs and running a dbt-project over them to transform the data.
Meltano uses Airflow as orchestrator for the pipelines. It’s as simple as adding Airflow as a plugin to your project and then running
meltano schedule add gitlab-to-postgres --extractor tap-gitlab --loader target-postgres --interval @daily
to add the schedule. Meltano also provides commands to start an Airflow instance to execute on these schedules. You can find out more about it in the Orchestrate Data Section.
Need to add additional steps to your data pipeline? Here’s a complete setup also pulling in Superset as visualization tool.
This was just a glance at why you should use Meltano. If you’re now as excited to use Meltano as we are, we recommend you head over to the Getting Started Tutorial.
If you cannot find an answer to your question, there’s always an active Meltano Slack Community to help you out.