Skip to main content
November2023

Meltano Inc. is now Arch Data Inc.

We've changed our company name to Arch (or Arch Data if you're the government)! We did this for a number of reasons, many of which we detailed in a few blog posts.

For those of you in the Meltano community we recommend you checkout this post on the Meltano blog. There's also a post on the new Arch blog which details our vision behind this new product and how we got to this point.

The TL;DR is that the Meltano project and community that you know and love aren’t going anywhere. Arch is built on top of Meltano and we can’t bring data engineering capabilities to software teams without great underlying data engineering tooling.

This also means that Meltano Cloud is also changing names to Arch. We will have a separate site and changelog for updates there while this changelog will still be used for relevant community updates around Meltano, the SDK, and the Hub.

MeltanoHub

Extractors

  • Added the Ticketswap variant of tap-delighted - Hub Link
  • Added the akurdyukov variant of tap-clickhouse - Hub Link
  • Added the Stitch Data variant of tap-googleads - Hub Link
  • Added the Automattic variant of tap-upwork - Hub Link
  • Added the Storebrand variant of tap-pxwebapi - Hub Link
  • Added the Storebrand variant of tap-ssb-klass - Hub Link

Loaders

  • Added the Automattic variant of target-hdfs - Hub Link

Utilities

  • Added a PowerBI Extension used to trigger a refresh of a dataset - Hub Link

SDK

We shipped 1 release of the SDK:

September2023

☁️ Meltano Cloud

🏃 Run State

With Meltano Core you're able to easily manage the state of your incremental runs using the meltano state command. Most of this functionality was ported to Meltano Cloud so that you can control your runs as you see fit. If you need to clear state to run a backfill, that's as easy as meltano cloud state delete --state-id <id>. Read more about the command in our Cloud docs.

✅ SOC2 Type 1 Audit Passed

We're extremely proud to announce that we've passed our SOC2 Type 1 Audit. Read more about this milestone in our blog post.

Meltano 3.0

We released v3.0 of Meltano Core! Checkout this blog post for more details on new features and any breaking changes.

We also released the first patch release (v3.1) of Meltano which includes a number of community contributions.

MeltanoHub

  • The python setting is now supported in the metadata of all connectors. This is a new feature enabled in Meltano 3.0. Checkout the docs for more information.

Extractors

  • Added the danielptv variant of tap-db2 based on the SDK - Hub Link
  • Added the Meltano Labs variant of tap-salesforce based on the SDK - Hub Link
  • Added the felippecaso variant of tap-applehealth based on the SDK - Hub Link
  • Added the Matatika variant of tap-capsulecrm - Hub Link
  • Added the Meltano Labs variant of tap-bigquery - Hub Link

SDK

We shipped 1 release of the SDK:

August2023

The team has been heads down this month bringing improvements and squashing bugs to all things Meltano. Let's take a look at some of the bigger items we shipped recently!

☁️ Meltano Cloud

🚀 Automated deployments

Now when you set your deployment to track a git branch, any new commits that appear on that branch will automatically be deployed in Meltano Cloud. Previously users had to complete an additional manual step to get any new committed changes deployed.

Static deployments are still possible by referencing a particular git hash.

🚨 Custom Notifications

Cloud customers now have greater flexibility over their notifications. With the updated config CLI command, you can specify exactly how you want to receive notifications. Email and webhooks are the initially supported destinations and the payload of the notification can be filtered to exactly how you want it - failures only, successes only, or anything!

meltano cloud config notification set

➕ Add new projects from the CLI

New projects can now easily be added from the CLI to your Meltano Cloud account. This simplifies onboarding for new users as well as our power users that have 10s or 100s of projects they need to onboard.

Meltano Core

v2.20.0 of Meltano Core was released. This was mostly a bug, performance, and doc update release.

MeltanoHub

We added a "Most Popular" listing to the Extractors page. Check it out!

The Meltano Team and Community have also been busy adding new connectors and utilities to the Hub. Many of the new connectors are related to our recent blog post about how LLMs are mostly data pipelines.

Extractors

  • Added the MeltanoLabs variant of tap-jira - Link
  • Added the MeltanoLabs variant of tap-mysql. This tap is still under active development. - Link
  • Added the MeltanoLabs variant of tap-beautifulsoup - Link
  • Added the sehnem variant of tap-shopify which supports accessing data via graphql - Link

Mappers

  • Added the map-gpt-embeddings mapper. This can be used to connect to OpenAI to generate embeddings for any data from an extractor - Link

Loaders

  • Added the MeltanoLabs variant of target-pinecone which is a Vector Database for storing embeddings. - Link

Utilities

  • Added Tableau utility which can be used to trigger a refresh of Tableu data source - Link
  • Added the dbt Artifacts utility which can be used to process dbt-generated artifacts for other use cases such as model lineage - Link

SDK

We shipped 2 releases of the SDK:

Highlights include:

  • Improved connection handling in SQL targets by sharing a connector instance among stream sinks
  • Expose builtin add_record_metadata and batch_config target settings
July2023

Welcome to the new Meltano Changelog!

This is going to be our new place to share and keep a record of all of the product updates happening around the Meltano ecosystem. Previously we'd shared these in our newsletter and in Slack. We'll likely keep putting updates there, but this will be the primary destination for updates going forward.

Let's dive in to what's improved over the past month or so!

☁️ Meltano Cloud entered Public Beta

Right at the end of May this year we launched Meltano Cloud into Public Beta. This was a key milestone for us at it represented the threshold where early Alpha users were seeing enough value to pay for Cloud use. And it means we can start working through our large waitlist to onboard more users.

Request Access to the Meltano Cloud Beta

Since then we've:

  • Added the ability to cancel a running job from the CLI
  • Added syntax highlighting to the logs in the Cloud UI
  • Added support for manual, once, and none aliases for schedules. Useful for one-off runs!
  • Improved the error code response when creating or updating a deployment
  • Squashed quite a few bugs. Thanks to our early users for all of their feedback!

Core

We released v2.20.0 which was primarily a bugfix and documentation release. We're also incrementally starting to improve the settings definitions for plugins. We now recommend using hidden: true instead of kind: hidden on plugins. This is part of an effort to make kind indicate just the data type and not any sort of UX choices that may be desired.

MeltanoHub

The Meltano Community has been busy adding new connectors to MeltanoHub. In June and July so far we've had the following additions and updates:

Extractors

  • Updated the Google Analytics connectors and docs due to the GA4 change - Link
  • Added the MeltanoLabs variant of tap-hubspot - Link
  • Added the datarts-tech variant of tap-cassanda - Link
  • Added the acarter24 variant of tap-mailchimp - Link
  • Updated the default variant fo SaaSOptics to the datarts-tech variant - Link
  • Updated the default variant of tap-postgres and tap-snowflake to the MeltanoLabs variant - Link
  • Added tap-theme-parks - Link
  • Added the sehnem variant of tap-netsuite - Link
  • Updated the default variant of tap-recruitee to the rawwar variant - Link

Loaders

  • Added a loader for Clickhouse - Link
  • Added target-sqs and target-parquet variants from Hotglue - Link
  • Updated the default variant of target-postgres and target-snowflake to the MeltanoLabs variant - Link

We've also got an introductory video on how to go from Zero to Snowflake with the help of target-snowflake.

Connectors

We're continuing to invest in building first-party connectors based on the Meltano SDK.

We also have new extractors for Jira and Salesforce landing soon.

SDK

We shipped 3 SDK releases

Some highlights from these releases are:

  • Supporting optional headers in OAuth requests
  • Publishing the supported python version in --about
  • Supporting union schemas which is useful for connectors that accept mutually exclusive settings (e.g. OAuth vs API key auth)
  • Validation of a parsed/transformed record against the SCHEMA message
  • Add Apache 2.0 as an optional license when developing a connector

And of course lots of fixes, under the hood changes, and improved documentation.