v3.5.1

v3.5.1 (2024-08-23)

• Docker Hub 🐋
• PyPI 🐍

🐛 Fixes

• #8689 Emit a clearer error message when trying to set/unset an unknown setting in .env
• #8658 List only the state IDs within the specified prefix in the S3 URI
• #8698 meltano config ... test now uses the elt.buffer setting
• #8699 Add missing project_readonly setting to JSON schema
• #8691 Addressed structlog warning by removing format_exc_info from the processor chain

📚 Documentation Improvements

• #8721 Added instructions for installing Meltano with uv
• #8695 Fixed line highlighting and handling of == in code blocks

v3.5.0

v3.5.0 (2024-07-23)

• Docker Hub 🐋
• PyPI 🐍

Highlights

• Plugins are now auto-installed when commands require them. This change involves two new features:

  • New CLI options:
    • --install
    • --no-install
    • --only-install
  • A new project-level setting auto_install. The value of this determines whether plugins are auto-installed when none of the above options are passed (the default). Set it to false to require --install to auto-install plugins.

  There are some known gaps that will be addressed in future releases, give them a 👍 to let us know they are affecting your use of this feature:

  • #8660

  Thanks to @ReubenFrankel for contributing this to Meltano!

• A new CLI option --refresh-catalog and plugin-level extra use_cached_catalog. Use this if you need the extractor's catalog to reflect changes that occurred upstream, like often happens with database sources.

  Thanks to @andyoneal for contributing this feature!

✨ New

• #8633 Added a --only-install option to commands that can auto-install plugins
• #8482 Plugins are now auto-installed when commands require them -- Thanks @ReubenFrankel!
• #8620 Sensitive values in meltano config <plugin> are now hidden by default
• #8580 Added CLI --refresh-catalog option and extractor extra use_cached_catalog to ignore the cached source catalog

🐛 Fixes

• #8648 Listing Cloud (S3, GCS, etc.) state IDs no longer crashes if there is a file at the root of the bucket, and files in GCS buckets are only listed within the prefix specified in the state_backend_uri -- Thanks @jx2lee!
• #8636 Values with $ can now be escaped instead of trying to expand them from environment variables
• #8590 meltano config ... test now also checks for BATCH messages

⚙️ Under the Hood

• #8265 Use timezone-aware datetime objects

📚 Documentation Improvements

• #8646 Documented how to escape setting values with $
• #8627 Fixed a broken link in the plugin concept page
• #8621 Update to current year and recommendations
• #8610 Remove Meltano Cloud docs
• #8609 Redirect Meltano Cloud docs to Arch docs
• #8608 Meltano Cloud has been shut down in favor of Arch -- Thanks @hulet!
• #8505 Added 2024-Q2 changelog
• #8589 Recommend adding namespace key to custom plugins when migrating to Meltano 3
• #8578 Update environments.md to fix link
• #8570 Added a few dbt-postgres troubleshooting notes to the Getting Started guide
• #8566 Added a note about state IDs when using meltano schedule run

v3.4.2

v3.4.2 (2024-05-15)

🐛 Fixes

• #8542 State from interrupted pipelines or using the --merge-state flag no longer causes crashes

📚 Documentation Improvements

• #8544 Added YAML docs examples of global and per-plugin python executable option

v3.4.1

v3.4.1 (2024-05-06)

🐛 Fixes

• #8509 Mapper configuration is no longer ignored and is passed to direct mapper invocations as well as mappings in pipelines
• #8527 Made the output of meltano select ... list consistent between different Python versions
• #8213 meltano config <plugin> test false-negative on Windows -- Thanks @ReubenFrankel!
• #8508 Cron regex pattern on meltano.schema.json -- Thanks @sabino!

⚙️ Under the Hood

• #8470 Started enforcing usage of structlog.stdlib.get_logger over logging.getLogger and banned use of the root logger
• #8510 Plugin installation status is now logged instead of printed

📚 Documentation Improvements

• #8524 Changed references of Airflow orchestrator to utility
• #8506 Document venv.backend usage to configure uv

v3.4.0

v3.4.0 (2024-04-18)

• Docker Hub 🐋
• PyPI 🐍

Highlights

• Meltano can now use uv to manage plugin installation. Start using it with

  MELTANO_VENV_BACKEND=uv meltano install --clean

  Other ways of configuring it are available. Let us know if you run into any issues!

• Plugins of multiple types can now be installed with a single command

  meltano install - tap-hubspot meltano-map-transformer target-postgres

• Shout out to @nkclemson for their work on enabling new Ruff rules and addressing the violations 🎉

✨ New

• #8459 meltano run now has a --run-id option to allow for custom run UUIDs
• #8465 Support uv as an optional virtualenv backend
• #8355 Support installing multiple plugins of any type -- Thanks @ReubenFrankel!

🐛 Fixes

• #8486 "kind: {kind} is deprecated..." is no longer logged if the corresponding replacement is in place
• #8489 Environment variables from .env are now passed to the plugin installation subprocesses
• #8490 An explicit error message is now logged when Meltano fails to retrieve tap state from the state backend
• #8447 meltano run no longer creates empty venv plugin directories for inherited plugins
• #8446 Added run_id and job_name properties to meltano run log messages

⚙️ Under the Hood

• #8499 Only lookup uv executable once
• #8469 Fixed some log serialization issues with meltano.core.proj…ec_plugins_service.DefinitionSource and meltano.core.plugin.project_plugin.ProjectPlugin

📚 Documentation Improvements

• #8449 Fixed a broken link to custom plugin definitions in the project concept page
• #8432 Added the 2024-Q1 changelog

v3.3.2

v3.3.2 (2024-03-06)

🐛 Fixes

• #8436 Terminal output from plugin installation is now safely decoded
• #8381 Print Plugin definition is already locked to stdout instead of stderr
• #8399 Handle non-UTF-8 lines in plugin output for logging -- Thanks @nkclemson!

📚 Documentation Improvements

• #8434 Added a note about using the --require-virtualenv flag for installing Meltano with pip
• #8433 Removed some stale references to the old API

v3.3.1

v3.3.1 (2024-01-26)

🐛 Fixes

• #8379 --from-ref for a plugin definition missing a variant -- Thanks @ReubenFrankel!

📚 Documentation Improvements

• #8378 Mention --from-ref in the custom extractor guide -- Thanks @ReubenFrankel!

v3.3.0

v3.3.0 (2024-01-23)

Highlights

• Some workflows should see increased performance thanks to #8343. Thanks to @TyShkan for reporting, investigating and sharing the profiling results!
• Warnings are now logged whenever a selected stream or property is not present in the tap's catalog.

✨ New

• #8351 Warn if select rules are not used on catalog -- Thanks @haleemur!
• #8176 Add new --update flag to meltano add to re-add plugins to a project -- Thanks @ReubenFrankel!
• #7874 Add setting sensitive flag -- Thanks @ReubenFrankel!

🐛 Fixes

• #8365 Retain stderr in meltano config <plugin> test -- Thanks @ReubenFrankel!
• #8354 Emit pip installation errors

⚡ Performance Improvements

• #8343 Cache Project.dotenv_env

📚 Documentation Improvements

• #8306 Add changelog updates

v3.2.0

v3.2.0 (2023-12-05)

Highlights

• The new --from-file option lets you pass setting values to meltano config from files and STDIN, thanks to @XshubhamX!

• The new option --merge-state lets you run partial full-table pipelines, for example when you want to refresh a single stream without losing the state of other streams.

• Environment variables are now expanded in array settings, for example MY_BUCKET will now expanded in the example below:

  plugins:
    extractors:
    - name: tap-spreadsheets-anywhere
      variant: ets
      pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git smart_open[s3]
      config:
        tables:
        - name: customers
          path: s3://${MY_BUCKET}
          pattern: files/customers\.csv\.*gz
          start_date: '2017-05-01T00:00:00Z'
          key_properties: [id]

✨ New

• #8184 Log Meltano version and Operating System at invocation of CLI -- Thanks @ashu565!
• #8228 Support setting configuration from files and STDIN -- Thanks @XshubhamX!
• #8194 Allow escaping stream names with periods in select rules
• #8241 Support Python 3.12
• #8215 Support authenticating to Azure state backend without a connection string -- Thanks @XshubhamX!
• #8258 Add a --merge-state flag to meltano run to merge the current pipeline state with that of the latest run
• #8268 Expand environment variables in array setting values

🐛 Fixes

• #8249 Display special characters in column names in the output of meltano select --list
• #8227 Validate input to settings with kind: options
• #8232 Interval validation for meltano schedule set -- Thanks @XshubhamX!
• #8225 Redirect tap's stderr to /dev/null during a configuration test -- Thanks @raiatul14!
• #8301 Interactive config now indicates that you're typing a redacted value -- Thanks @abastola0!

📚 Documentation Improvements

• #8267 Add Q4 Changelog
• #8260 Add information about database schema destination -- Thanks @EChaffraix!
• #8259 Fix links in v3 migration guide
• #8251 Fix dbt model in part3.mdx -- Thanks @diegoquintanav!
• #8214 Document other supported ways of configuring an S3 endpoint URL
• #8211 Sync sidebar order with index page link-lists, remove broken /index links -- Thanks @mjsqu!
• #8209 Remove broken link (reference/index) and order links to match sidebar -- Thanks @mjsqu!
• #8210 Add tabs for all remaining env/cli/config settings -- Thanks @mjsqu!
• #8201 Use tabs for examples on settings page -- Thanks @mjsqu!
• #8200 Update contribution guidelines -- Thanks @mjsqu!
• #8155 Add september changelog

Full Changelog: v3.1.0...v3.2.0

v3.1.0

v3.1.0 (2023-09-26)

✨ New

• #8169 Added --force-install CLI flag for meltano add command. -- Thanks @XshubhamX!
• #8145 Save pip logs to a file in the system directory
• #7907 Add plugin from ref -- Thanks @ReubenFrankel!

🐛 Fixes

• #8179 Removed the --verbose option from CLI, which no longer had any effect -- Thanks @arorarohan981!
• #8044 Better error message when Azure connection string is missing
• #8115 Do not perform destructive state ops when answering 'no' in prompt

📚 Documentation Improvements

• #8178 Top-level ordered lists are now rendered correctly in docs
• #8156 Added docs for --cwd CLI option -- Thanks @XshubhamX!
• #8143 Remove Cloud announcement banner
• #8113 Update Cloud "Getting Started" guide to include the lock command
• #8111 Add example of database URI for SQL Server -- Thanks @wesseljt!
• #8087 Add cloud state command