Valentine's Day let you down? That's OK. You'll love this. Airbyte 1.5.0 was released on February 20, 2025. We’re excited to share new improvements and changes to the Airbyte platform. ## 🚀 Platform Changes These changes improve Airbyte for all Self-Managed users. ### Declarative OAuth 2.0 in the connector builder If you're building a connector to an API and want to authenticate with OAuth, you can now authenticate directly in the Connector Builder. This simplified flow ensures you can start building connectors to APIs that support OAuth without writing code to authenticate yourself and provide Airbyte with tokens. OAuth 2.0 is also supported in the low-code connector development kit (CDK), where you can define your authentication using YAML. To learn more, see the authentication documentation: [No-code Connector Builder](../connector-development/connector-builder-ui/authentication#oauth) | [Low-code CDK](../connector-development/config-based/advanced-topics/oauth).  ### Connection tags You can now classify your connections with tags. As time goes on, your number of connections tends to increase. This popular classification tool gives you the power to organize connections based on any set of criteria you like, then filter for the ones you want to see. [**Learn more >**](https://docs.airbyte.com/using-airbyte/tagging)  ### Configure workspace notifications in the API Use the Airbyte API to view and change email and webhook notification settings in your workspace. Previously, this was only available in Airbyte's user interface. - [List workspaces](https://reference.airbyte.com/reference/listworkspaces) returns a notifications object with details about each workspace's notification settings. - [Get workspace details](https://reference.airbyte.com/reference/getworkspace) returns a notifications object with details about that workspace's notification settings. - [Create a workspace](https://reference.airbyte.com/reference/createworkspace) has a notifications object parameter so you can configure notifications when you create the workspace. - [Update a workspace](https://reference.airbyte.com/reference/updateworkspace) has a notifications object parameter so you can configure notifications later. Use notifications if you want to be alerted to important events in Airbyte, like failed syncs, schema changes, and mandatory connector upgrades. [**Learn more about notifications >**](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications) ### Full Refresh - Overwrite + Deduped in the API Use the API to set "Full Refresh - Overwrite + Deduped" as your [sync mode](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/) when [creating](https://reference.airbyte.com/reference/createconnection) and [updating](https://reference.airbyte.com/reference/patchconnection) a connection. Previously, this sync mode was only available in Airbyte's user interface. ## 🚀 Self-Managed Enterprise Changes These changes bring new capabilities to Airbyte's Self-Managed Enterprise customers. ### Mappings in the UI We introduced mappings in the API in version 1.3, and it's now available in Airbyte's user interface, too. Use mappings to hash, encrypt, and rename fields, and filter rows. You set up mappings on each stream, ensuring your source data arrives in your destination in a more meaningful way. [**Learn more >**](https://docs.airbyte.com/using-airbyte/mappings)  ### OpenTelemetry (OTel) metrics monitoring Self-Managed Enterprise now generates a number of crucial metrics about syncs and volumes of data moved. You can configure Airbyte to send telemetry data to an OTel collector endpoint so you can consume these metrics in your downstream monitoring tool of choice. The following metrics are available: - Sync details - Sync duration - Volume of data moved - API requests To start sending metrics, update your `values.yaml` file with the following configurations. [**Learn more >**](../operator-guides/collecting-metrics) ```yaml title="values.yaml" global: edition: enterprise # This is an enterprise-only feature metrics: enabled: true otlp: enabled: true collectorEndpoint: "YOUR_ENDPOINT" # The OTel collector endpoint Airbyte sends metrics to. You configure this endpoint outside of Airbyte as part of your OTel deployment. ``` ### Resource allocation on connectors In Self-Managed Enterprise, you can now define resource allocations for individual connectors as part of that connector's configuration, both in the user interface and the API. Airbyte's default CPU and memory allocations aren't always appropriate for every situation, and different connectors have different resource requirements. This can make it challenging to run a large number of concurrent syncs. Historically, you had to tweak a number of variables to configure this. Now, you can set CPU and memory allocation when you set up a source or destination, and it applies to all connections using that connector. [**Learn more >**](https://docs.airbyte.com/operator-guides/configuring-connector-resources)  :::note Resource allocation can still be set on specific connections, on all connectors of a type using `resourceRequirements`, and using environment variables. In cases where resource allocation is defined multiple times, there is an order of precedence. Narrower definitions have higher precedence. From highest to lowest: **Connection** > **Connector** > **Connector yaml definition** > **environment variables**. See [the documentation](https://docs.airbyte.com/operator-guides/configuring-connector-resources) to learn more about precedence. ::: ## 🐛 Bug fixes - [Fixed issue #46097](https://github.com/airbytehq/airbyte/issues/46097). When using AWS Secrets Manager, updating a connector that used secrets caused its secrets to be deleted. For connectors with secrets that were deleted because of the bug, upgrade to 1.5 and then update your connector's secrets. They will persist correctly. For sources/destinations that were not affected, upgrading to 1.5 prevents the issue. ## Other Changes Docusaurus is upgraded to version 3.7. Local docs site builds are a bit faster.

# Airbyte 1.4.0 Happy new year! Airbyte version 1.4.0 was released on January 17, 2025. We’re excited to share new improvements and changes to the Airbyte platform. ## 🚀 Platform Changes Platform changes improve Airbyte for everyone with a self-managed instance. ### Configure the schema refresh rate At the start of a sync, Airbyte occasionally re-runs schema discovery to detect changes in your source, such as new fields or a change in column type. But running schema discovery has a performance cost, and may not always be necessary before every sync. You may now use a new toggle, `DISCOVER_REFRESH_WINDOW_MINUTES`, to set the minimum number of minutes Airbyte will wait to refresh the schema for your sources. By setting a larger number, you run automatic schema detection less frequently, which can result in gains in sync performance. The default configuration in Airbyte Open Source is `1440`, which refreshes schemas every 24 hours. The lowest interval you can set is `1`, which refreshes schemas before every sync. ```yml title="values.yaml" worker: env_vars: DISCOVER_REFRESH_WINDOW_MINUTES: 1440 # Airbyte automatically refreshes schemas no more than once per day (1440 minutes). ``` Set this to 0 to disable automatic schema refreshes. All schema refreshes will need to be done manually via the Airbyte UI or API. ```yml title="values.yaml" worker: env_vars: DISCOVER_REFRESH_WINDOW_MINUTES: 0 # Airbyte does not automatically detect schema changes. ``` ### Connectors support custom image registries Connectors can now use custom image registries rather than Airbyte’s public Docker registry. If you configure Airbyte to use a custom image registry, it now automatically uses that registry for connector images as well as platform images. Previously, only platform images supported this. In this example, we set Airbyte’s `values.yaml` file to pull all images from GitHub. ```yml title="values.yaml" global: image: registry: ghcr.io/NAMESPACE ``` You must ensure copies of platform and connector images are available in your custom image registry. [Learn how to set up custom image registries](../deploying-airbyte/integrations/custom-image-registries). [Custom Docker connectors](../operator-guides/using-custom-connectors/) in your workspace that specify an image using a fully qualified domain name (for example, `example.com/airbyte/your-custom-source`) ignore your configured custom image registry and pull images from the domain specified by that connector. ## 🐛 Bug fixes - **Reduced resource consumption**: Instances of Airbyte running multiple connections at a time now consume fewer resources, thanks to optimizations to the `airbyte-worker` pod. - **Canceled syncs do not rerun until the next scheduled sync**: If you previously cancelled an in-progress sync, and the next sync as configured in the connection frequency was behind schedule, Airbyte would immediately start a new sync. In practice, many users had to choose to ‘Cancel Sync’ twice in a row to stop moving data. Airbyte now automatically waits until the next scheduled sync to move data. - **Reduce rate limit errors from the `airbyte-cron` service**: We fixed an [issue](https://github.com/airbytehq/airbyte/issues/30691) reported by the community that caused excessive rate limit errors on the `airbyte-cron` pod when users scheduled connections using the Airbyte CRON capability. - **Autorecovery for hanging connections**: Airbyte now packages a service to detect connections blocked by the unlikely event that a sync becomes stuck, and remains in a perpetual ‘in-progress’ state without moving data. This heartbeat service will detect syncs that are hanging, and automatically create a new job attempt.

It's already December! We are excited to release the following set of improvements and changes. ## 🚀 Platform Changes - **Improved Log Viewer and Filtering**: You can now quickly filter to error or warning logs when needed, as well as by log source - such as the source connector, destination connector, or platform. Together, this allows you to quickly diagnose issues with Airbyte connections if they arise, reducing time to resolution on issues.  - **OpenShift Support**: Starting today, Airbyte supports deployments to OpenShift. This is the culmination of a journey over much of the past year: we've updated our platform and connectors to be rootless, we've improved pod to pod communication within our platform, and made a wide number of security improvements. To succeed, the user running helm deployment commands needs to be a `cluster-admin` user. Your Kubernetes namespace will also require the following annotations to ensure that all of the Airbyte resources run as UID 1000 and GID 1000. If you run into any edge cases or issues deploying to OpenShift, as always, please let us know by opening a GitHub issue. ```yaml openshift.io/sa.scc.supplemental-groups: 1000/1 openshift.io/sa.scc.uid-range: 1000/1 ``` ## 🚀 Self-Managed Enterprise Changes - **Compliance Mappers in Airbyte API**: Airbyte Self-Managed Enterprise now supports field hashing, field encryption (with self-managed encryption keys), field renaming and row filtering for connections from a common interface available in the Airbyte API. This functionality is coming soon to the Airbyte UI. To get started, see our [API documentation](https://reference.airbyte.com/reference/patchconnection). 

Happy Halloween! We are excited to release the following set of improvements and changes. ## 🚀 Platform Changes - **File Transfers**: Airbyte now supports transfering unstructured text data, non-text data and compressed files. This capability is in early access, and currently only available for moving data from the "SFTP Bulk" source to the "S3" destination. Support in the "S3" source is coming soon. File transfers allow you to copy raw files in Airbyte without parsing their contents. Bits are copied into the destination exactly as they appeared in the source. This is recommended for use with unstructured text data, non-text and compressed files. This is an entirely new way of moving data through Airbyte, as all pre-existing data movement methods involved parsing individual records.  - **Custom Image Registry Support**: Introduces support for specifying a custom Docker registry from which all platform images are pulled. This should simplify helm chart configurations for users who require images to be pulled from internally approved image repositories. Configuring the following will prefix all docker images with `my-registry.foo.com`. The value provided in the `registry` field must have a trailing slash: ```yaml global: image: registry: my-registry.foo.com/ ``` - **XML Support in the Connector Builder**: Airbyte now supports custom API connectors built via the Airbyte Connector Builder which return responses as XML format.  ## 🚀 Self-Managed Enterprise Changes - **Runner RBAC Role**: Users provisioned with the 'Runner' role are able to start or stop syncs, and run backfills for individual connections (in the workspaces where they've been granted Runner permissions). Outside of these explicit tasks, users with the 'Runner' role have an entirely read-only experience (Self-Managed Enterprise only).  - **Self-Managed Diagnostics**: Diagnostics have been expanded to include deployment statistics. Diagnostics can be downloaded through the Organization Settings page. This exports a .zip file which can be easily shared with Airbyte Support (Self-Managed Enterprise only). - **Removing SSO Users**: You may now remove SSO users from your Airbyte Organization. This removes them from the list of Organization Members.

## 🚀 Features The 1.1.0 release includes the following enhancements and bug fixes. ## Platform Releases - Adds field hashing for Self-Managed Enterprise - Adds the [Connection Timeline](../../cloud/managing-airbyte-cloud/review-connection-timeline), replacing the Job History for a connection. For assistance with the migration of existing jobs to the new timeline events, follow our [guide](https://github.com/airbytehq/airbyte/discussions/46319). - Redesigned the schema tab to improve the selection & deselection of streams and fields. Sync modes, cursor fields, and primary keys are also selected here.  - Custom Docker-based connectors can now be renamed and deleted ## Support Experience - Diagnostics can be downloaded through the Organization Settings page, which will export a JSON file to facilitate sharing with Support. This JSON file includes relevant infrastructure and connection information. (Self-Managed Enterprise only) ## Billing & Licenses - Self-Managed Enterprise users can now view directly in the UI when the license is expiring. ## Changelog See the [Full Changelog](https://github.com/airbytehq/airbyte/releases)

## airbyte v1.0.0 This page includes new features and improvements to the Airbyte products we're excited to share as a part of the 1.0 release. Airbyte v1.0 contains many improvements and additions to enhance the reliability, scalability, and uses of Airbyte. Moving forward, Airbyte will release official new platform versions on a monthly cadence, with the associated [changelog](https://github.com/airbytehq/airbyte/releases). ## Ease of deployment - [`abctl`](https://github.com/airbytehq/abctl/releases) is the easiest, quickest way to get started with Airbyte Self-managed. See our [quickstart docs](using-airbyte/getting-started/oss-quickstart#2-run-airbyte) for more details. - [Native authentication](/deploying-airbyte/integrations/authentication) by email and password is available so that any instance of Airbyte is secure by default. ## Proven Reliability - [Automatic detection of dropped records](https://airbyte.com/blog/automatic-detection-of-dropped-records) ensures data is passed without fail through the Airbyte platform. - Enhanced our [notification suite](https://airbyte.com/blog/airbyte-notifications-and-webhooks-effortless-etl-jobs-monitoring) to enable webhook integrations and added more contextual information about sync failures or schema changes. - Connections now offer rate limited messaging, improved error handling, and live monitoring of ongoing syncs.  - Many sources, in particular S3, have become faster. With CDK 2.0, the protocol now uses Pydantic V2 and removes the serialization of each record. We also improved Databricks and S3 destinations to enable faster sync speeds, checkpointing, and modernization to our Destinations V2 framework. ## Resiliency at Scale - [Refreshes](operator-guides/refreshes) bring an improved experience to resyncing all of your data again. This enables data to never be deleted from final tables during a historical resyncing of data. - [Resumable Full Refresh](https://airbyte.com/blog/resumable-full-refresh-building-resilient-systems-for-syncing-data) allows for large streams syncing in Full Refresh to sync without failures. - Our database sources added [resumability](/operator-guides/refreshes#resumability) to reduce failures and stay resilient when incoming data is inconsistent. We gracefully handle record-level error in the destination field `_airbyte_meta.errrors` for large records or mistyped data. Read more in our [typing & deduping documentation](/using-airbyte/core-concepts/typing-deduping#_airbyte_meta-errors). - [Large initial CDC syncs](https://airbyte.com/blog/supporting-very-large-cdc-syncs-with-wass) are gracefully handled by consuming WAL and acknowledges logs periodically to ensure disk space is freed in a timely manner. - [Workloads](understanding-airbyte/jobs#workloads) provide a more scalable and reliable architecture to run sync jobs by separating scheduling and orchestration from data movement tasks. This [improvement](https://airbyte.com/blog/introducing-workloads-how-airbyte-1-0-orchestrates-data-movement-jobs) unlocks more automated management of workloads by managing job spikes and enables horizonal scaling (for Cloud and Enterprise users). ## Tackling the long-tail of connectors - We launched our connector Marketplace for our community-maintained connectors. [Automated testing](https://airbyte.com/blog/how-we-test-airbyte-and-marketplace-connectors) ensures our connectors stay high quality, and most connectors have been migrated to low/no-code. - The [Connector Builder](https://airbyte.com/blog/maintaining-hundreds-of-api-connectors-with-the-low-code-cdk-and-connector-builder) seamlessly switches betweem UI and YAML modes, supports custom components, and contains extensive testing and error handling. - AI Assist now builds connectors from scratch for you with just a link to the API docs. - Anyone can now contribute new connectors directly from the builder to add a new connector directly to Airbyte's Marketplace. Additionally, connectors can be forked directly from the UI for faster edits, ensuring our connectors stay up-to-date. - Airbyte supports writing to several vector store destinations, including pgVector, Pinecone, and Snowflake Cortex. ## Self-Managed Enterprise Self-Managed Enterprise extends on Airbyte 1.0 by introducing new classes of functionality: multitenancy and access management, enterprise source connectors , sensitive data masking and enterprise support – all while data never leaves your infrastructure. - Support for Okta and OpenID Connect (OIDC) SSO - Availability of [Role-based Access Control](/access-management/rbac) and multitenancy - Columns can now be hashed to protect sensitive data ## Announcements - To ensure adherence to security best practices, Airbyte is migrating all connectors to [non-root versions](https://github.com/airbytehq/airbyte/discussions/44924). It is highly recommended that you upgrade your platform version to [v0.63.9](https://github.com/airbytehq/airbyte-platform/releases/tag/v0.63.9) or later before October 2024 to ensure your syncs continue to succeed. - As we prepare to deprecate Docker Compose, we published a [migration guide](using-airbyte/getting-started/oss-quickstart#migrating-from-docker-compose-optional) for those migrating from Docker Compose to abctl.

2e62443f1c [source-postgres/mysql/mssql/mongodb-v2] Use latest CDK to adopt latest apache sshd mina to handle tcpkeepalive requests (#45639) 398a1bba71 source-buzzsprout contribution from btkcodedev (#45608) ed214d045d [cdk] upgrade apache mina sshd to 2.13.2 to handle openssh tcpkeepalive request (#45638) 5f93772798 [ISSUE_TEMPLATE] replace deprecated docker template for ai-assist (#45629) 6d74db749c change signature of orphanedThreadFilter (#45469) 6469111428 [source-mysql-v2] cdk and spec change for ssl (#45351)

6502f92096 🐛 Source Greenhouse: faster check stream (#45625) 4027d0e69e [docs] add troubleshooting abctl page + move trb. from quickstarts (#45460) 28f82140ab 🤖 Cut version 5.6.0 of source-declarative-manifest c5cff577d6 🤖 minor bump Python CDK to version 5.6.0 199a8078f2 [airbyte-cdk] Decouple request_options_provider from datetime_based_cursor + concurrent_cursor features for low-code (#45413)