Gcp composer airflow 2.0 install#
Gcp composer airflow 2.0 upgrade#

You can see which worker types are available for your cluster in the Clusters menu of the Cloud UI. : Specify the worker type for your default worker queue. : Specify the same minimum and maximum worker count as in source Airflow environment. : Specify name of the Astro cluster in which you want to create this Deployment. : Specify the same number of schedulers as in your source Airflow environment. Use these recommendations as a starting point, then adjust your resource usage after tracking your performance on Astro. Although you might not need more CPU or memory than what's recommended, some environments might parse DAGs slower than in your source Airflow environment to start. ² Some of the following recommendations for CPU and memory are smaller than what you have in the equivalent source environment. Use the following table to determine how many AUs you need based on the size of your source Airflow environment. An AU is a unit of CPU and memory allocated to each scheduler in a Deployment. Scheduler: Set your scheduler size in Astronomer Units (AU). See Manage Hybrid clusters.Įxecutor: Choose the same executor as in your source Airflow environment. If you don't have the option of choosing between standard or dedicated, that means you are an Astro Hybrid user and must choose a cluster that has been configured for your Organization. To configure and use dedicated clusters, see Create a dedicated cluster. If you don't have specific networking or cloud requirements, Astronomer recommends using the default Standard cluster configurations. For a complete list of supported Airflow versions, see Astro Runtime release and lifecycle schedule.ĭescription: (Optional) Enter a description for your Deployment.Ĭluster: Choose whether you want to run your Deployment in a Standard cluster or Dedicated cluster. There are no known risks for upgrading directly from Airflow 2.0 to Airflow 2.1.1 during migration. ¹The earliest available Airflow version on Astro Runtime is 2.1.1.

Gcp composer airflow 2.0 install#

To install the Starship plugin on your Cloud Composer 1 or Cloud Composer 2 instance, install the astronomer-starship package in your source Airflow environment.

See the following table for information on which versions of Starship are available depending on your source Airflow environment: Source Airflow environment The Starship migration utility works as a plugin with a user interface, or as an Airflow operator if you are migrating from a more restricted Airflow environment. The Astronomer Starship migration utility connects your source Airflow environment to your Astro Deployment and migrates your Airflow connections, Airflow variables, environment variables, and DAGs. You can additionally use the gcloud CLI to expedite some steps in this guide.

Gcp composer airflow 2.0 upgrade#

Astronomer professional services can help you with the upgrade process. (Optional) Permission to create new CI/CD pipelines.Īll source Airflow environments on 1.x need to be upgraded to at least Airflow 2.0 before you can migrate them.(Optional) Access to your secrets backend.Permission to create new repositories on your source control tool.Read access to any source control tool that hosts your current Airflow code, such as GitHub.

Read access to any cloud storage buckets that store your DAGs.

Read access to the source Airflow environment.

A source Airflow environment on Airflow 2 or later.

On the cloud service from which you're migrating, ensure that you have: On your local machine, make sure you have:

(Optional) You configured your identity provider (IdP).

(Optional) You created a network connection from Astro to your external cloud resources.

You have an Astro user account and can log in to Astro.

Migrate DAGs and additional Airflow components from your source Airflow environment.īefore starting the migration, ensure that the following are true:.

Migrate metadata from your source Airflow environment.

Prepare your source Airflow and set up your Astro Airflow environment.

To complete the migration process, you will:

This is where you'll find instructions for migrating an Airflow environment from Google Cloud Composer (GCC) to Astro. Migrate to Astro from Google Cloud Composer