g1-g2-ostack-cloud-migration

e-INFRA CZ G1 to G2 OpenStack cloud workload migration.

This is automation CI pipeline is core of the G1 to G2 migration process. More details of the progress can be found here.

G1 e-INFRA CZ / MetaCentrum OpenStack cloud is available here. G2 Brno e-INFRA CZ OpenStack is available here.

Additional email communication is being generated in Knowledgebase.

Migration checklist

• Describe to user migration steps and get acknowledge that source servers may be stopped. Are there SLAs?
• Define date and time of the specific project workload migration
• Define who is contack person (AAI group manager) and/or someone else
• (group project only) Define expiration of the project
• Decide what resources has to be migrated.
• Decide whether migrate workload at once or step by step.
• Are there any data in object store?
• Are there any DNS records on FIPs?
• Do you need to migrate non running VMs?
• Do you need to migrate not-attached openstack volumes? (wording)
• Do you request block of FIPs or random is ok for you?
• To agree on how user gives final migration signoff handshake.

How to launch G1 to G2 migration

1. Go to the pipenines page

2. Launch your own pipeline by clicking button Run Pipeline

3. Click on the next page just click again button Run Pipeline without providing any configuration variables

4. Click on displayed ostack-einfra_cz-project-migration job

5. Provide below listed migration input arguments as pipeline variables and click Run Job

6. Pipeline should finish within 10 - 120 minutes based on number of cloud entities

Input arguments are:

• (required) PROJECT_NAME
  • Common project name in G1 and in G2 OpenStack clouds.
• (required) VALIDATION_SERVER_ID
  • id of any existing server in the G1 cloud
• (optional) MIGRATE_EXPLICIT_SERVER_NAMES
  • migrate just part of the source G1 project. You may specify explicitly server names delimited by space or comma. This option can be used for explicit migration of inactive VM servers as those are by default not migrated.
• (optional) MIGRATE_EXPLICIT_VOLUME_NAMES
  • all volumes connected to migrated VM servers are migrated. Unused i.e. not attached volumes are by default skipped. This option allows migration of not-attached volumes.
• (optional) PROJECT_MIGRATOR_EXTRA_ARGS
  • the expert mode arguments

What are the cloud migration steps?

1. Cloud team or cloud user initiates cloud workload migration
2. All necessary information are gathered (see above checklist)
3. Migration pipeline is configured and launched a. Connects to source (G1) and destination (G2) clouds as privileged user[s] and tests project exists in both clouds b. Privileged users are reconnecting under scope of source and destination project c. Server entities are listed in source and destination cloud d. In main loop migrator iterates over source servers and
   • receives all necessary information
   • stops the source virtualized server
   • transfers all attached volumes from source (G1) to destination (G2) cloud using low-level ceph RBD operations
   • starts back the source virtualized server
   • starts up the destination server in destination cloud from the cloned volumes (so using mapped flavor, network, volumes, keypair, security-groups)
   • associate FIP address if source VM has one e. Configured extra volumes are transfered to destination cloud
4. Cloud user validates workload is properly migrated
5. Source cloud resources are made inactive for 2-4 weeks by cloud team
6. Source cloud resources are archived

Migration limitations, additional notes

• Servers in SHUTOFF are not migrated by default, apply PROJECT_MIGRATOR_EXTRA_ARGS=--migrate-inactive-servers=true.
• Project volumes which are not attached are not migrated by default, see MIGRATE_EXPLICIT_VOLUME_NAMES option description above.
• FIP addresses in source and destination cloud will differ
• cloud resources in destination cloud have have same names except security groups which are prefixed migrated-*
• Retry of the workload migration is possible when resources are removed in destination cloud.
• As of writing (2024-07-09) ephemeral VM servers cannot be migrated.

Tools needed for manual execution

• https://gitlab.ics.muni.cz/cloud/g2/common-cloud-entities/-/blob/master/ci/requirements.apt?ref_type=heads
• https://gitlab.ics.muni.cz/cloud/g1-g2-ostack-cloud-migration/-/blob/master/ci/requirements.apt?ref_type=heads
• https://gitlab.ics.muni.cz/cloud/g2/common-cloud-entities/-/blob/master/ci/requirements.pip?ref_type=heads
• https://gitlab.ics.muni.cz/cloud/g1-g2-ostack-cloud-migration/-/blob/master/ci/requirements.pip

Known migration / migrator issues

• project-migrator.py
  • [A] as openstack entities work with entity names it has troubles with migration of networks/subnets/routers with empty name (``)
    • trigered by job#1560463.
    • as workaround feel free to change names in G1 OpenStack cloud for instance openstack subnet set --name "FormR - Subnet" 3aafeb98-b4cb-4b42-9842-be9e2b1e4d6e and re-run the migration
  • [B] found on 2024-07-04 that VMs with multiple ports from same network are not added
    • as workaround feel free to add it manually

# original
[freznicek@lenovo-t14 ~ 0]$ openstack port list --server 1da4acd8-18a1-4e53-ad2f-b1fdfbf5aae2
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------------------+--------+
| ID                                   | Name | MAC Address       | Fixed IP Addresses                                                                                  | Status |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------------------+--------+
| 3dd88f22-87b5-4405-8691-6ed71139cb19 |      | fa:16:3e:74:aa:6d | ip_address='2001:718:801:432:f816:3eff:fe74:aa6d', subnet_id='3fbfcca9-fd7e-4fc1-98c0-8231c5029206' | ACTIVE |
| 6650fb3d-5245-4014-9d88-bd288da83184 |      | fa:16:3e:ff:2f:3a | ip_address='2001:718:801:432:f816:3eff:feff:2f3a', subnet_id='3fbfcca9-fd7e-4fc1-98c0-8231c5029206' | ACTIVE |
| a92ba85e-0bd1-40c9-b92a-d112eb821575 |      | fa:16:3e:eb:06:ff | ip_address='192.168.0.37', subnet_id='7d80ec40-9ce1-4bad-90a8-5149f1739daf'                         | ACTIVE |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------------------+--------+
# migrated
[freznicek@lenovo-t14 ~ 0]$ openstack port list --server 8b945e9a-6605-46ef-b469-d1c3fd59d8bb
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------------------+--------+
| ID                                   | Name | MAC Address       | Fixed IP Addresses                                                                                  | Status |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------------------+--------+
| 14fedf2a-6a3d-4a1b-859f-dc449da24c40 |      | fa:16:3e:eb:06:ff | ip_address='192.168.0.37', subnet_id='add69a21-374e-4f51-bda3-8fd2618ac06f'                         | ACTIVE |
| bc62a55f-2fe2-48b7-a9a6-e35d607bce12 |      | fa:16:3e:74:aa:6d | ip_address='2001:718:801:43b:f816:3eff:fe74:aa6d', subnet_id='05c6ae86-9d11-4d3f-9fce-ba387c08345b' | ACTIVE |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------------------------+--------+
openstack port create --network external-ipv6-general-public  --mac-address fa:16:3e:ff:2f:3a --enable --host crf-010-ics.priv.g2.cloud.muni.cz --project meta-economic-experiments --security-group  907e6c79-7672-4d72-abf0-217ea019a983 dExperiment-manual-port
...
openstack server add port 8b945e9a-6605-46ef-b469-d1c3fd59d8bb edf2a43b-8f28-4319-85ae-987af3fc4347
...
[freznicek@lenovo-t14 ~ 0]$ openstack port list --server 8b945e9a-6605-46ef-b469-d1c3fd59d8bb
+--------------------------------------+-------------------------+-------------------+-----------------------------------------------------------------------------------------------------+--------+
| ID                                   | Name                    | MAC Address       | Fixed IP Addresses                                                                                  | Status |
+--------------------------------------+-------------------------+-------------------+-----------------------------------------------------------------------------------------------------+--------+
| bc62a55f-2fe2-48b7-a9a6-e35d607bce12 |                         | fa:16:3e:74:aa:6d | ip_address='2001:718:801:43b:f816:3eff:fe74:aa6d', subnet_id='05c6ae86-9d11-4d3f-9fce-ba387c08345b' | ACTIVE |
| edf2a43b-8f28-4319-85ae-987af3fc4347 | dExperiment-manual-port | fa:16:3e:ff:2f:3a | ip_address='2001:718:801:43b:f816:3eff:feff:2f3a', subnet_id='05c6ae86-9d11-4d3f-9fce-ba387c08345b' | ACTIVE |
| 14fedf2a-6a3d-4a1b-859f-dc449da24c40 |                         | fa:16:3e:eb:06:ff | ip_address='192.168.0.37', subnet_id='add69a21-374e-4f51-bda3-8fd2618ac06f'                         | ACTIVE |
+--------------------------------------+-------------------------+-------------------+-----------------------------------------------------------------------------------------------------+--------+

• [C] migration of the cinder volume snapshots / backups are not automated
  • use --migrate-volume-snapshots to get snapshot data
  • you need to perform following steps to get them visible in openstack