Migration from on-premises deployment to SaaS instance

Before we start

Migration might be time-consuming. We strongly advise planning the downtime with the QA team before the process starts.

Prerequisites :: On-prem instance release

Data migration is possible from Allure TestOps version 24 onwards. Preferably to have the most recent release deployed. We cannot migrate data from versions 3 or 4. If your deployed version is 3 or 4, you will need to upgrade it to at least version 5, then to the most recent version, and verify that everything is working correctly.

Prerequisites :: Availability of SaaS instance

Starting of the migration process requires the presence of a cloud instance, so, please, request one before we could start of the migration process. Please provide the payment information for the created instance (Administration → License → Manage Subscription) to prevent any data loss and instance suspension.

Prerequisites :: Compacting the data to be transferred

Accordingly to the GTC of SaaS offering, the storage is now limited by 60 Gbytes for the whole instance, so we need to make sure that the data we are going to transwer does not exceed this number. Preferably migration is to be perfomed with minimal historical data.

The easiest way to avoid moving old data to the cloud is the deletion of old launches.

Limitations

All existing data from the SaaS instance will be deleted and replaced by the data from the on-prem instance. Merging of the data is not possible.
Data from the on-prem instance can be restored only once. There is no procedure and capability to restore the data, and then provide some delta in addition to the restored data.

Cleaning the launches

The upload, download and restoiration of data processes can take many hours if you have an old Allure TestOps deployment you've been using for months or years – copying of the database with old results could be a very long process and will increase the downtime.

The historical data is stores as Launches, these contain test results details in the database and could consume a lot of space. Launches can be cleaned using this tool.

You can to execute the cleaner tool on a VM or on your local workstation.

The following execution command is recommended.

docker run -e "ALLURE_ENDPOINT=http://localhost:8080" \
           -e "ALLURE_USERNAME=REPLACE_WITH_YOUR_USERNAME" \
           -e "ALLURE_PASSWORD=REPLACE_WITH_YOUR_PASSWORD" \
           -e "PROJECT_ID=REPLACE_WITH_PROJECT_ID" \
           -e "LAUNCH_FILTER=true" \
           -e "LAUNCH_CREATEDBEFORE=30d 0h 0m" \
           ghcr.io/eroshenkoam/allure-testops-utils clean-launches

You need to pprovide the username and password (not API key) of a user with at least write permissions in a project, and the project ID. You need to execute the tool for each project.

The string

-e "LAUNCH_CREATEDBEFORE=30d 0h 0m"

defines that all launched older than 30 days, will be deleted by the tool. We recommend decreasing this value to the acceptabvle minimum.

For the remaining launches we recommend deleting test results artefacts from the database and the artefacts storage.

Cleaning the test results artefacts

The process of the transferring of test results artefacs can take considerable amount of time, and would inevitably lead to longer downtime when your team won't be able using both instances – the one you have deployed locally (source) and the one you want to move to (destination). So, the cleanup on the source system is required before the migration process.

Please read the following article covering the data types stored in Allure TestOps.
Please read the article covering the creation of the cleanup rules.

Enable aggressive cleanup rules for artefacts

Rules must include the following (creating global rules for all projects would be sufficient):

Passed tests: scenarios, fixtures, attachments. Preferably keep last 48 hours.
Failed tests: scenarios, fixtures, attachments. Preferably keep last 48 hours.
Unknown tests: scenarios, fixtures, attachments. Preferably keep last 24 hours.
Broken tests: scenarios, fixtures, attachments. Preferably keep last 48 hours.

Access the swagger page (ALLURE-URL/swagger-ui.html) and locate the section "cleanup controller", and execute following three methods in the displayed order

If you have a huge amount of the artefacts the commands 1 and 2 could result in an error in the UI, but on the backend the process will be completed.
If never used before, the Cleanup process if never used before, can trigger huge workload for S3 and can cripple the performance, so we highly recommend running this during the less loaded hours.
The deletion of the artefacts will take considerable time, especially if you haven't had any cleanup rules created. You can check the progress by executing the SQL command against the report service database, see the command here. As soon as the queue is zero, you can proceed to the next steps.

Preparing the backups for the migration

Create the database dump migration

At this moment, your Allure TestOps instance must be already upgraded to at least 4.26.5 version or preferably to version 5 to decrease the downtime.

Data dumps creation

Stop all activities on Allure TestOps instance (stop the instance).

Database dump

Create dump of the database used by Allure TestOps as follows:

IMPORTANT: you need to use pg_dump at least version 16.

pg_dump --file=NAME_OF_DB.dump --host=DB_HOST \
--username=DB_USERNAME --dbname=DB_NAME \
--compress=9 --format=c --schema=public --verbose --blobs \
--no-owner --no-privileges --no-comments -W

Artefact storage dump

Artefact storage contains attachments to test cases and test results. Without this data, all manual test cases will lose the attachments and team would need to add these again.

Create an archive with full backup of the artefacts storage – full archive of the bucket's used to store the artefacts of the test results. If the total size of the bucket is more than 4 Gigabytes, it's advised to create multi-volume archive to avoid re-upload and re-download of whole big archive in case of interruptions.
Consult with the support on how to locate the artefacts storage if you have any concerns.

Provide dumps via created support request

Upload archives with the collected data to a cloud storage of your choice.
Provide links to the uploaded files to the current request alongside with the passwords if any to the created support request.

Timelines

The data restoration usually takes up to 24 hours from the moment SaaS team fully downloaded data, so you need to plan full downtime for the usage of Allure TestOps.

In case of many big amount of artefacts the upload and download process could take more than 24 hours depending on the cloud storage.

As soon as the databases are restored, files are copied, you will be informed by the support ticket in the created request.