Testing-Release.md

 This serves as a template for every KYPO release testing. Create an issue named **Testing Release `YY.MM`**. Copy below content and fill in needed information like release date, links to issues, etc. Feel free to contribute and extend these tests with other test cases.
-## Testing Angular frontend
-- [ ] preferred pagination is stored for the session
-- [ ] thoughtful testing of [linear training](https://gitlab.ics.muni.cz/groups/muni-kypo-crp/-/wikis/Testing-KYPO-instance/Linear-Training)
-- [ ] thoughtful testing of [adaptive training](https://gitlab.ics.muni.cz/groups/muni-kypo-crp/-/wikis/Testing-KYPO-instance/Adaptive-Training)
-- [ ] thoughtful testing of [sandbox interactions](https://gitlab.ics.muni.cz/groups/muni-kypo-crp/-/wikis/Testing-KYPO-instance/Sandbox-Interactions)
-
-##### Training Agenda - Definition
-- [ ] Overview of TD has table actions expanded
-- [ ] Detail pages for adaptive/linear training definition (contain all level/phase types)
-- [ ] Save strategy unification in adaptive/linear training definition edit (Level/Phase edit)
-- [ ] Variant Answers
-- [ ] Reference Solution
-- [ ] Stepper bar working in a preview of linear and adaptive TD
-- [ ] Show stepper bar option for a preview of linear and adaptive TD
-- [ ] Question stepper present in assessment level/adaptive questionnaire phase/general 
-questionnaire phase
-- [ ] Placeholder Variables present in Access level/phase
-- [ ] Preview of Reference graph is present and is updated on save - linear TD only
-- [ ] predefined phases/levels can be created when creating new TD
-- [ ] Mitre techniques can be defined in TD
-- [ ] Mitre techniques table can be accessed from TD overiew
-- [ ] Adaptive model simulator for a single trainee's performance is present and working for the training phases
-- [ ] Adaptive model simulator for trainees' performance is present and working (accessible from adaptive definition overview page). Instructors can upload previously exported training instance, adjust decision matrix and generate the trainees' performance 
-
-##### Training Agenda - Instance
-- [ ] Detail page for adaptive/linear instance (detail for training run in instance detail)
-- [ ] Save strategy unification in adaptive/linear training instance edit (Pool edit)
-- [ ] Selecting from more than 10 pools - pagination size should be set to `MAX_SAFE_INTEGER`
-- [ ] Training instance can not be saved if it does not have Local Environment enabled or it has not selected Pool and Local Environment option is disabled
-- [ ] Create a training instance with assigned pool and save it. Edit it again and save - check whether the pool is still assigned. [Detailed description](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/300).
-- [ ] Check whether adaptive/linear training instance overview table displays a correct number of free sandboxes (pagination size is **NOT** set to 10(default) or user-specific one).
-- [ ] Linear/Adaptive - check whether filtering options work - filter by name.
-- [ ] Linear/Adaptive - check whether sorting in instance overview table works.
-- [ ] Instance has Local environment option which allows to select **Sandbox definition**
-- [ ] Instance with non Local environment option selected allows to select **Pool**
-- [ ] Backward mode option is available
-- [ ] Access token prefix is showing only prefix
-- [ ] Selecting from more than 10 TDs - pagination size should be set to `MAX_SAFE_INTEGER`
-- [ ] Create APG TD and validate that validation of this definition is working while saving TI created from this TD. Also, validate that if the TD has incorrectly set `Variant Answers` in one of its levels, the validation will trigger an error. ([example TD and SD](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/network-attacks-apg))
-- [ ] Check whether the table for Linear/Adaptive training runs of training instance has an information logging of commands and events work. Check whether the provided information is correct.
-- [ ] automatic and systematic check of whether logging of events and commands works (see [issue](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/353))
-- [ ] infinite loading of pool size in instance overview table for records with deleted pools is no longer present
-- [ ] access token trimming is present in instance create
-- [ ] command timeline for ongoing trainings is present in training progress for Instructor
-- [ ] an option to export score of all trainees from a training instance is present
-- [ ] Training instance with non-local environment can not be saved unless a pool is selected
-- [ ] Training instance with local environment can be saved without selected sandbox definition
-- [ ] Deleting training run with assigned sandbox deletes the training run and assigned sandbox 
-
-##### Training Agenda - Training Instance - Cheating Detection
-
-- [ ] test navigators and URL mappings of cheating detections overview, cheating detection create, detection events overview and detection event detail
-- [ ] cheating detection creation correctly assigns all specified attributes to DTO
-- [ ] cheating detection deletion from table action correctly refreshes the table data
-- [ ] cheating detection rerun from table action correctly refreshes the table data
-- [ ] detection events are correctly assigned to a cheating detection they belong to by it's id in the detection event table 
-- [ ] detection events are correctly distinguished based on their type in detection event detail
-- [ ] detection events and cheating detections can in no way be accessed by Trainees
-
-##### Training Agenda - Run
-- [ ] Spice console preload is done when run is started and consoles' URLs are stored in Local Storage
-- [ ] connecting through Guacamole console
-- [ ] connecting through SSH config
-- [ ] adaptive/linear training run (hints), solution, answer field, get ssh config works as expected
-- [ ] topology legend, open console replaced generate console URL
-- [ ] topology is present for the local sandbox, if no sandbox definition is provided placeholder is displayed
-- [ ] initial expansion of a small topology (up to 12 nodes)
-- [ ] backward mode works for all types of levels/phases (when allowed in the instance)
-- [ ] backward mode forbidden for all types of levels/phases (when disabled in the instance)
-- [ ] next button is displayed in assessments, questionnaires, training levels/phases when an error occurred during Get next request (is correct answer request has been successful)
-- [ ] check whether trainee is able to finish the training with local sandbox
-- [ ] linear run can be accessed and fully finished after the update to sandbox uuid
-- [ ] adaptive run can be accessed and fully finished after the update to sandbox uuid
-
-##### Sandbox Agenda - Definition
-- [ ] title of sandbox definition redirects to GitLab repository for sandbox definition
-- [ ] display topology option is present and loads topology (not the placeholder) - right click provides Copy host info to clipboard
-
-##### Sandbox Agenda - Pool
-- [ ] clicking on pool title redirects to pool detail
-- [ ] clicking on sandbox title redirects to sandbox detail with Allocation Request Stages, stage details are present for all stages
-- [ ] display topology action preloads VM consoles for spice and has refactored topology
-- [ ] delete failed, and delete all functionality
-- [ ] allocation by default opens dialog with allocate all option. Users can adjust this value via input, slider or `+` and `-` icons. 
-- [ ] polling of stages status
-- [ ] get ssh config
-- [ ] lock/unlock sandbox
-- [ ] filtering in the resource page
-- [ ] check whether in the pool create there is a correct number of sandbox definitions displayed when creating one (pagination size is not set to 10 (default)).
-- [ ] retry of second and third stage
-- [ ] redirect to stage detail from stage pipeline in overview table
-- [ ] topology is working after the update to sandbox uuid
-- [ ] one expandable button with options to delete all, unlocked and failed is working as expected according to button description
-- [ ] topology now loads spice consoles on backend, meaning that frontend only asks backend if the spice consoles are ready and does not wait until the loading is done, more [here](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/347)
-
-##### Resources
-- [ ] Filtering of only KYPO images is available and functional
-- [ ] refactored resource page provides all new information and filtering options according to [this issue](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/357)
-
-#### User and Group Agenda - User
-- [ ] detail page for user, expandable role info
-- [ ] import users option is displayed above the overview table and imports users from a file
-- [ ] sorting by id is working for users table
-
-#### User and Group Agenda - Group
-- [ ] detail page for group, expandable role info, filtering of users, filtering of role types
-- [ ] change filtering of users by name to fullname
-
-#### Visualizations
-- [ ] instructor view - proper functionality of hurdling - progress visualization in ongoing training
-- [ ] trainee view - proper functionality of adaptive transition visualization after adaptive training (Check Show results of adaptive run as well from training run table).
-- [ ] trainee view - proper functionality of redesigned overview visualization after linear training - without reference solution
-- [ ] trainee view - proper functionality of redesigned overview visualization after linear training and command analysis components - with reference solution
-- [ ] instructor view - proper functionality of redesigned overview visualization after linear training in dashboard - without reference solution
-- [ ] instructor view - proper functionality of redesigned overview visualization after linear training in the dashboard and command analysis components - with reference solution
-- [ ] across training instance visualization is present from TI overview table and by default displays only instance from which the page redirect happened
-- [ ] assessment visualization properly displays all user answers and highlights user across questions on select
-- [ ] progress part of the hurdling visualization displays the zoom bar at the bottom and is working as expected after the refactor
-- [ ] walkthrough visualization is present in post training analysis
+
+# TOC
+
+[[_TOC_]]
+
+# Testing Angular frontend
+
+* [ ] preferred pagination is stored for the session
+* [ ] thoughtful testing of [linear training](https://gitlab.ics.muni.cz/groups/muni-kypo-crp/-/wikis/Testing-KYPO-instance/Linear-Training)
+* [ ] thoughtful testing of [adaptive training](https://gitlab.ics.muni.cz/groups/muni-kypo-crp/-/wikis/Testing-KYPO-instance/Adaptive-Training)
+* [ ] thoughtful testing of [sandbox interactions](https://gitlab.ics.muni.cz/groups/muni-kypo-crp/-/wikis/Testing-KYPO-instance/Sandbox-Interactions)
+
+## Training Agenda
+
+### Definition
+
+* [ ] Overview of TD has table actions expanded
+* [ ] Detail pages for adaptive/linear training definition (contain all level/phase types)
+* [ ] Save strategy unification in adaptive/linear training definition edit (Level/Phase edit)
+* [ ] Variant Answers
+* [ ] Reference Solution
+* [ ] Stepper bar working in a preview of linear and adaptive TD
+* [ ] Show stepper bar option for a preview of linear and adaptive TD
+* [ ] Question stepper present in assessment level/adaptive questionnaire phase/general questionnaire phase
+* [ ] Placeholder Variables present in Access level/phase
+* [ ] Preview of Reference graph is present and is updated on save - linear TD only
+* [ ] predefined phases/levels can be created when creating new TD
+* [ ] Mitre techniques can be defined in TD
+* [ ] Mitre techniques table can be accessed from TD overiew
+* [ ] Adaptive model simulator for a single trainee's performance is present and working for the training phases
+* [ ] Adaptive model simulator for trainees' performance is present and working (accessible from adaptive definition overview page). Instructors can upload previously exported training instance, adjust decision matrix and generate the trainees' performance
+
+### Instance
+
+* [ ] Detail page for adaptive/linear instance (detail for training run in instance detail)
+* [ ] Save strategy unification in adaptive/linear training instance edit (Pool edit)
+* [ ] Selecting from more than 10 pools - pagination size should be set to `MAX_SAFE_INTEGER`
+* [ ] Training instance can not be saved if it does not have Local Environment enabled or it has not selected Pool and Local Environment option is disabled
+* [ ] Create a training instance with assigned pool and save it. Edit it again and save - check whether the pool is still assigned. [Detailed description](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/300 "Pool is unassinged after training instance is saved").
+* [ ] Check whether adaptive/linear training instance overview table displays a correct number of free sandboxes (pagination size is **NOT** set to 10(default) or user-specific one).
+* [ ] Linear/Adaptive - check whether filtering options work - filter by name.
+* [ ] Linear/Adaptive - check whether sorting in instance overview table works.
+* [ ] Instance has Local environment option which allows to select **Sandbox definition**
+* [ ] Instance with non Local environment option selected allows to select **Pool**
+* [ ] Backward mode option is available
+* [ ] Access token prefix is showing only prefix
+* [ ] Selecting from more than 10 TDs - pagination size should be set to `MAX_SAFE_INTEGER`
+* [ ] Create APG TD and validate that validation of this definition is working while saving TI created from this TD. Also, validate that if the TD has incorrectly set `Variant Answers` in one of its levels, the validation will trigger an error. ([example TD and SD](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/network-attacks-apg))
+* [ ] Check whether the table for Linear/Adaptive training runs of training instance has an information logging of commands and events work. Check whether the provided information is correct.
+* [ ] automatic and systematic check of whether logging of events and commands works (see [issue](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/353 "Add an automatic and systematic check of whether logging works"))
+* [ ] infinite loading of pool size in instance overview table for records with deleted pools is no longer present
+* [ ] access token trimming is present in instance create
+* [ ] command timeline for ongoing trainings is present in training progress for Instructor
+* [ ] an option to export score of all trainees from a training instance is present
+* [ ] Training instance with non-local environment can not be saved unless a pool is selected
+* [ ] Training instance with local environment can be saved without selected sandbox definition
+* [ ] Deleting training run with assigned sandbox deletes the training run and assigned sandbox
+
+## Cheating Detection
+
+* [ ] test navigators and URL mappings of cheating detections overview, cheating detection create, detection events overview and detection event detail
+* [ ] cheating detection creation correctly assigns all specified attributes to DTO
+* [ ] cheating detection deletion from table action correctly refreshes the table data
+* [ ] cheating detection rerun from table action correctly refreshes the table data
+* [ ] detection events are correctly assigned to a cheating detection they belong to by it's id in the detection event table
+* [ ] detection events are correctly distinguished based on their type in detection event detail
+* [ ] detection events and cheating detections can in no way be accessed by Trainees
+
+### Run
+
+* [ ] Spice console preload is done when run is started and consoles' URLs are stored in Local Storage
+* [ ] connecting through Guacamole console
+* [ ] connecting through SSH config
+* [ ] adaptive/linear training run (hints), solution, answer field, get ssh config works as expected
+* [ ] topology legend, open console replaced generate console URL
+* [ ] topology is present for the local sandbox, if no sandbox definition is provided placeholder is displayed
+* [ ] initial expansion of a small topology (up to 12 nodes)
+* [ ] backward mode works for all types of levels/phases (when allowed in the instance)
+* [ ] backward mode forbidden for all types of levels/phases (when disabled in the instance)
+* [ ] next button is displayed in assessments, questionnaires, training levels/phases when an error occurred during Get next request (is correct answer request has been successful)
+* [ ] check whether trainee is able to finish the training with local sandbox
+* [ ] linear run can be accessed and fully finished after the update to sandbox uuid
+* [ ] adaptive run can be accessed and fully finished after the update to sandbox uuid
+
+## Sandbox Agenda
+
+### Definition
+
+* [ ] title of sandbox definition redirects to GitLab repository for sandbox definition
+* [ ] display topology option is present and loads topology (not the placeholder) - right click provides Copy host info to clipboard
+
+### Pool
+
+* [ ] clicking on pool title redirects to pool detail
+* [ ] clicking on sandbox title redirects to sandbox detail with Allocation Request Stages, stage details are present for all stages
+* [ ] display topology action preloads VM consoles for spice and has refactored topology
+* [ ] delete failed, and delete all functionality
+* [ ] allocation by default opens dialog with allocate all option. Users can adjust this value via input, slider or `+` and `-` icons.
+* [ ] polling of stages status
+* [ ] get ssh config
+* [ ] lock/unlock sandbox
+* [ ] filtering in the resource page
+* [ ] check whether in the pool create there is a correct number of sandbox definitions displayed when creating one (pagination size is not set to 10 (default)).
+* [ ] retry of second and third stage
+* [ ] redirect to stage detail from stage pipeline in overview table
+* [ ] topology is working after the update to sandbox uuid
+* [ ] one expandable button with options to delete all, unlocked and failed is working as expected according to button description
+* [ ] topology now loads spice consoles on backend, meaning that frontend only asks backend if the spice consoles are ready and does not wait until the loading is done, more [here](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/347 "Optimize sandbox agenda")
+
+### Resources
+
+* [ ] Filtering of only KYPO images is available and functional
+* [ ] refactored resource page provides all new information and filtering options according to [this issue](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/357 "Refactor Resources page")
+
+## Administration (User and Group Agenda)
+  
+### User
+
+* [ ] detail page for user, expandable role info
+* [ ] import users option is displayed above the overview table and imports users from a file
+* [ ] sorting by id is working for users table
+
+### Group
+
+* [ ] detail page for group, expandable role info, filtering of users, filtering of role types
+* [ ] change filtering of users by name to fullname
+
+## Visualizations
+
+* [ ] instructor view - proper functionality of hurdling - progress visualization in ongoing training
+* [ ] trainee view - proper functionality of adaptive transition visualization after adaptive training (Check Show results of adaptive run as well from training run table).
+* [ ] trainee view - proper functionality of redesigned overview visualization after linear training - without reference solution
+* [ ] trainee view - proper functionality of redesigned overview visualization after linear training and command analysis components - with reference solution
+* [ ] instructor view - proper functionality of redesigned overview visualization after linear training in dashboard - without reference solution
+* [ ] instructor view - proper functionality of redesigned overview visualization after linear training in the dashboard and command analysis components - with reference solution
+* [ ] across training instance visualization is present from TI overview table and by default displays only instance from which the page redirect happened
+* [ ] assessment visualization properly displays all user answers and highlights user across questions on select
+* [ ] progress part of the hurdling visualization displays the zoom bar at the bottom and is working as expected after the refactor
+* [ ] walkthrough visualization is present in post training analysis
+
+# Backend tests
 
 ## Testing Java backend
-- [ ] boolean checkbox whether commands are required in particular level 
-- [ ] check that export score from training instance returns a csv file that includes correct information (all users, correct score)
-- [ ] check that stack trace is not included in errors
-- [ ] check that it is not possible to create a cloud instance (Linear/Adaptive) without assigning a pool
-- [ ] check that a trainee cannot retrieve a training definition (Linear/Adaptive) by id from `/training-definitions/{definitionId}` endpoint
-- [ ] check that only designers of the specific training definition can clone/export/modify it using the rest endpoints
-- [ ] check that only organizers of the specific training instance can modify it using the rest endpoint
-- [ ] check that post-training table contains the correct score, specifically users that start a level don't receive the score without finishing it
-- [ ] check that instance access codes are trimmed during creation
-- [ ] check that training runs returned from `/training-instances/{instanceId}/training-runs` contain correct logging information (Linear/Adaptive)
-
-##### Training - Cheating Detection
-- [ ] `createAndExecuteCheatingDetection` rest waypoint correctly creates and executes a cheating detection on training instance
-- [ ] `rerunCheatingDetection` rest waypoint correctly executes the existing cheating detection from scratch
-- [ ] `deleteDetectionEvents` rest waypoint correctly deletes a cheating detection and all its associated detection events
-- [ ] `findAllDetectionEventsOfCheatingDetection` rest waypoint finds all and only detection events of specified cheating detection by id 
-- [ ] `findAllCheatingDetectionsOfInstance` rest waypoint finds all and only cheating detections of specified training instance by id 
-- [ ] test execution of detection method of type `answerSimilarity`, correctness of generated detection events (no incorrectly detected events, no missing occured events)  
-- [ ] test execution of detection method of type `locationSimilarity`, correctness of generated detection events (no incorrectly detected events, no missing occured events) 
-- [ ] test execution of detection method of type `timeProximity`, correctness of generated detection events (no incorrectly detected events, no missing occured events) 
-- [ ] test execution of detection method of type `minimalSolveTime`, correctness of generated detection events (no incorrectly detected events, no missing occured events) 
-- [ ] test execution of detection method of type `noCommands`, correctness of generated detection events (no incorrectly detected events, no missing occured events) 
-- [ ] test that training runs in which detection events were found are correctly assigned true value in field hasDetectionEvents. Also test that when the detection events are deleted, the value is set to false
+
+* [ ] boolean checkbox whether commands are required in particular level
+* [ ] check that export score from training instance returns a csv file that includes correct information (all users, correct score)
+* [ ] check that stack trace is not included in errors
+* [ ] check that it is not possible to create a cloud instance (Linear/Adaptive) without assigning a pool
+* [ ] check that a trainee cannot retrieve a training definition (Linear/Adaptive) by id from `/training-definitions/{definitionId}` endpoint
+* [ ] check that only designers of the specific training definition can clone/export/modify it using the rest endpoints
+* [ ] check that only organizers of the specific training instance can modify it using the rest endpoint
+* [ ] check that post-training table contains the correct score, specifically users that start a level don't receive the score without finishing it
+* [ ] check that instance access codes are trimmed during creation
+* [ ] check that training runs returned from `/training-instances/{instanceId}/training-runs` contain correct logging information (Linear/Adaptive)
+
+## Training - Cheating Detection
+
+* [ ] `createAndExecuteCheatingDetection` rest waypoint correctly creates and executes a cheating detection on training instance
+* [ ] `rerunCheatingDetection` rest waypoint correctly executes the existing cheating detection from scratch
+* [ ] `deleteDetectionEvents` rest waypoint correctly deletes a cheating detection and all its associated detection events
+* [ ] `findAllDetectionEventsOfCheatingDetection` rest waypoint finds all and only detection events of specified cheating detection by id
+* [ ] `findAllCheatingDetectionsOfInstance` rest waypoint finds all and only cheating detections of specified training instance by id
+* [ ] test execution of detection method of type `answerSimilarity`, correctness of generated detection events (no incorrectly detected events, no missing occured events)
+* [ ] test execution of detection method of type `locationSimilarity`, correctness of generated detection events (no incorrectly detected events, no missing occured events)
+* [ ] test execution of detection method of type `timeProximity`, correctness of generated detection events (no incorrectly detected events, no missing occured events)
+* [ ] test execution of detection method of type `minimalSolveTime`, correctness of generated detection events (no incorrectly detected events, no missing occured events)
+* [ ] test execution of detection method of type `noCommands`, correctness of generated detection events (no incorrectly detected events, no missing occured events)
+* [ ] test that training runs in which detection events were found are correctly assigned true value in field hasDetectionEvents. Also test that when the detection events are deleted, the value is set to false
 
 ## Testing Python backend
-*When performing these tests, read ahead to see if maybe you can cover multiple test cases with the same sandbox to minimize allocations.*
-- [ ] check if it is possible to create a sandbox (if the cloud project has enough resources to allocate a number of sandboxes, try creating a pool which has too many sandboxes and try to allocate them. It should not be possible.)
-- [ ] test all states of the sandbox allocation unit (whether the sandbox is in the allocation or cleanup stage, fail or success)
-  - [ ] test a simultaneous allocation of 4 sandboxes (using a simple sandbox [definition](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/general-testing-definition)) at the same time (using a pool of 4 sandboxes and pressing the allocate all button). Test this about 10 times. There may be errors in the first stage due to the locking of Terraform workspace in the apply stage. This is expected but should not happen too often (If a failure of this type occurs, multiple sandboxes will often fail). Too often is more than twice out of the 10 allocations. ,A failure in networking stage may also occur, which should be rare and should be explored.
-    - [ ] if some sandboxes fail, try to re-run their allocation (do not delete them) and see if the re-run helped.
-    - [ ] when deleting sandboxes after these tests, try using different approaches of deletion. Try deleting one by one, the delete all, delete all failed and delete unlocked (lock sandboxes before)
-    - [ ] test that failed user ansible stage can be re-run, using a definition in which you create a new branch and deliberately add invalid information into the user ansible. A re-run will also fail, but test that is it possible.
-  - [ ] test allocating multiple sandboxes in a big pool (use the above definition and allocate a pool of 12 sandboxes) Repeat this three times.
-  - [ ] test that at any of the allocation stages, a cleanup can be performed, which should interrupt the allocation and start deallocating the sandbox (include a test case where you run the cleanup immediately after a sandbox has started allocating). Perform this test with pool of 4 sandboxes.
-- [ ] test APG (for example with this [definition](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/network-attacks-apg))
-- [ ] test that the cleanup stage is not depending on the correct user git repo
-- [ ] try to access a sandbox with ssh, spice, and guacamole
-  - [ ] test a big [sandbox definition](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/five-os-sandbox), which includes different hosts of multiple different OSs, including windows and trz connecting to each one, also see if the ssh config is generated properly.
-- [ ] check the topology visualization is generated properly
-- [ ] `nmap` shouldn't discover hosts in private address ranges outside the topology definition. [Detailed description](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/206). Hosts with IPs of X.X.X.1 (router) and X.X.X.2 (OpenStack DHCP) that share the first three digits of IP with the network defined in the definition and part of the sandbox and are supposed to be found, this test is in regard to scanning machines completely outside of the defined range of each network/networks connected to it.
-- [ ] test the docker containers in [this definition](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/kypo-docker-containers), try to SSH into these docker containers
-- [ ] visit the resources agenda and try that the images can be correctly sorted using available methods, also test the only munikypo images filter
-- [ ] test the disk volumes [functionality in topology](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/360) using [this branch](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/five-os-sandbox/-/blob/testing-volumes/topology.yml#L9).
-
-##### Testing [kypo-mitre-matrix-service](https://gitlab.ics.muni.cz/muni-kypo-crp/backend-python/kypo-mitre-technique-service)
-- [ ] try to run to service with all the other services and ensure that the map gets generated and can be viewed on the frontend. It can be viewed at two places, in training run agenda and in training definitions overview. The table will only display trainings that are **released**. In the mitre repository, go to the test_definition and download the json for the definition. Upload this defintion and release it. Check both mitre tables and see if the marks for the definition are present.
-- [ ] create a training instance with this testing definition and play it, then check the mitre table in the training run agenda and see if it is in the "played" section of it.
-- [ ] check the last update on this [website](https://attack.mitre.org/resources/updates/). If it is later than **October 2022**, the MITRE ATTACK matrix may have been changed singe the last check
-  - [ ] check the content of the update and verify that the service still works under this update (if there are new tactics/techniques, the instructor should be able to add them to new levels of linear training and phases of adaptive training). If some tactics/techniques were removed, instructors should no longer be able to add them, but if there are training definitions which still contain such techniques, the service should not fail because of them, instead they should be ignored.
-  - [ ] run the service locally on its own (clone its repository and run `pipenv run python manage.py runserver 8001`)
-  - [ ] open the [swagger documentation](http://127.0.0.1:8001/mitre-technique/api/v1/doc) and call the `/mitre-technique-index` endpoint (it is not authenticated so it should work fine). Ensure that a list of techniques and their codes are returned and also check that the console where the service runs does **NOT** output `"The method getting tactics and techniques failed with:..."`. 
-    - [ ] if the message is in the output, check the exception that is thrown. There is likely a problem with getting the MITRE ATTACK data from the taxii2 server, thus localy stored static data is used. This may be fine if the external functionality of the endpoints that are called by this service on the MITRE side has been changed severely and if this service is not worth updating.
-    - [ ] if that is not the case and the endpoint works fine, this service will update the `mitre_attack_backup_data` (by default stored in `kypo/mitre_matrix_visualizer_app/templates/`). Update the existing `mitre_attack_backup_data` in the gitlab repository of this service with the new one.
-  - [ ] update the time of the last update of the MITRE ATTACK matrix data in this manual
-
-## Integration testing
-- [ ] import all publicly available sandbox definitions stored at https://gitlab.ics.muni.cz/muni-kypo-trainings/games + display their topology in sandbox definition overview table
-- [ ] nice to have: import all private sandbox definitions stored at https://gitlab.ics.muni.cz/muni-kypo-trainings/games + display their topology in sandbox definition overview table
-- [ ] create pools of all **public** imported sandbox definitions and allocate a sandbox
-- [ ] in pool detail check whether topology is displayed correctly for **public** sandbox definitions
-- [ ] create pools of all **private** imported sandbox definitions and allocate a sandbox
-- [ ] in pool detail check whether topology is displayed correctly for **private** sandbox definitions
-- [ ] verify that all **public** games using the `kali` image (such as [junior-hacker](https://gitlab.ics.muni.cz/muni-kypo-trainings/games/junior-hacker), [junior-hacker-adaptive](https://gitlab.ics.muni.cz/muni-kypo-trainings/games/junior-hacker-adaptive))  can be accessed via Guacamole console
-- [ ] web console does not contain any errors which are of unknown origin for developers.
 
+_When performing these tests, read ahead to see if maybe you can cover multiple test cases with the same sandbox to minimize allocations._
+
+* [ ] check if it is possible to create a sandbox (if the cloud project has enough resources to allocate a number of sandboxes, try creating a pool which has too many sandboxes and try to allocate them. It should not be possible.)
+* [ ] test all states of the sandbox allocation unit (whether the sandbox is in the allocation or cleanup stage, fail or success)
+  * [ ] test a simultaneous allocation of 4 sandboxes (using a simple sandbox [definition](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/general-testing-definition)) at the same time (using a pool of 4 sandboxes and pressing the allocate all button). Test this about 10 times. There may be errors in the first stage due to the locking of Terraform workspace in the apply stage. This is expected but should not happen too often (If a failure of this type occurs, multiple sandboxes will often fail). Too often is more than twice out of the 10 allocations. ,A failure in networking stage may also occur, which should be rare and should be explored.
+    * [ ] if some sandboxes fail, try to re-run their allocation (do not delete them) and see if the re-run helped.
+    * [ ] when deleting sandboxes after these tests, try using different approaches of deletion. Try deleting one by one, the delete all, delete all failed and delete unlocked (lock sandboxes before)
+    * [ ] test that failed user ansible stage can be re-run, using a definition in which you create a new branch and deliberately add invalid information into the user ansible. A re-run will also fail, but test that is it possible.
+  * [ ] test allocating multiple sandboxes in a big pool (use the above definition and allocate a pool of 12 sandboxes) Repeat this three times.
+  * [ ] test that at any of the allocation stages, a cleanup can be performed, which should interrupt the allocation and start deallocating the sandbox (include a test case where you run the cleanup immediately after a sandbox has started allocating). Perform this test with pool of 4 sandboxes.
+* [ ] test APG (for example with this [definition](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/network-attacks-apg))
+* [ ] test that the cleanup stage is not depending on the correct user git repo
+* [ ] try to access a sandbox with ssh, spice, and guacamole
+  * [ ] test a big [sandbox definition](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/five-os-sandbox), which includes different hosts of multiple different OSs, including windows and trz connecting to each one, also see if the ssh config is generated properly.
+* [ ] check the topology visualization is generated properly
+* [ ] `nmap` shouldn't discover hosts in private address ranges outside the topology definition. [Detailed description](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/206 "Test case for (in)secure sandbox networking"). Hosts with IPs of X.X.X.1 (router) and X.X.X.2 (OpenStack DHCP) that share the first three digits of IP with the network defined in the definition and part of the sandbox and are supposed to be found, this test is in regard to scanning machines completely outside of the defined range of each network/networks connected to it.
+* [ ] test the docker containers in [this definition](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/kypo-docker-containers), try to SSH into these docker containers
+* [ ] visit the resources agenda and try that the images can be correctly sorted using available methods, also test the only munikypo images filter
+* [ ] test the disk volumes [functionality in topology](https://gitlab.ics.muni.cz/muni-kypo-crp/kypo-crp-issues/-/issues/360 "New functionality: definition of disk size and number of discs for hosts in pool") using [this branch](https://gitlab.ics.muni.cz/muni-kypo-crp/prototypes-and-examples/sandbox-definitions/five-os-sandbox/-/blob/testing-volumes/topology.yml#L9).
+
+## Testing [kypo-mitre-matrix-service](https://gitlab.ics.muni.cz/muni-kypo-crp/backend-python/kypo-mitre-technique-service)
+
+* [ ] try to run to service with all the other services and ensure that the map gets generated and can be viewed on the frontend. It can be viewed at two places, in training run agenda and in training definitions overview. The table will only display trainings that are **released**. In the mitre repository, go to the test_definition and download the json for the definition. Upload this defintion and release it. Check both mitre tables and see if the marks for the definition are present.
+* [ ] create a training instance with this testing definition and play it, then check the mitre table in the training run agenda and see if it is in the "played" section of it.
+* [ ] check the last update on this [website](https://attack.mitre.org/resources/updates/). If it is later than **October 2022**, the MITRE ATTACK matrix may have been changed singe the last check
+  * [ ] check the content of the update and verify that the service still works under this update (if there are new tactics/techniques, the instructor should be able to add them to new levels of linear training and phases of adaptive training). If some tactics/techniques were removed, instructors should no longer be able to add them, but if there are training definitions which still contain such techniques, the service should not fail because of them, instead they should be ignored.
+  * [ ] run the service locally on its own (clone its repository and run `pipenv run python manage.py runserver 8001`)
+  * [ ] open the [swagger documentation](http://127.0.0.1:8001/mitre-technique/api/v1/doc) and call the `/mitre-technique-index` endpoint (it is not authenticated so it should work fine). Ensure that a list of techniques and their codes are returned and also check that the console where the service runs does **NOT** output `"The method getting tactics and techniques failed with:..."`.
+    * [ ] if the message is in the output, check the exception that is thrown. There is likely a problem with getting the MITRE ATTACK data from the taxii2 server, thus localy stored static data is used. This may be fine if the external functionality of the endpoints that are called by this service on the MITRE side has been changed severely and if this service is not worth updating.
+    * [ ] if that is not the case and the endpoint works fine, this service will update the `mitre_attack_backup_data` (by default stored in `kypo/mitre_matrix_visualizer_app/templates/`). Update the existing `mitre_attack_backup_data` in the gitlab repository of this service with the new one.
+  * [ ] update the time of the last update of the MITRE ATTACK matrix data in this manual
+
+# Integration testing
+
+* [ ] import all publicly available sandbox definitions stored at https://gitlab.ics.muni.cz/muni-kypo-trainings/games + display their topology in sandbox definition overview table
+* [ ] nice to have: import all private sandbox definitions stored at https://gitlab.ics.muni.cz/muni-kypo-trainings/games + display their topology in sandbox definition overview table
+* [ ] create pools of all **public** imported sandbox definitions and allocate a sandbox
+* [ ] in pool detail check whether topology is displayed correctly for **public** sandbox definitions
+* [ ] create pools of all **private** imported sandbox definitions and allocate a sandbox
+* [ ] in pool detail check whether topology is displayed correctly for **private** sandbox definitions
+* [ ] verify that all **public** games using the `kali` image (such as [junior-hacker](https://gitlab.ics.muni.cz/muni-kypo-trainings/games/junior-hacker), [junior-hacker-adaptive](https://gitlab.ics.muni.cz/muni-kypo-trainings/games/junior-hacker-adaptive)) can be accessed via Guacamole console
+* [ ] web console does not contain any errors which are of unknown origin for developers.
 
 **Bugs/issues found during testing of frontend should be reported in this issue** (link appropriate issue here)
 
 **Bugs/issues regarding sandbox agenda (python backend) should be reported in this issue** (link appropriate issue here)
 
 ##### Deployed instance URL:
+
 https://`<instance URL>`
\ No newline at end of file