Merge branch 'freznicek-doc-update' into 'master'

Documentation improvement (use-cases)

See merge request !48

README.md

@@ -10,11 +10,21 @@
    </div>
 
 
-
 This guide aims to provide a walk-through for setting up a rudimentary
 virtual infrastructure in MetaCentrum Cloud. It is a good jumping-off
 point for most users.
 
+MetaCentrum Cloud is the [IaaS cloud](https://en.wikipedia.org/wiki/Infrastructure_as_a_service) on top of [open-source OpenStack project](https://opendev.org/openstack).
+Users may configure and use cloud resources for reaching individual (scientific) goals.
+
+Most important cloud resources are:
+ * virtual machines
+ * virtual networking (VPNs, firewalls, routers)
+ * private and/or public IP addresses
+ * storage
+ * cloud load balancers
+
+
 The left sidebar can be used for navigation throughout the documentation.
 The whole guide can also be downloaded as PDFs for printing or later use.
 

SUMMARY.md

 * [News](/news/README.md)
 * [Introduction](/README.md)
+* [Cloud best practices](best-practices/README.md)
 * [Get Access](/register/README.md)
 * [Quick Start](/quick-start/README.md)
 * [Advanced Features](/gui/README.md)

best-practices/README.md

+# Cloud best-practices
+
+Following article summarizes effective approaches how to use our cloud.
+
+## How many public IP addresses I need?
+There are [two pools of public IPv4 addresses available](/network/#group-project).
+**Unfortunately the amount of available public IPv4 addresses is limited.** Read the details on [how to set-up your project networking](/network/).
+
+In most cases even when you build huge cloud infrastructure you should be able to access it via few (up-to two) public IP addresses.
+
+![](/best-practices/images/accessing-vms-through-jump-host-6-mod.png)
+
+There are following project VMs in the cloud architecture:
+
+| VM name         | VM orerating system | VM IP addresses | VM flavor type | VM description |
+| :---            | :---:               | :-----------:   | :------:       | :------:       |
+| freznicek-cos8  | centos-8-x86_64     | 172.16.0.54, 147.251.21.72 (public) | standard.medium | jump host |
+| freznicek-ubu   | ubuntu-focal-x86_64 | 172.16.1.67     | standard.medium | internal VM   |
+| freznicek-deb10 | debian-10-x86_64    | 172.16.0.158    | standard.medium | internal VM   |
+| ...             | ...                 | ...             | ...             | internal VM   |
+
+
+
+### Setting-up the VPN tunnel via encrypted SSH with [sshuttle](https://github.com/sshuttle/sshuttle)
+
+```sh
+# terminal A
+# Launch tunnel through jump-host VM
+
+# Install sshuttle
+if grep -qE 'ID_LIKE=.*debian' /etc/os-release; then
+  # on debian like OS
+  sudo apt-get update
+  sudo apt-get -y install sshuttle
+elif grep -qE 'ID_LIKE=.*rhel' /etc/os-release; then
+  # on RHEL like systems
+  sudo yum -y install sshuttle
+fi
+
+# Establish the SSH tunnel (and stay connected) where
+# 147.251.21.72 is IP address of example jump-host
+# 172.16.0.0/22 is IP subnet where example cloud resources are internally available
+sshuttle -r centos@147.251.21.72 172.16.0.0/22
+```
+
+### Accessing (hidden) project VMs through the VPN tunnel
+
+```sh
+# terminal B
+# Access all VMs allocated in project in 172.16.0.0/22 subnet (a C5 instance shown on picture)
+$ ssh debian@172.16.0.158 uname -a
+Linux freznicek-deb10 4.19.0-14-cloud-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux
+
+# Access is not limited to any protocol, you may access web servers as well as databases...
+$ curl 172.16.1.67:8080
+Hello, world, cnt=1, hostname=freznicek-ubu
+```
+
+## How to store project data
+
+Every project generates an amount of data which needs to be stored. There are options (sorted by preference):
+ * as objects or files in a [S3 compatible storage](https://en.wikipedia.org/wiki/Amazon_S3)
+   * S3 compatible storage may be requested as separate cloud storage resource ([OpenStack Swift storage + S3 API](https://docs.openstack.org/swift/latest/s3_compat.html))
+   * S3 storage may be also easily launched on one of the project VMs ([minio server](https://github.com/minio/minio))
+ * as files on 
+   * separate (ceph) volume
+   * virtual machine disk volume (i.e. no explicit volume for the project data)
+ * as objects or files in the [OpenShift Swift storage](https://docs.openstack.org/swift/train/admin/objectstorage-intro.html)
+
+MetaCentrum Cloud stores raw data:
+ * in ceph cloud storage on rotation disks (SSDs will be available soon)
+ * in hypervisor (bare metal) disks (rotational, SSD, SSD NVMe)
+
+We encourage all users to backup important data themselves while we work on cloud native backup solution.
+
+## How to compute (scientific) tasks
+
+Your application may be:
+ * `A.` single instance application, running on one of cloud computation resources
+ * `B.` multi-instance application with messaging support (MPI), where all instances run on same cloud computation resource
+ * `C.` true distributed computing, where application runs in jobs scheduled to multiple cloud computation resources
+
+Applications running in single cloud resource (`A.` and `B.`) are direct match for MetaCentrum Cloud OpenStack. Distributed applications (`C.`) are best handled by [MetaCentrum PBS system](https://metavo.metacentrum.cz/cs/state/personal).
+
+## How to create and maintain cloud resources
+
+Your project is computed within MetaCentrum Cloud Openstack project where you can claim MetaCentrum Cloud Openstack resources (for example virtual machine, floating IP, ...). There are multiple ways how to set-up the MetaCentrum Cloud Openstack resources:
+ * manually using [MetaCentrum Cloud Openstack Dashboard UI](https://dashboard.cloud.muni.cz) (Openstack Horizon)
+ * automated approaches
+   * [terraform](https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs) ([example project](https://github.com/terraform-provider-openstack/terraform-provider-openstack/tree/main/examples/app-with-networking))
+   * ansible
+   * [openstack heat](https://docs.openstack.org/heat/train/template_guide/hello_world.html)
+
+If your project infrastructure (MetaCentrum Cloud Openstack resources) within cloud is static you may select manual approach with [MetaCentrum Cloud Openstack Dashboard UI](https://dashboard.cloud.muni.cz). There are projects which need to allocate MetaCentrum Cloud Openstack resources dynamically, in such cases we strongly encourage automation even at this stage.
+
+## How to transfer your work to cloud resources and make it up-to-date
+
+There are several options how to transfer project to cloud resources:
+ * manually with `scp`
+ * automatically with `ansible` ([example](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi/-/blob/de673766b832c48142c6ad1be73f5bce046b02a2/ansible/roles/cloud-project-native/tasks/init.yml#L29-47))
+ * automatically with `terraform`
+ * indirectly in a project container (example: [releasing a project container image](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi/-/blob/de673766b832c48142c6ad1be73f5bce046b02a2/.gitlab-ci.yml#L17-80), [pulling and running a project image](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi/-/blob/de673766b832c48142c6ad1be73f5bce046b02a2/ansible/roles/cloud-project-container/tasks/deploy.yml#L16-28))
+ * indirectly in OpenStack (glance) image (you need to obtain image-uploader role)
+   * OpenStack Glance images may be [public, private, community or shared](/gui/#image-visibility).
+
+### ssh to cloud VM resources and manual update
+
+In this scenario you log to your cloud VM and perform all needed actions manually. This approach does not scale well, is not effective enough as different users may configure cloud VM resources different ways resulting sometimes in different resource behavior.
+
+### automated work transfer and synchronization with docker (or podman)
+
+There are automation tools which may help you to ease your cloud usage:
+ * ansible and/or terraform
+ * container runtime engine (docker, podman, ...)
+
+Ansible is cloud automation tool which helps you with:
+ * keeping your VM updated
+ * automatically migrating your applications or data to/from cloud VM
+
+
+Container runtime engine helps you to put your into a container stored in a container registry.
+Putting your work into container has several advantages:
+ * share the code including binaries in consistent environment (even across different Operating Systems)
+ * avoids application [re]compilation in the cloud
+ * your application running in the container is isolated from the host's container runtime so
+   * you may run multiple instances easily
+   * you may easily compare different versions at once without collisions
+ * you become ready for future kubernetes cloud
+
+As a container registry we suggest either:
+ * public quay.io ([you need to register for free first](https://quay.io/signin/))
+ * private Masaryk University [registry.gitlab.ics.muni.cz:443](registry.gitlab.ics.muni.cz:443)
+
+Example of such approach is demonstrated in [`cloud-estimate-pi` project](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi).
+
+## How to receive data from project's experiments to your workstation
+
+It certainly depends on how your data are stored, the options are:
+ * files transfer
+   * manual file transfer with `scp` (and possibly `sshuttle`)
+   * automated file transfer with `scp` + `ansible` (and possibly `sshuttle`), demonstrated in [`cloud-estimate-pi` project](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi/-/blob/master/ansible/roles/cloud-project-container/tasks/download.yml)
+ * database data/objects transfer
+   * data stored in the S3 compatible database may be easily received via [minio client application MC](https://docs.min.io/docs/minio-client-complete-guide)
+   * date stored in [OpenStack Swift python client `swift`](https://docs.openstack.org/python-swiftclient/train/swiftclient.html)
+
+
+## How to make your application in the cloud highly available
+
+Let's assume your application is running in multiple instances in cloud already.
+To make you application highly available (HA) you need to
+ * run the application instances on different cloud resources
+ * use MetaCentrum Cloud load-balancer component (based on [OpenStack Octavia](https://docs.openstack.org/octavia/train/reference/introduction.html#octavia-terminology)) which is goint to balance traffic to one of the app's instances.
+
+Your application surely need Fully Qualified Domain Name (FQDN) address to become popular. Setting FQDN is done on the public floating IP linked to the load-balancer.
+
+
+## Cloud project example and workflow recommendations
+
+This chapter summarizes effective cloud workflows on the (example) [`cloud-estimate-pi` project](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi).
+
+The project recommendations are:
+ 1. Project files should be versioned in [a VCS](https://en.wikipedia.org/wiki/Version_control) (git)
+ 1. The project repository should
+    * contain the documentation
+    * follow standard directory structure `src/`, `conf/`, `kubernetes/`
+    * include CI/CD process pipeline ([`.gitlab-ci.yml`](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi/-/blob/master/.gitlab-ci.yml), ...)
+    * contain deployment manifests or scripts (kubernetes manifests or declarative deployment files (ansible, terraform, puppet, ...))
+ 1. The project release should be automated and triggered by pushing a [semver v2 compatible](https://semver.org/) [tag](https://dev.to/neshaz/a-tutorial-for-tagging-releases-in-git-147e)
+ 1. The project should support execution in a container as there are significant benefits: ([`Dockerfile`](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi/-/blob/master/Dockerfile))
+    * consistent environment (surrounding the application)
+    * application portability across all Operating Systems
+    * application isolation from host Operating System
+    * multiple ways how to execute the application (container cloud support advanced container life-cycle management)
+ 1. The project should have a changelog (either manually written or generated) (for instance [`CHANGELOG.md`](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi/-/blob/master/CHANGELOG.md))
+
+
+We recommend every project defines cloud usage workflow which may consist of:
+ 1. Cloud resource initialization, performing
+    * cloud resource update to latest state
+    * install necessary tools for project compilation and execution
+    * test container infrastructure (if it is used)
+    * transfer project files if need to be compiled
+ 1. Project deployment (and execution) in the cloud, consisting of
+    * compilation of the project in the cloud (if native execution selected)
+    * execution of the project application[s] in the cloud
+    * storing or collecting project data and logs
+ 1. Download project data from cloud to workstation (for further analysis or troubleshooting)
+    * download of project data from cloud to user's workstation
+ 1. Cloud resource destroy
+  
+## Road-map to effective cloud usage
+
+A project automation is usually done in CI/CD pipelines. Read [Gitlab CI/CD article](https://docs.gitlab.com/ee/ci/introduction/) for more details.
+![](https://docs.gitlab.com/ee/ci/introduction/img/gitlab_workflow_example_extended_v12_3.png)
+
+Following table shows the different cloud usage phases:
+
+| Cloud usage phase | Cloud resource management | Project packaging | Project deployment | Project execution | Project data synchronization | Project troubleshooting |
+| :---              | :---:                     | :-----------:     | :------:           | :------------:    | :------------:               | :------------:          |
+| ineffective manual approach | manual (`ssh`)      | manually built binaries (versioned?) | manual deployment (scp) | manual execution (ssh) | manual transfers (scp) | manual investigation on VM (scp) |
+| ...               | ...                       | ...               | ...                | ...               | ...                          | ...                     |
+| [continuous delivery](https://docs.gitlab.com/ee/ci/introduction/#continuous-delivery) (automated, but deploy manual) | semi-automated (GUI + `ansible` executed manually) | container ([semver](https://semver.org) versioned) | semi-automated (`ansible` executed manually) | semi-automated (`ansible` executed manually) | semi-automated (`ansible` executed manually) | semi-automated (`ansible` and `ssh` manually)  |
+| [continuous deployment](https://docs.gitlab.com/ee/ci/introduction/#continuous-deployment) (fully-automated) | automated (`terraform` and/or `ansible` in CI/CD) | container ([semver](https://semver.org) versioned) | automated (`ansible` in CI/CD) | automated (`ansible` in CI/CD) | automated (`ansible` in CI/CD) | semi-automated (`ansible` in CI/CD and `ssh` manually)  |
+
+
+
+## How to convert legacy application into a container for a cloud?
+
+Containerization of applications is one of the best practices when you want to share your application and execute in a cloud. Read about [the benefits](https://cloud.google.com/containers).
+
+Application containerization process consists of following steps:
+ * Select a container registry (where container images with your applications are stored)
+   * Publicly available registries like [quay.io](https://quay.io) are best as everyone may receive your application even without credentials
+ * Your project applications should be containerized via creating a `Dockerfile` ([example](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi/-/blob/master/Dockerfile))
+   * Follow [docker guide](https://www.freecodecamp.org/news/a-beginners-guide-to-docker-how-to-create-your-first-docker-application-cc03de9b639f/) if you are not familiar with `Dockerfile` syntax
+   * If your project is huge and contains multiple applications, then it is recommended to divide them in few parts by topic each part building separate container.
+ * Project CI/CD jobs should build applications, create container image[s] and finally release (push) container image[s] with applications to container registry
+ * Everyone is then able to use your applications (packaged in a container image) regardless of which Operating System (OS) he or she uses. Container engine (docker, podman, ...) is available for all mainstream OSes.
+   * Cloud resources are then told to pull and run your container image[s]. ([example](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi/-/blob/de673766b832c48142c6ad1be73f5bce046b02a2/ansible/roles/cloud-project-container/tasks/deploy.yml#L11-28))
+
+Learn best-practices on our cloud example [project `cloud-estimate-pi`](https://gitlab.ics.muni.cz/cloud/cloud-estimate-pi).
+
+
+
+
+
+

faq/README.md

 # Frequently Asked Questions
 
+## Where I can find how to use MetaCentrum Cloud effectively?
+Read our [cloud best-practice tips](/register/README.md).
+
+## What to expect from the cloud and cloud computing
+
+[Migration of Legacy Systems to Cloud Computing](https://www.researchgate.net/publication/280154501_Migration_of_Legacy_Systems_to_Cloud_Computing) article gives the overwiew what to expect when joining a cloud with personal legacy application.
+
+###  What are the cloud computing benefits?
+
+The most visible [cloud computing](https://en.wikipedia.org/wiki/Cloud_computing) benefits are:
+ * cost savings
+ * online access to the cloud resources for everyone authorized
+ * cloud project scalability (elasticity)
+ * online cloud resource management and improved sustainability
+ * security and privacy improvements
+ * encouraged cloud project agility
+
 ## How do I register?
 Follow instructions for registering in [MetaCentrum Cloud](/register/README.md).