Starterpack

This repo is dedicated to get you started using a custom k8s deployment developement environment that you can ssh to.

How to set up

Prerequisites

• Have an e-infra account
• Have an access to the fi-lmi-ns namespace on kubernetes and fi-lmi-data group in metacentrum
• Have kubernetes set up (custom config file, kubectl installed), etc.

If you don't have any of those prerequisites checked out, let me know at slaninakova@ics.muni.cz

Setting up the custom branch / environment

1. Create a custom branch from master, name it appropriately (e.g. mine is called 'terka')
2. Rename:
   • <username> in deployment.yaml with your <username> (where <username> can be anything you want, just use it consistently throughout this tutorial)
   • do the same for in start.sh
3. Log into any metacentrum front-end (e.g. skirit.metacentrum.cz), navigate to /storage/brno12-cerit/projects/fi-lmi-data and create your personal folder - mkdir <username>
4. Make sure you have kubectl, if you don't, follow this tutorial: docs.cerit.io/docs/kubectl.html
5. Create a custom ssh key using kubectl, so that you can authenticate yourself to the deployment. To do that:

• generate your key (e.g. ssh-keygen -t rsa -b 4096 -C " ")
• copy the entire content of your public key to your clipboard (the start should look like this: ssh-rsa AAAAB...)
• run:

kubectl create secret generic -n fi-lmi-ns <username>-ssh-key --from-literal=<username>-ssh-key='<key>'

6. Commit and push your changes to the branch
7. Check out the pipeline triggered by the push in GitLab Note: If the pipeline fails on the following error:

error during connect: Post "https://docker:2376/v1.24/auth": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "docker:dind CA")

just ignore it and trigger a re-run

8. Check out rancher to see if the deployment is running

9. ssh to the pod using the following command:

ssh -v -p 2222 jovyan@<username>-ssh.dyn.cloud.e-infra.cz

or visit the jupyterlab via the following link:

https://<username>.dyn.cloud.e-infra.cz/

and check the deployment's logs in rancher to find out the authentication token.

Modifications

Feel free to modify Dockerfile or start.sh script to your liking. You can also increase resources in deployment.yaml if you need more memory or CPU. Allocating a GPU in deployments is also possible, but discouraged for long-term deployments such as this one, use kubernetes jobs instead.

Warning: Persisting data

The home directory of the user in the deployment is not persistent. It will survive thoughout login-logouts, but since running on kubernetes, the pod can be restarted at any time and you could lose data.

In order to make sure your data is persistent, you should use the created directory (/storage/brno12-cerit/projects/fi-lmi-data/<username> == /data/<username>) to store your data. Additionally, the directory will be automatically used to store your JupyterLab's configuration (see start.sh for details).

In case of any questions, feel free to contact me at slaninakova@ics.muni.cz