Diego Vasquez
04/04/2022, 9:25 PMjuanlu
Diego Vasquez
04/04/2022, 10:21 PMjuanlu
Jacopo
04/05/2022, 7:41 AMDiego Vasquez
04/05/2022, 3:31 PMjuanlu
Diego Vasquez
04/05/2022, 3:39 PMorchest installation
docker
sudo apt-get remove docker docker-engine <http://docker.io|docker.io> containerd runc
sudo apt-get update
sudo apt-get install \
ca-certificates \
curl \
gnupg \
lsb-release
curl -fsSL <https://download.docker.com/linux/ubuntu/gpg> | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] <https://download.docker.com/linux/ubuntu> \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli <http://containerd.io|containerd.io>
sudo docker run hello-world
sudo usermod -aG docker $USER && newgrp docker
sudo nano /lib/systemd/system/docker.service
# change entry to volumes
ExecStart=/usr/bin/dockerd --data-root /mnt/docker/volumes/ -H fd:// $DOCKER_OPTS
sudo service docker restart
minikube
curl -LO <https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64>
sudo install minikube-linux-amd64 /usr/local/bin/minikube
minikube start
helm
wget <https://get.helm.sh/helm-v3.8.1-linux-amd64.tar.gz>
tar -zxvf helm-v3.8.1-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm
helm help
kubectl
curl -LO "<https://dl.k8s.io/release/$(curl> -L -s <https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl>"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
orchest
sudo apt-get install build-essential
git clone <https://github.com/orchest/orchest.git> && cd orchest
./orchest install
# add minikube ip
minikube ip
sudo nano /etc/hosts
nginx
sudo apt-get install nginx
sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/localorchest.io
sudo ln -s /etc/nginx/sites-available/localorchest.io /etc/nginx/sites-enabled/
sudo nano /etc/nginx/sites-available/localorchest.io
server {
listen 80 default_server;
listen [::]:80 default_server;
#
server_name orchest;
#
root /var/www/example.com;
index index.html;
location / {
proxy_pass <http://localorchest.io>;
}
}
sudo nano /etc/nginx/sites-available/default
# remove default_server
sudo service nginx restart
Yannick
# jupyter-eg
kubectl -n orchest logs \
$(kubectl get pods --namespace orchest -l "app=jupyter-eg" -o jsonpath="{.items[0].metadata.name}")
# jupyter-server
kubectl -n orchest logs \
$(kubectl get pods --namespace orchest -l "app=jupyter-server" -o jsonpath="{.items[0].metadata.name}")
# kernel logs
kubectl -n orchest logs \
$(kubectl get pods --namespace orchest -l "app=enterprise-gateway" -o jsonpath="{.items[0].metadata.name}")
I am not sure why the kernel fails to connect, I hope the logs are of help here.
On another note, I saw that you start the minikube cluster with minikube start
. I would recommend doing minikube start --cpus=4
(or even more CPUs if you have more available), this does require a minikube delete
first though.Diego Vasquez
04/05/2022, 4:45 PM/usr/local/bin/bootstrap-kernel.sh env: RABBITMQ_SERVER_PORT_5672_TCP=<tcp://10.96.164.73:5672> SHELL=/bin/bash JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_ADDR=10.101.109.24 KUBERNETES_SERVICE_PORT_HTTPS=443 EG_MIRROR_WORKING_DIRS=True ORCHEST_API_PORT_80_TCP_ADDR=10.107.113.77 DOCKER_REGISTRY_SERVICE_HOST=10.96.145.139 UPDATE_SIDECAR_SERVICE_PORT=80 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_ADDR=10.104.209.251 KUBERNETES_SERVICE_PORT=443 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP=<tcp://10.101.109.24:8888> ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT=<tcp://10.110.235.235:2746> UPDATE_SIDECAR_PORT_80_TCP=<tcp://10.111.210.62:80> CONDA_EXE=/opt/conda/bin/conda _CE_M= EG_NAMESPACE=orchest JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_HOST=10.99.59.44 RABBITMQ_SERVER_PORT_5672_TCP_PORT=5672 EG_IMPERSONATION_ENABLED=False HOSTNAME=jupyter-eg-932b2d80-7cb6-4fe462615a7c-ce20-4121-6cbd658cd568kl5 LANGUAGE=en_US.UTF-8 KERNEL_SPARK_CONTEXT_INIT_MODE=none EG_SHARED_NAMESPACE=True EG_ENV_PROCESS_WHITELIST=ORCHEST_PIPELINE_UUID,ORCHEST_PIPELINE_PATH,ORCHEST_PROJECT_UUID,ORCHEST_USERDIR_PVC,ORCHEST_PROJECT_DIR,ORCHEST_PIPELINE_FILE,ORCHEST_HOST_GID,ORCHEST_SESSION_UUID,ORCHEST_SESSION_TYPE,ORCHEST_GPU_ENABLED_INSTANCE,ORCHEST_REGISTRY EG_PORT=8888 EG_LOG_LEVEL=10 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP=<tcp://10.97.27.138:80> JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT=<tcp://10.104.209.251:80> CONDA_ENV=base JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_PROTO=tcp EG_ALLOW_ORIGIN=* KERNEL_ID=14ef3137-13b9-4d8d-b6ce-beaf972a11d6 EG_KERNEL_WHITELIST= NB_UID=1000 DOCKER_REGISTRY_PORT_443_TCP_ADDR=10.96.145.139 KERNEL_UID=1000 KERNEL_IMAGE=orchest-env-932b2d80-7cb6-4fe4-a4c6-ca59240cec9b-378cd435-c476-45b3-b425-dfd451928a96 EG_PROHIBITED_UIDS=[] UPDATE_SIDECAR_SERVICE_HOST=10.111.210.62 DOCKER_REGISTRY_SERVICE_PORT_HTTPS_443=443 KERNEL_LAUNCH_TIMEOUT=40 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT=<tcp://10.102.197.135:8888> ORCHEST_API_PORT=<tcp://10.107.113.77:80> PWD=/project-dir EG_CULL_IDLE_TIMEOUT=43200 CONDA_PREFIX=/opt/conda JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP=<tcp://10.99.59.44:80> JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_PROTO=tcp ORCHEST_PROJECT_UUID=932b2d80-7cb6-4fe4-a4c6-ca59240cec9b ORCHEST_API_PORT_80_TCP_PORT=80 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_PROTO=tcp EG_DEFAULT_KERNEL_NAME=python_docker ORCHEST_DATABASE_SERVICE_HOST=10.96.198.198 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_PORT=80 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_HOST=10.97.27.138 KERNEL_SERVICE_ACCOUNT_NAME=default EG_RESPONSE_ADDRESS=172.17.0.19:48105 ORCHEST_DATABASE_PORT_5432_TCP=<tcp://10.96.198.198:5432> JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_PORT=8888 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_SERVICE_HOST=10.110.235.235 UPDATE_SIDECAR_PORT_80_TCP_ADDR=10.111.210.62 PLOTLY_RENDERER=iframe AUTH_SERVER_SERVICE_HOST=10.110.178.112 EG_ENABLE_TUNNELING=False ORCHEST_WEBSERVER_SERVICE_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_PORT=8888 EG_KERNEL_LAUNCH_TIMEOUT=600 HOME=/home/jovyan ORCHEST_WEBSERVER_PORT_80_TCP_ADDR=10.106.49.193 ORCHEST_API_PORT_80_TCP_PROTO=tcp JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT=<tcp://10.99.59.44:80> ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP=<tcp://10.110.235.235:2746> JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_ADDR=10.97.27.138 LANG=en_US.UTF-8 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT=<tcp://10.110.53.237:8888> KUBERNETES_PORT_443_TCP=<tcp://10.96.0.1:443> EG_LIST_KERNELS=True ORCHEST_PIPELINE_UUID=62615a7c-ce20-4121-935b-bf6d5bf5e163 EG_SSH_PORT=2122 ORCHEST_USERDIR_PVC=userdir-pvc AUTH_SERVER_PORT=<tcp://10.110.178.112:80> NB_GID=100 UPDATE_SIDECAR_PORT_80_TCP_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP=<tcp://10.110.53.237:8888> ORCHEST_VERSION=v2022.03.8 CONDA_PROMPT_MODIFIER=(base) RABBITMQ_SERVER_PORT=<tcp://10.96.164.73:5672> ORCHEST_DATABASE_PORT_5432_TCP_PORT=5432 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP=<tcp://10.102.197.135:8888> PYDEVD_USE_FRAME_EVAL=NO EG_UID_BLACKLIST=["-1"] UPDATE_SIDECAR_PORT=<tcp://10.111.210.62:80> ORCHEST_API_PORT_80_TCP=<tcp://10.107.113.77:80> JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT=<tcp://10.97.27.138:80> ORCHEST_PIPELINE_FILE=test_rechazos.orchest ORCHEST_WEBSERVER_PORT_80_TCP_PROTO=tcp ORCHEST_DATABASE_PORT_5432_TCP_ADDR=10.96.198.198 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_PROTO=tcp KG_PORT=8888 DOCKER_REGISTRY_PORT=<tcp://10.96.145.139:443> ORCHEST_API_SERVICE_PORT=80 EG_CULL_CONNECTED=False ORCHEST_API_SERVICE_HOST=10.107.113.77 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_PORT=8888 EG_PORT_RETRIES=0 RABBITMQ_SERVER_PORT_5672_TCP_PROTO=tcp EG_MIN_PORT_RANGE_SIZE=1000 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_SERVICE_PORT=2746 ORCHEST_GPU_ENABLED_INSTANCE=False JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_PROTO=tcp EG_BASE_URL=/jupyter-server-932b2d80-7cb6-4fe462615a7c-ce20-4121 AUTH_SERVER_PORT_80_TCP_PORT=80 RABBITMQ_SERVER_SERVICE_PORT=5672 KG_IP=0.0.0.0 JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_PORT=80 RABBITMQ_SERVER_PORT_5672_TCP_ADDR=10.96.164.73 JPY_PARENT_PID=10 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_HOST=10.104.209.251 ORCHEST_WEBSERVER_PORT_80_TCP=<tcp://10.106.49.193:80> JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_PORT=8888 EG_CULL_INTERVAL=60 _CE_CONDA= UPDATE_SIDECAR_PORT_80_TCP_PROTO=tcp ORCHEST_WEBSERVER_SERVICE_HOST=10.106.49.193 CONDA_SHLVL=1 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_PORT=80 ORCHEST_WEBSERVER_PORT=<tcp://10.106.49.193:80> KERNEL_POD_NAME=root-14ef3137-13b9-4d8d-b6ce-beaf972a11d6 EG_IP=0.0.0.0 SHLVL=1 ORCHEST_HOST_GID=1 DOCKER_REGISTRY_SERVICE_PORT=443 CONDA_DIR=/opt/conda KERNEL_GATEWAY=1 KUBERNETES_PORT_443_TCP_PROTO=tcp JUPYTER_PATH=/opt/conda/share/jupyter ORCHEST_WEBSERVER_PORT_80_TCP_PORT=80 KG_PORT_RETRIES=0 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_ADDR=10.110.53.237 ORCHEST_PROJECT_DIR=/userdir/projects/Chile KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 ORCHEST_REGISTRY=10.96.145.139 RABBITMQ_SERVER_SERVICE_HOST=10.96.164.73 KERNEL_WORKING_DIR=/project-dir KERNEL_USERNAME=root CONDA_PYTHON_EXE=/opt/conda/bin/python JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_ADDR=10.102.197.135 ORCHEST_SESSION_UUID=932b2d80-7cb6-4fe462615a7c-ce20-4121 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP=<tcp://10.104.209.251:80> JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_PROTO=tcp KERNEL_LANGUAGE=python CONDA_DEFAULT_ENV=base JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_PORT=8888 JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_ADDR=10.99.59.44 ORCHEST_DATABASE_SERVICE_PORT=5432 ORCHEST_DATABASE_PORT_5432_TCP_PROTO=tcp JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_PROTO=tcp NB_USER=jovyan KUBERNETES_SERVICE_HOST=10.96.0.1 LC_ALL=en_US.UTF-8 KUBERNETES_PORT=<tcp://10.96.0.1:443> DOCKER_REGISTRY_PORT_443_TCP=<tcp://10.96.145.139:443> KUBERNETES_PORT_443_TCP_PORT=443 ORCHEST_DATABASE_PORT=<tcp://10.96.198.198:5432> ORCHEST_PIPELINE_PATH=/pipeline.json AUTH_SERVER_PORT_80_TCP_PROTO=tcp PATH=/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin KERNEL_EXECUTOR_IMAGE=orchest-env-932b2d80-7cb6-4fe4-a4c6-ca59240cec9b-378cd435-c476-45b3-b425-dfd451928a96 AUTH_SERVER_SERVICE_PORT=80 DOCKER_REGISTRY_PORT_443_TCP_PROTO=tcp KERNEL_GID=100 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_PORT=8888 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_HOST=10.102.197.135 EG_UNAUTHORIZED_USERS=["dummy"] ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_PORT=2746 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_HOST=10.110.53.237 AUTH_SERVER_PORT_80_TCP_ADDR=10.110.178.112 DOCKER_REGISTRY_PORT_443_TCP_PORT=443 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_HOST=10.101.109.24 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_ADDR=10.110.235.235 DEBIAN_FRONTEND=noninteractive EG_MAX_PORT_RANGE_RETRIES=5 AUTH_SERVER_PORT_80_TCP=<tcp://10.110.178.112:80> JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_PORT=80 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT=<tcp://10.101.109.24:8888> KERNEL_NAMESPACE=orchest ORCHEST_SESSION_TYPE=interactive JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_PORT=80 _=/usr/bin/env
+ python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --RemoteProcessProxy.kernel-id 14ef3137-13b9-4d8d-b6ce-beaf972a11d6 --RemoteProcessProxy.port-range 0..0 --RemoteProcessProxy.response-address 172.17.0.19:48105 --RemoteProcessProxy.spark-context-initialization-mode none
[D 2022-04-05 17:04:10,090.090 launch_ipykernel] Using connection file '/tmp/kernel-14ef3137-13b9-4d8d-b6ce-beaf972a11d6_9v2d68pr.json'.
[I 2022-04-05 17:04:10,090.090 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 52413
[D 2022-04-05 17:04:10,091.091 launch_ipykernel] JSON Payload 'b'{"shell_port": 53173, "iopub_port": 40457, "stdin_port": 37341, "control_port": 60105, "hb_port": 48173, "ip": "0.0.0.0", "key": "de8f896c-14a1-45cd-9495-f52c73686760", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": "23", "pgid": "7", "comm_port": 52413}'
[D 2022-04-05 17:04:10,092.092 launch_ipykernel] Encrypted Payload 'b'N6tlrHYQmKwKKmmUopJyIHBXhot29BqWDMQasari0z0+BnpRpmR2WhnsAqs8Q+ytIq+jPruxiX3Bkb0MrHyBkpK16eMJ/MZmowEtIoNCinvLm+IdDleEaqYcAcQWvxiA+Df7K//qm9W5S3qGq3KYHaCTn6ygXMl7ejzGgqbu6kL9qqyxT809028YbRoKqOgsI2XoPTXDgtbAXtXo7FbegyWphqMI0IDt5cTHlAnxI+0q/P5UE6VERT6Ps1kRWDLwrymKP551lhePezFuqw632OsRfVt1Z35XEngee01Nm4yndqpi0iIFsMPaSnYITa1j14KlxTOXeHWzac3iTDCbafQGXspXETqns1brr82PJXTzMwVuLLLyXKE/lMxHG1g36/0q6QGlOcyP3WS9hUBCzg=='
Yannick
Diego Vasquez
04/05/2022, 5:33 PMYannick
Open a Notebook and trigger the kernel problemso that it also triggers the
disconnected
.
The logs you've shared don't point to anything abnormal 🤔Diego Vasquez
04/05/2022, 5:37 PMdeploy@orchest-k8s:~$ kubectl -n orchest logs \
> $(kubectl get pods --namespace orchest -l "app=enterprise-gateway" -o jsonpath="{.items[0].metadata.name}")
/usr/local/bin/bootstrap-kernel.sh env: RABBITMQ_SERVER_PORT_5672_TCP=<tcp://10.96.164.73:5672> SHELL=/bin/bash JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_ADDR=10.101.109.24 KUBERNETES_SERVICE_PORT_HTTPS=443 EG_MIRROR_WORKING_DIRS=True ORCHEST_API_PORT_80_TCP_ADDR=10.107.113.77 DOCKER_REGISTRY_SERVICE_HOST=10.96.145.139 UPDATE_SIDECAR_SERVICE_PORT=80 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_ADDR=10.104.209.251 KUBERNETES_SERVICE_PORT=443 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP=<tcp://10.101.109.24:8888> ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT=<tcp://10.110.235.235:2746> UPDATE_SIDECAR_PORT_80_TCP=<tcp://10.111.210.62:80> CONDA_EXE=/opt/conda/bin/conda _CE_M= EG_NAMESPACE=orchest JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_HOST=10.101.71.238 RABBITMQ_SERVER_PORT_5672_TCP_PORT=5672 EG_IMPERSONATION_ENABLED=False HOSTNAME=jupyter-eg-932b2d80-7cb6-4fe462615a7c-ce20-4121-6cbd658cd5tlhvp LANGUAGE=en_US.UTF-8 KERNEL_SPARK_CONTEXT_INIT_MODE=none EG_SHARED_NAMESPACE=True EG_ENV_PROCESS_WHITELIST=ORCHEST_PIPELINE_UUID,ORCHEST_PIPELINE_PATH,ORCHEST_PROJECT_UUID,ORCHEST_USERDIR_PVC,ORCHEST_PROJECT_DIR,ORCHEST_PIPELINE_FILE,ORCHEST_HOST_GID,ORCHEST_SESSION_UUID,ORCHEST_SESSION_TYPE,ORCHEST_GPU_ENABLED_INSTANCE,ORCHEST_REGISTRY EG_PORT=8888 EG_LOG_LEVEL=10 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP=<tcp://10.97.27.138:80> JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT=<tcp://10.104.209.251:80> CONDA_ENV=base JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_PROTO=tcp EG_ALLOW_ORIGIN=* KERNEL_ID=1be178e4-fdc7-4267-b2d7-2ee01b3eda91 EG_KERNEL_WHITELIST= NB_UID=1000 DOCKER_REGISTRY_PORT_443_TCP_ADDR=10.96.145.139 KERNEL_UID=1000 KERNEL_IMAGE=orchest-env-932b2d80-7cb6-4fe4-a4c6-ca59240cec9b-378cd435-c476-45b3-b425-dfd451928a96 EG_PROHIBITED_UIDS=[] UPDATE_SIDECAR_SERVICE_HOST=10.111.210.62 DOCKER_REGISTRY_SERVICE_PORT_HTTPS_443=443 KERNEL_LAUNCH_TIMEOUT=40 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT=<tcp://10.102.197.135:8888> ORCHEST_API_PORT=<tcp://10.107.113.77:80> PWD=/project-dir EG_CULL_IDLE_TIMEOUT=43200 CONDA_PREFIX=/opt/conda JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_PROTO=tcp JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP=<tcp://10.101.71.238:80> ORCHEST_PROJECT_UUID=932b2d80-7cb6-4fe4-a4c6-ca59240cec9b ORCHEST_API_PORT_80_TCP_PORT=80 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_PROTO=tcp EG_DEFAULT_KERNEL_NAME=python_docker ORCHEST_DATABASE_SERVICE_HOST=10.96.198.198 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_PORT=80 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_HOST=10.97.27.138 KERNEL_SERVICE_ACCOUNT_NAME=default EG_RESPONSE_ADDRESS=172.17.0.18:56853 ORCHEST_DATABASE_PORT_5432_TCP=<tcp://10.96.198.198:5432> JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_PORT=8888 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_SERVICE_HOST=10.110.235.235 UPDATE_SIDECAR_PORT_80_TCP_ADDR=10.111.210.62 PLOTLY_RENDERER=iframe AUTH_SERVER_SERVICE_HOST=10.110.178.112 EG_ENABLE_TUNNELING=False ORCHEST_WEBSERVER_SERVICE_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_PORT=8888 EG_KERNEL_LAUNCH_TIMEOUT=600 HOME=/home/jovyan ORCHEST_WEBSERVER_PORT_80_TCP_ADDR=10.106.49.193 ORCHEST_API_PORT_80_TCP_PROTO=tcp ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP=<tcp://10.110.235.235:2746> JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT=<tcp://10.101.71.238:80> JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_ADDR=10.97.27.138 LANG=en_US.UTF-8 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT=<tcp://10.98.31.174:8888> KUBERNETES_PORT_443_TCP=<tcp://10.96.0.1:443> EG_LIST_KERNELS=True ORCHEST_PIPELINE_UUID=62615a7c-ce20-4121-935b-bf6d5bf5e163 EG_SSH_PORT=2122 ORCHEST_USERDIR_PVC=userdir-pvc AUTH_SERVER_PORT=<tcp://10.110.178.112:80> NB_GID=100 UPDATE_SIDECAR_PORT_80_TCP_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP=<tcp://10.98.31.174:8888> ORCHEST_VERSION=v2022.03.8 CONDA_PROMPT_MODIFIER=(base) RABBITMQ_SERVER_PORT=<tcp://10.96.164.73:5672> ORCHEST_DATABASE_PORT_5432_TCP_PORT=5432 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP=<tcp://10.102.197.135:8888> PYDEVD_USE_FRAME_EVAL=NO EG_UID_BLACKLIST=["-1"] UPDATE_SIDECAR_PORT=<tcp://10.111.210.62:80> ORCHEST_API_PORT_80_TCP=<tcp://10.107.113.77:80> JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT=<tcp://10.97.27.138:80> ORCHEST_PIPELINE_FILE=test_rechazos.orchest ORCHEST_WEBSERVER_PORT_80_TCP_PROTO=tcp ORCHEST_DATABASE_PORT_5432_TCP_ADDR=10.96.198.198 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_PROTO=tcp KG_PORT=8888 DOCKER_REGISTRY_PORT=<tcp://10.96.145.139:443> ORCHEST_API_SERVICE_PORT=80 EG_CULL_CONNECTED=False ORCHEST_API_SERVICE_HOST=10.107.113.77 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_PORT=8888 EG_PORT_RETRIES=0 RABBITMQ_SERVER_PORT_5672_TCP_PROTO=tcp EG_MIN_PORT_RANGE_SIZE=1000 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_SERVICE_PORT=2746 ORCHEST_GPU_ENABLED_INSTANCE=False JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_PROTO=tcp EG_BASE_URL=/jupyter-server-932b2d80-7cb6-4fe462615a7c-ce20-4121 AUTH_SERVER_PORT_80_TCP_PORT=80 RABBITMQ_SERVER_SERVICE_PORT=5672 KG_IP=0.0.0.0 JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_PORT=80 RABBITMQ_SERVER_PORT_5672_TCP_ADDR=10.96.164.73 JPY_PARENT_PID=8 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_HOST=10.104.209.251 ORCHEST_WEBSERVER_PORT_80_TCP=<tcp://10.106.49.193:80> JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_PORT=8888 EG_CULL_INTERVAL=60 _CE_CONDA= UPDATE_SIDECAR_PORT_80_TCP_PROTO=tcp ORCHEST_WEBSERVER_SERVICE_HOST=10.106.49.193 CONDA_SHLVL=1 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_PORT=80 ORCHEST_WEBSERVER_PORT=<tcp://10.106.49.193:80> KERNEL_POD_NAME=root-1be178e4-fdc7-4267-b2d7-2ee01b3eda91 EG_IP=0.0.0.0 SHLVL=1 ORCHEST_HOST_GID=1 DOCKER_REGISTRY_SERVICE_PORT=443 CONDA_DIR=/opt/conda KERNEL_GATEWAY=1 KUBERNETES_PORT_443_TCP_PROTO=tcp JUPYTER_PATH=/opt/conda/share/jupyter ORCHEST_WEBSERVER_PORT_80_TCP_PORT=80 KG_PORT_RETRIES=0 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_ADDR=10.98.31.174 ORCHEST_PROJECT_DIR=/userdir/projects/Chile KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 ORCHEST_REGISTRY=10.96.145.139 RABBITMQ_SERVER_SERVICE_HOST=10.96.164.73 KERNEL_WORKING_DIR=/project-dir KERNEL_USERNAME=root CONDA_PYTHON_EXE=/opt/conda/bin/python JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_ADDR=10.102.197.135 ORCHEST_SESSION_UUID=932b2d80-7cb6-4fe462615a7c-ce20-4121 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP=<tcp://10.104.209.251:80> JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_PROTO=tcp KERNEL_LANGUAGE=python CONDA_DEFAULT_ENV=base JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_PORT=8888 JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_ADDR=10.101.71.238 ORCHEST_DATABASE_SERVICE_PORT=5432 ORCHEST_DATABASE_PORT_5432_TCP_PROTO=tcp JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_PROTO=tcp NB_USER=jovyan KUBERNETES_SERVICE_HOST=10.96.0.1 LC_ALL=en_US.UTF-8 KUBERNETES_PORT=<tcp://10.96.0.1:443> DOCKER_REGISTRY_PORT_443_TCP=<tcp://10.96.145.139:443> KUBERNETES_PORT_443_TCP_PORT=443 ORCHEST_DATABASE_PORT=<tcp://10.96.198.198:5432> ORCHEST_PIPELINE_PATH=/pipeline.json AUTH_SERVER_PORT_80_TCP_PROTO=tcp PATH=/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin KERNEL_EXECUTOR_IMAGE=orchest-env-932b2d80-7cb6-4fe4-a4c6-ca59240cec9b-378cd435-c476-45b3-b425-dfd451928a96 AUTH_SERVER_SERVICE_PORT=80 DOCKER_REGISTRY_PORT_443_TCP_PROTO=tcp KERNEL_GID=100 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_PORT=8888 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_HOST=10.102.197.135 EG_UNAUTHORIZED_USERS=["dummy"] ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_PORT=2746 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_HOST=10.98.31.174 AUTH_SERVER_PORT_80_TCP_ADDR=10.110.178.112 DOCKER_REGISTRY_PORT_443_TCP_PORT=443 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_HOST=10.101.109.24 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_ADDR=10.110.235.235 DEBIAN_FRONTEND=noninteractive EG_MAX_PORT_RANGE_RETRIES=5 AUTH_SERVER_PORT_80_TCP=<tcp://10.110.178.112:80> JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_PORT=80 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT=<tcp://10.101.109.24:8888> KERNEL_NAMESPACE=orchest ORCHEST_SESSION_TYPE=interactive JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_PORT=80 _=/usr/bin/env
+ python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --RemoteProcessProxy.kernel-id 1be178e4-fdc7-4267-b2d7-2ee01b3eda91 --RemoteProcessProxy.port-range 0..0 --RemoteProcessProxy.response-address 172.17.0.18:56853 --RemoteProcessProxy.spark-context-initialization-mode none
[D 2022-04-05 17:38:42,633.633 launch_ipykernel] Using connection file '/tmp/kernel-1be178e4-fdc7-4267-b2d7-2ee01b3eda91_qyazgm4t.json'.
[I 2022-04-05 17:38:42,633.633 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 47105
[D 2022-04-05 17:38:42,634.634 launch_ipykernel] JSON Payload 'b'{"shell_port": 45391, "iopub_port": 51123, "stdin_port": 40961, "control_port": 37819, "hb_port": 54241, "ip": "0.0.0.0", "key": "5cda7bd8-019c-4608-998b-e921aca848c0", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": "23", "pgid": "7", "comm_port": 47105}'
[D 2022-04-05 17:38:42,635.635 launch_ipykernel] Encrypted Payload 'b'dQTVwTDQT3E6b/k0ohxogoTSpyd41lEPhtU0fsQfgYpYyETq+uwLONhDvafPyLoTSu46lsRDi2K3QWt/VrZvm7vS/Seb4X1pI1rEiNmA2PbOKOm0hfJpnSpaUSJvpR6L/pRivZ3NuEjMcgZBayhGX8S3o+imMjHIXSlQ9/3h4onX44pDhlvdrtsJStIio+mvklX05+90dSZAA5ZDD7AOhRjVwwP0c9iU/lfU7iOyyHQM2NcDR9Pny0ZOMKPGa3cx+wbyPcp2nuZqN3brbDLcSBkfQivZeng+vZCGUOYiJDGZPJsezfFL94Z9430ZC57YfX/fgKN/kQwumr8HYwReuiJnmrudPovmSPH8sOtuRg2LNjsajf+hdHZrN2lRGItOohf8ip2D0DS4PDvhwSLJEA=='
Yannick
EG_KERNEL_LAUNCH_TIMEOUT
to 600s
and so it takes a long time before the enterprise gateway (EG) will attempt to start another kernel (but I don't expect this to work the second time), explaining the long Disconnected...
in the bottom of the JupyterLab screen.
I see you provided the steps to install Orchest, could you explain the machine/environment on which you ran these commands? Maybe that is something to look intoDiego Vasquez
04/05/2022, 6:21 PMYannick
Diego Vasquez
04/05/2022, 6:35 PMJacopo
04/05/2022, 6:41 PMminikube --cpus <as much as possible --memory <as much as possible> --disk-size <as much as possible>
example
minikube start --cpus 6 --memory 16g --disk-size 50g
Diego Vasquez
04/05/2022, 6:51 PMJacopo
04/05/2022, 6:52 PMDiego Vasquez
04/05/2022, 6:53 PMJacopo
04/05/2022, 6:56 PMjuanlu
minikube
on Ubuntu systems https://github.com/kubernetes/minikube/issues/12842 so it might be better to not pass the --memory
flag at allJacopo
04/05/2022, 6:59 PMDiego Vasquez
04/05/2022, 7:10 PMJacopo
04/06/2022, 8:47 AM/etc/nginx/sites-available/localorchest.io
to include websocket support will fix the issue.
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name orchest;
root /var/www/example.com;
index index.html;
location / {
proxy_pass <http://localorchest.io>;
# WebSocket support.
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 86400;
}
}
Then
sudo service nginx restart
Diego Vasquez
04/06/2022, 2:17 PMJacopo
04/06/2022, 2:24 PMDiego Vasquez
04/06/2022, 2:30 PMJacopo
04/06/2022, 2:35 PMDiego Vasquez
04/06/2022, 8:48 PMJacopo
04/07/2022, 6:26 AM