Title
d
Diego Vasquez
04/04/2022, 9:25 PMHi everyone. We've been using Orchest in the last couple of months and now we have some problems with the kernel connection of Jupyter Lab. We did a clean install on our server last week (we have now Orchest in version v2022.03.8).
Refferring to the problem with the kernel connection, when we run the script cells, in Jupyter Lab appears "Python 3 | Connecting" and after a few minutes, "Python 3 | Disconnected".
What could be the problem here? Because when we run our pipelines' jobs or even when we run scripts in .py files we don't have problems
j
juanlu
04/04/2022, 10:19 PMhi @Diego Vasquez, sorry you're having a bad experience. did you by any chance create a custom image or environment? or are you using just the defaults?
d
Diego Vasquez
04/04/2022, 10:21 PMI'm using the default python enviroment but with some other libraries like pykml, geopandas, twilio and others
j
juanlu
04/05/2022, 6:23 AMcan you share your setup script? to see if I can reproduce on my end, otherwise we'll try to look at the logs
j
Jacopo
04/05/2022, 7:41 AMHi @Diego Vasquez, are the kernels actually starting, and then cells time out? Or kernels aren't starting at all?
d
Diego Vasquez
04/05/2022, 3:31 PM@Jacopo Hi! Kernels are starting and then stop starting. The cells usually keep the * and after a while, stop without errors.
Do you need an example?
j
juanlu
04/05/2022, 3:38 PMyes please! @Diego Vasquez if you could record a screencast with any tool of your choice, that would help us understand the issue a bit
d
Diego Vasquez
04/05/2022, 3:39 PMorchest installation
docker
sudo apt-get remove docker docker-engine <http://docker.io|docker.io> containerd runc
sudo apt-get update
sudo apt-get install \
ca-certificates \
curl \
gnupg \
lsb-release
curl -fsSL <https://download.docker.com/linux/ubuntu/gpg> | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] <https://download.docker.com/linux/ubuntu> \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli <http://containerd.io|containerd.io>
sudo docker run hello-world
sudo usermod -aG docker $USER && newgrp docker
sudo nano /lib/systemd/system/docker.service
# change entry to volumes
ExecStart=/usr/bin/dockerd --data-root /mnt/docker/volumes/ -H fd:// $DOCKER_OPTS
sudo service docker restart
minikube
curl -LO <https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64>
sudo install minikube-linux-amd64 /usr/local/bin/minikube
minikube start
helm
wget <https://get.helm.sh/helm-v3.8.1-linux-amd64.tar.gz>
tar -zxvf helm-v3.8.1-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm
helm help
kubectl
curl -LO "<https://dl.k8s.io/release/$(curl> -L -s <https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl>"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
orchest
sudo apt-get install build-essential
git clone <https://github.com/orchest/orchest.git> && cd orchest
./orchest install
# add minikube ip
minikube ip
sudo nano /etc/hosts
nginx
sudo apt-get install nginx
sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/localorchest.io
sudo ln -s /etc/nginx/sites-available/localorchest.io /etc/nginx/sites-enabled/
sudo nano /etc/nginx/sites-available/localorchest.io
server {
listen 80 default_server;
listen [::]:80 default_server;
#
server_name orchest;
#
root /var/www/example.com;
index index.html;
location / {
proxy_pass <http://localorchest.io>;
}
}
sudo nano /etc/nginx/sites-available/default
# remove default_server
sudo service nginx restart@juanlu
in a few minutes I will share the screencast
👍🏼 1
@juanlu it's a bit longer, sorry
y
Yannick
04/05/2022, 4:42 PMHi @Diego Vasquez! Thank you for the elaborate response. Could you:
1. Stop the session
2. Start the session again
3. Open a Notebook and trigger the kernel problem
4. Share the output of the following commands (logs of the jupyter-enterprise-gateway, jupyter-server and kernel). Please make sure you are running only one kernel (so just opening just one notebook)
# jupyter-eg
kubectl -n orchest logs \
$(kubectl get pods --namespace orchest -l "app=jupyter-eg" -o jsonpath="{.items[0].metadata.name}")
# jupyter-server
kubectl -n orchest logs \
$(kubectl get pods --namespace orchest -l "app=jupyter-server" -o jsonpath="{.items[0].metadata.name}")
# kernel logs
kubectl -n orchest logs \
$(kubectl get pods --namespace orchest -l "app=enterprise-gateway" -o jsonpath="{.items[0].metadata.name}")minikube startminikube start --cpus=4minikube deleted
Diego Vasquez
04/05/2022, 4:45 PM@Yannick thanks for your reply. I will try it now
image.png,image.png
/usr/local/bin/bootstrap-kernel.sh env: RABBITMQ_SERVER_PORT_5672_TCP=<tcp://10.96.164.73:5672> SHELL=/bin/bash JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_ADDR=10.101.109.24 KUBERNETES_SERVICE_PORT_HTTPS=443 EG_MIRROR_WORKING_DIRS=True ORCHEST_API_PORT_80_TCP_ADDR=10.107.113.77 DOCKER_REGISTRY_SERVICE_HOST=10.96.145.139 UPDATE_SIDECAR_SERVICE_PORT=80 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_ADDR=10.104.209.251 KUBERNETES_SERVICE_PORT=443 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP=<tcp://10.101.109.24:8888> ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT=<tcp://10.110.235.235:2746> UPDATE_SIDECAR_PORT_80_TCP=<tcp://10.111.210.62:80> CONDA_EXE=/opt/conda/bin/conda _CE_M= EG_NAMESPACE=orchest JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_HOST=10.99.59.44 RABBITMQ_SERVER_PORT_5672_TCP_PORT=5672 EG_IMPERSONATION_ENABLED=False HOSTNAME=jupyter-eg-932b2d80-7cb6-4fe462615a7c-ce20-4121-6cbd658cd568kl5 LANGUAGE=en_US.UTF-8 KERNEL_SPARK_CONTEXT_INIT_MODE=none EG_SHARED_NAMESPACE=True EG_ENV_PROCESS_WHITELIST=ORCHEST_PIPELINE_UUID,ORCHEST_PIPELINE_PATH,ORCHEST_PROJECT_UUID,ORCHEST_USERDIR_PVC,ORCHEST_PROJECT_DIR,ORCHEST_PIPELINE_FILE,ORCHEST_HOST_GID,ORCHEST_SESSION_UUID,ORCHEST_SESSION_TYPE,ORCHEST_GPU_ENABLED_INSTANCE,ORCHEST_REGISTRY EG_PORT=8888 EG_LOG_LEVEL=10 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP=<tcp://10.97.27.138:80> JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT=<tcp://10.104.209.251:80> CONDA_ENV=base JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_PROTO=tcp EG_ALLOW_ORIGIN=* KERNEL_ID=14ef3137-13b9-4d8d-b6ce-beaf972a11d6 EG_KERNEL_WHITELIST= NB_UID=1000 DOCKER_REGISTRY_PORT_443_TCP_ADDR=10.96.145.139 KERNEL_UID=1000 KERNEL_IMAGE=orchest-env-932b2d80-7cb6-4fe4-a4c6-ca59240cec9b-378cd435-c476-45b3-b425-dfd451928a96 EG_PROHIBITED_UIDS=[] UPDATE_SIDECAR_SERVICE_HOST=10.111.210.62 DOCKER_REGISTRY_SERVICE_PORT_HTTPS_443=443 KERNEL_LAUNCH_TIMEOUT=40 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT=<tcp://10.102.197.135:8888> ORCHEST_API_PORT=<tcp://10.107.113.77:80> PWD=/project-dir EG_CULL_IDLE_TIMEOUT=43200 CONDA_PREFIX=/opt/conda JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP=<tcp://10.99.59.44:80> JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_PROTO=tcp ORCHEST_PROJECT_UUID=932b2d80-7cb6-4fe4-a4c6-ca59240cec9b ORCHEST_API_PORT_80_TCP_PORT=80 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_PROTO=tcp EG_DEFAULT_KERNEL_NAME=python_docker ORCHEST_DATABASE_SERVICE_HOST=10.96.198.198 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_PORT=80 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_HOST=10.97.27.138 KERNEL_SERVICE_ACCOUNT_NAME=default EG_RESPONSE_ADDRESS=172.17.0.19:48105 ORCHEST_DATABASE_PORT_5432_TCP=<tcp://10.96.198.198:5432> JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_PORT=8888 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_SERVICE_HOST=10.110.235.235 UPDATE_SIDECAR_PORT_80_TCP_ADDR=10.111.210.62 PLOTLY_RENDERER=iframe AUTH_SERVER_SERVICE_HOST=10.110.178.112 EG_ENABLE_TUNNELING=False ORCHEST_WEBSERVER_SERVICE_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_PORT=8888 EG_KERNEL_LAUNCH_TIMEOUT=600 HOME=/home/jovyan ORCHEST_WEBSERVER_PORT_80_TCP_ADDR=10.106.49.193 ORCHEST_API_PORT_80_TCP_PROTO=tcp JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT=<tcp://10.99.59.44:80> ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP=<tcp://10.110.235.235:2746> JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_ADDR=10.97.27.138 LANG=en_US.UTF-8 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT=<tcp://10.110.53.237:8888> KUBERNETES_PORT_443_TCP=<tcp://10.96.0.1:443> EG_LIST_KERNELS=True ORCHEST_PIPELINE_UUID=62615a7c-ce20-4121-935b-bf6d5bf5e163 EG_SSH_PORT=2122 ORCHEST_USERDIR_PVC=userdir-pvc AUTH_SERVER_PORT=<tcp://10.110.178.112:80> NB_GID=100 UPDATE_SIDECAR_PORT_80_TCP_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP=<tcp://10.110.53.237:8888> ORCHEST_VERSION=v2022.03.8 CONDA_PROMPT_MODIFIER=(base) RABBITMQ_SERVER_PORT=<tcp://10.96.164.73:5672> ORCHEST_DATABASE_PORT_5432_TCP_PORT=5432 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP=<tcp://10.102.197.135:8888> PYDEVD_USE_FRAME_EVAL=NO EG_UID_BLACKLIST=["-1"] UPDATE_SIDECAR_PORT=<tcp://10.111.210.62:80> ORCHEST_API_PORT_80_TCP=<tcp://10.107.113.77:80> JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT=<tcp://10.97.27.138:80> ORCHEST_PIPELINE_FILE=test_rechazos.orchest ORCHEST_WEBSERVER_PORT_80_TCP_PROTO=tcp ORCHEST_DATABASE_PORT_5432_TCP_ADDR=10.96.198.198 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_PROTO=tcp KG_PORT=8888 DOCKER_REGISTRY_PORT=<tcp://10.96.145.139:443> ORCHEST_API_SERVICE_PORT=80 EG_CULL_CONNECTED=False ORCHEST_API_SERVICE_HOST=10.107.113.77 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_PORT=8888 EG_PORT_RETRIES=0 RABBITMQ_SERVER_PORT_5672_TCP_PROTO=tcp EG_MIN_PORT_RANGE_SIZE=1000 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_SERVICE_PORT=2746 ORCHEST_GPU_ENABLED_INSTANCE=False JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_PROTO=tcp EG_BASE_URL=/jupyter-server-932b2d80-7cb6-4fe462615a7c-ce20-4121 AUTH_SERVER_PORT_80_TCP_PORT=80 RABBITMQ_SERVER_SERVICE_PORT=5672 KG_IP=0.0.0.0 JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_PORT=80 RABBITMQ_SERVER_PORT_5672_TCP_ADDR=10.96.164.73 JPY_PARENT_PID=10 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_HOST=10.104.209.251 ORCHEST_WEBSERVER_PORT_80_TCP=<tcp://10.106.49.193:80> JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_PORT=8888 EG_CULL_INTERVAL=60 _CE_CONDA= UPDATE_SIDECAR_PORT_80_TCP_PROTO=tcp ORCHEST_WEBSERVER_SERVICE_HOST=10.106.49.193 CONDA_SHLVL=1 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_PORT=80 ORCHEST_WEBSERVER_PORT=<tcp://10.106.49.193:80> KERNEL_POD_NAME=root-14ef3137-13b9-4d8d-b6ce-beaf972a11d6 EG_IP=0.0.0.0 SHLVL=1 ORCHEST_HOST_GID=1 DOCKER_REGISTRY_SERVICE_PORT=443 CONDA_DIR=/opt/conda KERNEL_GATEWAY=1 KUBERNETES_PORT_443_TCP_PROTO=tcp JUPYTER_PATH=/opt/conda/share/jupyter ORCHEST_WEBSERVER_PORT_80_TCP_PORT=80 KG_PORT_RETRIES=0 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_ADDR=10.110.53.237 ORCHEST_PROJECT_DIR=/userdir/projects/Chile KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 ORCHEST_REGISTRY=10.96.145.139 RABBITMQ_SERVER_SERVICE_HOST=10.96.164.73 KERNEL_WORKING_DIR=/project-dir KERNEL_USERNAME=root CONDA_PYTHON_EXE=/opt/conda/bin/python JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_ADDR=10.102.197.135 ORCHEST_SESSION_UUID=932b2d80-7cb6-4fe462615a7c-ce20-4121 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP=<tcp://10.104.209.251:80> JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_PROTO=tcp KERNEL_LANGUAGE=python CONDA_DEFAULT_ENV=base JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_PORT=8888 JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_ADDR=10.99.59.44 ORCHEST_DATABASE_SERVICE_PORT=5432 ORCHEST_DATABASE_PORT_5432_TCP_PROTO=tcp JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_PROTO=tcp NB_USER=jovyan KUBERNETES_SERVICE_HOST=10.96.0.1 LC_ALL=en_US.UTF-8 KUBERNETES_PORT=<tcp://10.96.0.1:443> DOCKER_REGISTRY_PORT_443_TCP=<tcp://10.96.145.139:443> KUBERNETES_PORT_443_TCP_PORT=443 ORCHEST_DATABASE_PORT=<tcp://10.96.198.198:5432> ORCHEST_PIPELINE_PATH=/pipeline.json AUTH_SERVER_PORT_80_TCP_PROTO=tcp PATH=/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin KERNEL_EXECUTOR_IMAGE=orchest-env-932b2d80-7cb6-4fe4-a4c6-ca59240cec9b-378cd435-c476-45b3-b425-dfd451928a96 AUTH_SERVER_SERVICE_PORT=80 DOCKER_REGISTRY_PORT_443_TCP_PROTO=tcp KERNEL_GID=100 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_PORT=8888 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_HOST=10.102.197.135 EG_UNAUTHORIZED_USERS=["dummy"] ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_PORT=2746 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_HOST=10.110.53.237 AUTH_SERVER_PORT_80_TCP_ADDR=10.110.178.112 DOCKER_REGISTRY_PORT_443_TCP_PORT=443 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_HOST=10.101.109.24 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_ADDR=10.110.235.235 DEBIAN_FRONTEND=noninteractive EG_MAX_PORT_RANGE_RETRIES=5 AUTH_SERVER_PORT_80_TCP=<tcp://10.110.178.112:80> JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_PORT=80 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT=<tcp://10.101.109.24:8888> KERNEL_NAMESPACE=orchest ORCHEST_SESSION_TYPE=interactive JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_PORT=80 _=/usr/bin/env
+ python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --RemoteProcessProxy.kernel-id 14ef3137-13b9-4d8d-b6ce-beaf972a11d6 --RemoteProcessProxy.port-range 0..0 --RemoteProcessProxy.response-address 172.17.0.19:48105 --RemoteProcessProxy.spark-context-initialization-mode none
[D 2022-04-05 17:04:10,090.090 launch_ipykernel] Using connection file '/tmp/kernel-14ef3137-13b9-4d8d-b6ce-beaf972a11d6_9v2d68pr.json'.
[I 2022-04-05 17:04:10,090.090 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 52413
[D 2022-04-05 17:04:10,091.091 launch_ipykernel] JSON Payload 'b'{"shell_port": 53173, "iopub_port": 40457, "stdin_port": 37341, "control_port": 60105, "hb_port": 48173, "ip": "0.0.0.0", "key": "de8f896c-14a1-45cd-9495-f52c73686760", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": "23", "pgid": "7", "comm_port": 52413}'
[D 2022-04-05 17:04:10,092.092 launch_ipykernel] Encrypted Payload 'b'N6tlrHYQmKwKKmmUopJyIHBXhot29BqWDMQasari0z0+BnpRpmR2WhnsAqs8Q+ytIq+jPruxiX3Bkb0MrHyBkpK16eMJ/MZmowEtIoNCinvLm+IdDleEaqYcAcQWvxiA+Df7K//qm9W5S3qGq3KYHaCTn6ygXMl7ejzGgqbu6kL9qqyxT809028YbRoKqOgsI2XoPTXDgtbAXtXo7FbegyWphqMI0IDt5cTHlAnxI+0q/P5UE6VERT6Ps1kRWDLwrymKP551lhePezFuqw632OsRfVt1Z35XEngee01Nm4yndqpi0iIFsMPaSnYITa1j14KlxTOXeHWzac3iTDCbafQGXspXETqns1brr82PJXTzMwVuLLLyXKE/lMxHG1g36/0q6QGlOcyP3WS9hUBCzg=='the last code block its from the third command @Yannick
y
Yannick
04/05/2022, 5:22 PMThanks. Were all commands run after starting the session and triggering the error?
d
Diego Vasquez
04/05/2022, 5:33 PMyes. I could run it again if you want
y
Yannick
04/05/2022, 5:36 PMMaybe adding a short wait after:
Open a Notebook and trigger the kernel problemso that it also triggers the
disconnectedd
Diego Vasquez
04/05/2022, 5:37 PMok, I will do it again, give me a second
👍 1
@Yannick
deploy@orchest-k8s:~$ kubectl -n orchest logs \
> $(kubectl get pods --namespace orchest -l "app=enterprise-gateway" -o jsonpath="{.items[0].metadata.name}")
/usr/local/bin/bootstrap-kernel.sh env: RABBITMQ_SERVER_PORT_5672_TCP=<tcp://10.96.164.73:5672> SHELL=/bin/bash JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_ADDR=10.101.109.24 KUBERNETES_SERVICE_PORT_HTTPS=443 EG_MIRROR_WORKING_DIRS=True ORCHEST_API_PORT_80_TCP_ADDR=10.107.113.77 DOCKER_REGISTRY_SERVICE_HOST=10.96.145.139 UPDATE_SIDECAR_SERVICE_PORT=80 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_ADDR=10.104.209.251 KUBERNETES_SERVICE_PORT=443 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP=<tcp://10.101.109.24:8888> ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT=<tcp://10.110.235.235:2746> UPDATE_SIDECAR_PORT_80_TCP=<tcp://10.111.210.62:80> CONDA_EXE=/opt/conda/bin/conda _CE_M= EG_NAMESPACE=orchest JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_HOST=10.101.71.238 RABBITMQ_SERVER_PORT_5672_TCP_PORT=5672 EG_IMPERSONATION_ENABLED=False HOSTNAME=jupyter-eg-932b2d80-7cb6-4fe462615a7c-ce20-4121-6cbd658cd5tlhvp LANGUAGE=en_US.UTF-8 KERNEL_SPARK_CONTEXT_INIT_MODE=none EG_SHARED_NAMESPACE=True EG_ENV_PROCESS_WHITELIST=ORCHEST_PIPELINE_UUID,ORCHEST_PIPELINE_PATH,ORCHEST_PROJECT_UUID,ORCHEST_USERDIR_PVC,ORCHEST_PROJECT_DIR,ORCHEST_PIPELINE_FILE,ORCHEST_HOST_GID,ORCHEST_SESSION_UUID,ORCHEST_SESSION_TYPE,ORCHEST_GPU_ENABLED_INSTANCE,ORCHEST_REGISTRY EG_PORT=8888 EG_LOG_LEVEL=10 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP=<tcp://10.97.27.138:80> JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT=<tcp://10.104.209.251:80> CONDA_ENV=base JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_PROTO=tcp EG_ALLOW_ORIGIN=* KERNEL_ID=1be178e4-fdc7-4267-b2d7-2ee01b3eda91 EG_KERNEL_WHITELIST= NB_UID=1000 DOCKER_REGISTRY_PORT_443_TCP_ADDR=10.96.145.139 KERNEL_UID=1000 KERNEL_IMAGE=orchest-env-932b2d80-7cb6-4fe4-a4c6-ca59240cec9b-378cd435-c476-45b3-b425-dfd451928a96 EG_PROHIBITED_UIDS=[] UPDATE_SIDECAR_SERVICE_HOST=10.111.210.62 DOCKER_REGISTRY_SERVICE_PORT_HTTPS_443=443 KERNEL_LAUNCH_TIMEOUT=40 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT=<tcp://10.102.197.135:8888> ORCHEST_API_PORT=<tcp://10.107.113.77:80> PWD=/project-dir EG_CULL_IDLE_TIMEOUT=43200 CONDA_PREFIX=/opt/conda JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_PROTO=tcp JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP=<tcp://10.101.71.238:80> ORCHEST_PROJECT_UUID=932b2d80-7cb6-4fe4-a4c6-ca59240cec9b ORCHEST_API_PORT_80_TCP_PORT=80 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_PROTO=tcp EG_DEFAULT_KERNEL_NAME=python_docker ORCHEST_DATABASE_SERVICE_HOST=10.96.198.198 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_PORT=80 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_HOST=10.97.27.138 KERNEL_SERVICE_ACCOUNT_NAME=default EG_RESPONSE_ADDRESS=172.17.0.18:56853 ORCHEST_DATABASE_PORT_5432_TCP=<tcp://10.96.198.198:5432> JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_PORT=8888 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_SERVICE_HOST=10.110.235.235 UPDATE_SIDECAR_PORT_80_TCP_ADDR=10.111.210.62 PLOTLY_RENDERER=iframe AUTH_SERVER_SERVICE_HOST=10.110.178.112 EG_ENABLE_TUNNELING=False ORCHEST_WEBSERVER_SERVICE_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_PORT=8888 EG_KERNEL_LAUNCH_TIMEOUT=600 HOME=/home/jovyan ORCHEST_WEBSERVER_PORT_80_TCP_ADDR=10.106.49.193 ORCHEST_API_PORT_80_TCP_PROTO=tcp ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP=<tcp://10.110.235.235:2746> JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT=<tcp://10.101.71.238:80> JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_ADDR=10.97.27.138 LANG=en_US.UTF-8 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT=<tcp://10.98.31.174:8888> KUBERNETES_PORT_443_TCP=<tcp://10.96.0.1:443> EG_LIST_KERNELS=True ORCHEST_PIPELINE_UUID=62615a7c-ce20-4121-935b-bf6d5bf5e163 EG_SSH_PORT=2122 ORCHEST_USERDIR_PVC=userdir-pvc AUTH_SERVER_PORT=<tcp://10.110.178.112:80> NB_GID=100 UPDATE_SIDECAR_PORT_80_TCP_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP=<tcp://10.98.31.174:8888> ORCHEST_VERSION=v2022.03.8 CONDA_PROMPT_MODIFIER=(base) RABBITMQ_SERVER_PORT=<tcp://10.96.164.73:5672> ORCHEST_DATABASE_PORT_5432_TCP_PORT=5432 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP=<tcp://10.102.197.135:8888> PYDEVD_USE_FRAME_EVAL=NO EG_UID_BLACKLIST=["-1"] UPDATE_SIDECAR_PORT=<tcp://10.111.210.62:80> ORCHEST_API_PORT_80_TCP=<tcp://10.107.113.77:80> JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT=<tcp://10.97.27.138:80> ORCHEST_PIPELINE_FILE=test_rechazos.orchest ORCHEST_WEBSERVER_PORT_80_TCP_PROTO=tcp ORCHEST_DATABASE_PORT_5432_TCP_ADDR=10.96.198.198 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_PROTO=tcp KG_PORT=8888 DOCKER_REGISTRY_PORT=<tcp://10.96.145.139:443> ORCHEST_API_SERVICE_PORT=80 EG_CULL_CONNECTED=False ORCHEST_API_SERVICE_HOST=10.107.113.77 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_PORT=8888 EG_PORT_RETRIES=0 RABBITMQ_SERVER_PORT_5672_TCP_PROTO=tcp EG_MIN_PORT_RANGE_SIZE=1000 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_SERVICE_PORT=2746 ORCHEST_GPU_ENABLED_INSTANCE=False JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_PROTO=tcp EG_BASE_URL=/jupyter-server-932b2d80-7cb6-4fe462615a7c-ce20-4121 AUTH_SERVER_PORT_80_TCP_PORT=80 RABBITMQ_SERVER_SERVICE_PORT=5672 KG_IP=0.0.0.0 JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_PORT=80 RABBITMQ_SERVER_PORT_5672_TCP_ADDR=10.96.164.73 JPY_PARENT_PID=8 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_HOST=10.104.209.251 ORCHEST_WEBSERVER_PORT_80_TCP=<tcp://10.106.49.193:80> JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_8888_TCP_PORT=8888 EG_CULL_INTERVAL=60 _CE_CONDA= UPDATE_SIDECAR_PORT_80_TCP_PROTO=tcp ORCHEST_WEBSERVER_SERVICE_HOST=10.106.49.193 CONDA_SHLVL=1 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP_PORT=80 ORCHEST_WEBSERVER_PORT=<tcp://10.106.49.193:80> KERNEL_POD_NAME=root-1be178e4-fdc7-4267-b2d7-2ee01b3eda91 EG_IP=0.0.0.0 SHLVL=1 ORCHEST_HOST_GID=1 DOCKER_REGISTRY_SERVICE_PORT=443 CONDA_DIR=/opt/conda KERNEL_GATEWAY=1 KUBERNETES_PORT_443_TCP_PROTO=tcp JUPYTER_PATH=/opt/conda/share/jupyter ORCHEST_WEBSERVER_PORT_80_TCP_PORT=80 KG_PORT_RETRIES=0 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_8888_TCP_ADDR=10.98.31.174 ORCHEST_PROJECT_DIR=/userdir/projects/Chile KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 ORCHEST_REGISTRY=10.96.145.139 RABBITMQ_SERVER_SERVICE_HOST=10.96.164.73 KERNEL_WORKING_DIR=/project-dir KERNEL_USERNAME=root CONDA_PYTHON_EXE=/opt/conda/bin/python JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_ADDR=10.102.197.135 ORCHEST_SESSION_UUID=932b2d80-7cb6-4fe462615a7c-ce20-4121 JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_80_TCP=<tcp://10.104.209.251:80> JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_PORT_8888_TCP_PROTO=tcp KERNEL_LANGUAGE=python CONDA_DEFAULT_ENV=base JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_PORT=8888 JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_ADDR=10.101.71.238 ORCHEST_DATABASE_SERVICE_PORT=5432 ORCHEST_DATABASE_PORT_5432_TCP_PROTO=tcp JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_PORT_80_TCP_PROTO=tcp NB_USER=jovyan KUBERNETES_SERVICE_HOST=10.96.0.1 LC_ALL=en_US.UTF-8 KUBERNETES_PORT=<tcp://10.96.0.1:443> DOCKER_REGISTRY_PORT_443_TCP=<tcp://10.96.145.139:443> KUBERNETES_PORT_443_TCP_PORT=443 ORCHEST_DATABASE_PORT=<tcp://10.96.198.198:5432> ORCHEST_PIPELINE_PATH=/pipeline.json AUTH_SERVER_PORT_80_TCP_PROTO=tcp PATH=/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin KERNEL_EXECUTOR_IMAGE=orchest-env-932b2d80-7cb6-4fe4-a4c6-ca59240cec9b-378cd435-c476-45b3-b425-dfd451928a96 AUTH_SERVER_SERVICE_PORT=80 DOCKER_REGISTRY_PORT_443_TCP_PROTO=tcp KERNEL_GID=100 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_PORT=8888 JUPYTER_EG_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_HOST=10.102.197.135 EG_UNAUTHORIZED_USERS=["dummy"] ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_PORT=2746 JUPYTER_SERVER_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT_80_TCP_PORT=80 JUPYTER_EG_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_HOST=10.98.31.174 AUTH_SERVER_PORT_80_TCP_ADDR=10.110.178.112 DOCKER_REGISTRY_PORT_443_TCP_PORT=443 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_SERVICE_HOST=10.101.109.24 ARGO_WORKFLOW_ARGO_WORKFLOWS_SERVER_PORT_2746_TCP_ADDR=10.110.235.235 DEBIAN_FRONTEND=noninteractive EG_MAX_PORT_RANGE_RETRIES=5 AUTH_SERVER_PORT_80_TCP=<tcp://10.110.178.112:80> JUPYTER_SERVER_76E7A73D_16CF_44945DCB8A70_C000_4C91_SERVICE_PORT=80 JUPYTER_EG_76E7A73D_16CF_4494DAC9A962_FC8C_42D1_PORT=<tcp://10.101.109.24:8888> KERNEL_NAMESPACE=orchest ORCHEST_SESSION_TYPE=interactive JUPYTER_SERVER_932B2D80_7CB6_4FE462615A7C_CE20_4121_SERVICE_PORT=80 _=/usr/bin/env
+ python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --RemoteProcessProxy.kernel-id 1be178e4-fdc7-4267-b2d7-2ee01b3eda91 --RemoteProcessProxy.port-range 0..0 --RemoteProcessProxy.response-address 172.17.0.18:56853 --RemoteProcessProxy.spark-context-initialization-mode none
[D 2022-04-05 17:38:42,633.633 launch_ipykernel] Using connection file '/tmp/kernel-1be178e4-fdc7-4267-b2d7-2ee01b3eda91_qyazgm4t.json'.
[I 2022-04-05 17:38:42,633.633 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 47105
[D 2022-04-05 17:38:42,634.634 launch_ipykernel] JSON Payload 'b'{"shell_port": 45391, "iopub_port": 51123, "stdin_port": 40961, "control_port": 37819, "hb_port": 54241, "ip": "0.0.0.0", "key": "5cda7bd8-019c-4608-998b-e921aca848c0", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": "23", "pgid": "7", "comm_port": 47105}'
[D 2022-04-05 17:38:42,635.635 launch_ipykernel] Encrypted Payload 'b'dQTVwTDQT3E6b/k0ohxogoTSpyd41lEPhtU0fsQfgYpYyETq+uwLONhDvafPyLoTSu46lsRDi2K3QWt/VrZvm7vS/Seb4X1pI1rEiNmA2PbOKOm0hfJpnSpaUSJvpR6L/pRivZ3NuEjMcgZBayhGX8S3o+imMjHIXSlQ9/3h4onX44pDhlvdrtsJStIio+mvklX05+90dSZAA5ZDD7AOhRjVwwP0c9iU/lfU7iOyyHQM2NcDR9Pny0ZOMKPGa3cx+wbyPcp2nuZqN3brbDLcSBkfQivZeng+vZCGUOYiJDGZPJsezfFL94Z9430ZC57YfX/fgKN/kQwumr8HYwReuiJnmrudPovmSPH8sOtuRg2LNjsajf+hdHZrN2lRGItOohf8ip2D0DS4PDvhwSLJEA=='y
Yannick
04/05/2022, 6:12 PMThank you! Seems like it got the exact same logs as the previous time (looking at the timestamps). Anyways, nothing out of the ordinary in the logs.
We also set
EG_KERNEL_LAUNCH_TIMEOUT600sDisconnected...d
Diego Vasquez
04/05/2022, 6:21 PMWe ran the installation commands on Docker in a server on Azure with Ubuntu @Yannick. I will tag @Max Moreno, our developer that made the installation
y
Yannick
04/05/2022, 6:27 PMThat all sounds good 🤔
I am currently out of ideas, but will get back to you as soon as I come up with something. For now you could try to do reinstall Orchest on a new minikube cluster. My apologies for the inconvenience.
d
Diego Vasquez
04/05/2022, 6:35 PMThanks for your reply @Yannick. Currently we are going to wait if you have other ideas, because last week we had some troubles running a new installation of Orchest.
Before the troubles, we had running in perfect conditions the version v2022.02.08. It could be possible that you give us the possibility to install that version in instead of someone newer?
j
Jacopo
04/05/2022, 6:41 PM@Diego Vasquez tomorrow morning first thing I'll try running your entire installation process on a fresh ubuntu image, could you tell me the version of ubuntu you are using?
As @Yannick mentioned starting minikube with more resources might help, the default settings are 2vcpus 8GB of ram and 20GB of disk
minikube --cpus <as much as possible --memory <as much as possible> --disk-size <as much as possible>minikube start --cpus 6 --memory 16g --disk-size 50gd
Diego Vasquez
04/05/2022, 6:51 PM@Jacopo this is the Ubuntu version:
Description: Ubuntu 20.04.4 LTS
Release: 20.04
j
Jacopo
04/05/2022, 6:52 PMAbout using an old version, on this branch https://github.com/orchest/orchest/tree/stale/pre-k8s-orchest we have the latest pre k8s version of Orchest, but it's not something we will support in the long term, and of course it's already missing some quite important features that are now part of orchest
ty, tomorrow I'll go through the entire process
d
Diego Vasquez
04/05/2022, 6:53 PMthanks @Jacopo
👍 1
j
Jacopo
04/05/2022, 6:56 PMNo problem, we are really sorry that things aren't working out of the box, we definitively aim to have Orchest be a "just works" product, we have recently moved to k8s and there are still some rough edges, but it will be worth the wait, promise 🙂
I'll keep you posted about how things go when deploying on a fresh image!
j
juanlu
04/05/2022, 6:58 PMnotice by the way that there's a bug with
minikube--memoryj
Jacopo
04/05/2022, 6:59 PMOh and one more thing. could you try to: rebuild the environment image, and rebuild jupyter server (settings -> configure jupytelab -> build), then start the session again?
d
Diego Vasquez
04/05/2022, 7:10 PMoh alright, no problem. Let me try to rebuild de jupyter server.
In other hand, we have some troubles related to log windows. In the build environment, in pipelines and in jobs, the "log panel" its always empty, that doesn't show the printed logs of the running scripts.
here is an example
👀 1
@Jacopo I tried to rebuild jupyter lab but the problem persist
👀 2
j
Jacopo
04/06/2022, 8:47 AMHi @Diego Vasquez, I was able to reproduce the issue on a fresh ubuntu instance, essentially it's due to websockets being used for logs and by jupyter. The nginx configuration wasn't supporting websockets, so some functionality was cut off. Setting
/etc/nginx/sites-available/localorchest.iomap $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name orchest;
root /var/www/example.com;
index index.html;
location / {
proxy_pass <http://localorchest.io>;
# WebSocket support.
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 86400;
}
}sudo service nginx restart🙌 1
🙌🏼 1
Side-note, I noticed that for the minikube version that was installed the default memory size is about 2 gigs, I think you want to set --cpus, --memory and --disk-size to something more suitable
d
Diego Vasquez
04/06/2022, 2:17 PMgreat, so the fix that you described before is going to fix the kernel problem or the log panel problem?
j
Jacopo
04/06/2022, 2:24 PMboth of them
They are actually the same issue pretty much
d
Diego Vasquez
04/06/2022, 2:30 PMoh ok! Thank you! We will do the corrections right now!
💪 1
j
Jacopo
04/06/2022, 2:35 PM👍 let me know how it goes!
d
Diego Vasquez
04/06/2022, 8:48 PMit's going good, now we don't have problems with kernel connection nor log panel.
🎉 2
j
Jacopo
04/07/2022, 6:26 AMhappy to hear that 👍