Title
#tech-support
Rafael Rodrigues Santana

Rafael Rodrigues Santana

10/31/2022, 5:21 PM
Hello everyone. We have a Orchest EKS deployment using version
v2022.10.5-1.0.1
. Sometimes, when trying to open Jupyter Lab, it get stuck trying to open it. Usually, this occurs when there is something wrong with the environment configuration. However, today we started to experience this in all projects that we have. By inspect one of the containers that was trying to launch the Jupyter Lab, it seems that the problem is occurring in the docker pull step as shown by the log below:
Error response from daemon: Get "<https://10.100.0.2/v2/>": dial tcp 10.100.0.2:443: connect: connection refused
Docker pull failed, pulling with buildah.
Trying to pull 10.100.0.2/orchest-env-bf496ede-cc70-4b92-b178-e4ab3929d5e9-d81de297-27a0-4864-8e85-584709a3ab7a:3...
initializing source <docker://10.100.0.2/orchest-env-bf496ede-cc70-4b92-b178-e4ab3929d5e9-d81de297-27a0-4864-8e85-584709a3ab7a:3>: pinging container registry 10.100.0.2: Get "<http://10.100.0.2/v2/>": dial tcp 10.100.0.2:80: i/o timeout
Any thoughts on this issue?
juanlu

juanlu

10/31/2022, 9:05 PM
hello @Rafael Rodrigues Santana, sorry you're having a bad experience! hopefully my colleagues @Yannick, @Navid H and @Jacopo can lend you a hand
Yannick

Yannick

11/01/2022, 8:14 AM
@Jacopo Does this ring a bell for you? Seems like something is going wrong with the image pulling in the
initContainer
(pull_image.sh). @Rafael Rodrigues Santana Did you change any configuration of the container runtime on your cluster (seems like you are using Docker)?
Jacopo

Jacopo

11/01/2022, 8:39 AM
Hi @Rafael Rodrigues Santana, some questions: ā€¢ is this a single or multi node deployment? ā€¢ like @Yannick mentioned, did you perform any change to the container runtime configuration? For example, did you install Orchest on a docker based EKS cluster and then migrated the cluster to containerd based nodes? ā€¢ was any change done to the registry service that is internally deployed by Orchest? Is the registry reachable? Can you verify that the IP
10.100.0.2
is the IP in use by the registry service created by Orchest? (
kubectl get services -n orchest | grep docker-registry
) ā€¢ what AMI is being used for the EKS node? Anything custom?
Navid H

Navid H

11/01/2022, 9:13 AM
The registry IP seems to be created by orchest, can you also verify that the docker registry deployment is up and running.
kubectl get deployment docker-registry -n orchest
Rafael Rodrigues Santana

Rafael Rodrigues Santana

11/01/2022, 6:03 PM
I'll check and let you guys know! Thanks for the suggestions.