Hello everyone. We have a Orchest EKS deployment using version

v2022.10.5-1.0.1

. Sometimes, when trying to open Jupyter Lab, it get stuck trying to open it. Usually, this occurs when there is something wrong with the environment configuration. However, today we started to experience this in all projects that we have. By inspect one of the containers that was trying to launch the Jupyter Lab, it seems that the problem is occurring in the docker pull step as shown by the log below:

Error response from daemon: Get "<https://10.100.0.2/v2/>": dial tcp 10.100.0.2:443: connect: connection refused
Docker pull failed, pulling with buildah.
Trying to pull 10.100.0.2/orchest-env-bf496ede-cc70-4b92-b178-e4ab3929d5e9-d81de297-27a0-4864-8e85-584709a3ab7a:3...
initializing source <docker://10.100.0.2/orchest-env-bf496ede-cc70-4b92-b178-e4ab3929d5e9-d81de297-27a0-4864-8e85-584709a3ab7a:3>: pinging container registry 10.100.0.2: Get "<http://10.100.0.2/v2/>": dial tcp 10.100.0.2:80: i/o timeout

Any thoughts on this issue?

hello @Rafael Rodrigues Santana, sorry you're having a bad experience! hopefully my colleagues @Yannick, @Navid H and @Jacopo can lend you a hand

@Jacopo Does this ring a bell for you? Seems like something is going wrong with the image pulling in the

initContainer

(pull_image.sh). @Rafael Rodrigues Santana Did you change any configuration of the container runtime on your cluster (seems like you are using Docker)?

Hi @Rafael Rodrigues Santana, some questions: • is this a single or multi node deployment? • like @Yannick mentioned, did you perform any change to the container runtime configuration? For example, did you install Orchest on a docker based EKS cluster and then migrated the cluster to containerd based nodes? • was any change done to the registry service that is internally deployed by Orchest? Is the registry reachable? Can you verify that the IP

10.100.0.2

is the IP in use by the registry service created by Orchest? (

kubectl get services -n orchest | grep docker-registry

) • what AMI is being used for the EKS node? Anything custom?

The registry IP seems to be created by orchest, can you also verify that the docker registry deployment is up and running.

kubectl get deployment docker-registry -n orchest

I'll check and let you guys know! Thanks for the suggestions.