https://www.orchest.io/ logo
a

Allan Sene

07/21/2022, 6:09 PM
Hi, guys! How you doing? I'm testing the new k8s deployment here, and I'm getting this error, when I try to start a kernel: ``HTTP 599: Unknown (Error attempting to connect to Gateway server url 'http://jupyter-eg-41d7c6a2-e579-467ec0ee3f1a-723f-4294:8888/jupyter-server-41d7c6a2-e579-467ec0ee3f1a-723f-4294'. Ensure gateway url is valid and the Gateway instance is running.)`` Do you have some tips about how to deal with it?
👀 2
j

juanlu

07/21/2022, 6:11 PM
hi @Allan Sene, sorry you're experiencing trouble. could you tell us what Orchest version are you using, and what kind of k8s implementation you have (minikube, AKS, other)?
a

Allan Sene

07/21/2022, 7:16 PM
v2022.07.5
deployed on Amazon EKS [edited]
it try to initialize, but fail without warning
And it does not log anything for this pipeline
seems to be all fine with the service on k8s. It is up and running, with 8888 port opened
j

Jacopo

07/22/2022, 5:44 AM
Might be an ingress problem, @Navid H dealt with something related to this quite recently, so he might be able to pitch in
👍 1
n

Navid H

07/22/2022, 8:26 AM
@Allan Sene: Could you list all the ingress resources created in your cluster (the one in orchest namespace should be enough) and give their statuses?
r

Rick Lamers

07/22/2022, 9:35 AM
It might also be informative to check the logs on the jupyter enterprise gateway pod/container. Have you tried restarting the pipeline session, do you always get the same error?
👍 1
a

Allan Sene

07/22/2022, 1:52 PM
Yes, @Jacopo. I just noticed that
jupyter-eg
Ingress is not spinning up
👀 1
@Rick Lamers restarting pipeline session, rebuilding our image, and so, does not fix it
image.png
n

Navid H

07/22/2022, 2:27 PM
@Allan Sene: it seems the Message part of the events you shared is truncated, could you share the whole Message part of that event of
jupyter-eg
?
a

Allan Sene

07/22/2022, 2:30 PM
Here, we noticed that the ingress is not up
@Navid H even using the
kubectl
, it is giving just this log 😞
n

Navid H

07/22/2022, 2:37 PM
Can you run the
kubectl
command with
-o json
option
a

Allan Sene

07/22/2022, 2:58 PM
same, David. I'm going to send you the full log
orchest.log
line 1664
👀 1
🙌 1
Every time that we try to run the kernel, Kubernetes try to spin up a
kernel-root
pod. But it fails then dies
n

Navid H

07/22/2022, 3:10 PM
Thanks for sharing these, it seems Jupyter server keeps failing, because
jupyter-eg
can't get an endpoint. let me try to see if I can reproduce it or not, will comeback to you.
👍 1
y

Yannick

07/22/2022, 3:16 PM
To shoot more ideas. Could it be that you used a custom base image when building the Environment (the one the Step is using for which the kernel fails to start)? Moreover, did you explicitly install
ipykernel
?
👀 1
Thank you for providing all the information so far by the way 💪
a

Allan Sene

07/22/2022, 6:08 PM
Using the base python image, it happens the same way, @Yannick, 😞
👀 1
using the quickstart project, with the base image, without anything extra
n

Navid H

07/22/2022, 6:44 PM
@Allan Sene: Thank you very much for providing all the information, it was super valuable to us, I was able to reproduce your issue, working on a fix.
a

Allan Sene

07/22/2022, 7:34 PM
Thank you, guys! I appreciate the effort. We are very eager to start using more of Orchest here at Dadosfera.
🙏🏼 1
🙏 3
y

Yannick

07/25/2022, 8:55 AM
@Allan Sene The fix is currently in the works (thanks @Navid H) and can be found here: https://github.com/orchest/orchest/pull/1150
a

Allan Sene

07/25/2022, 12:08 PM
Thanks, Yannick!
2 Views