Hi guys How you doing I m testing the new k8s deployment her

Question

Hi guys How you doing I m testing the new k8s deployment here and I m getting this error when I try to start a kernel ``HTTP 599 Unknown Error attempting to connect to Gateway server url <http jupyter eg 41d7c6a2 e579 467ec0ee3f1a 723f 4294 8888 jupyter server 41d7c6a2 e579 467ec0ee3f1a 723f 4294> Ensure gateway url is valid and the Gateway instance is running `` Do you have some tips about how to deal with it

juanlu · Answer

hi < Allan Sene> sorry you re experiencing trouble could you tell us what Orchest version are you using and what kind of k8s implementation you have minikube AKS other

Allan Sene · Answer

`v2022 07 5` deployed on Amazon EKS edited

Allan Sene · Answer

it try to initialize but fail without warning

Allan Sene · Answer

And it does not log anything for this pipeline

Allan Sene · Answer

seems to be all fine with the service on k8s It is up and running with 8888 port opened

Jacopo · Answer

Might be an ingress problem < Navid H> dealt with something related to this quite recently so he might be able to pitch in

Navid H · Answer

< Allan Sene> Could you list all the ingress resources created in your cluster the one in orchest namespace should be enough and give their statuses

Rick Lamers · Answer

It might also be informative to check the logs on the jupyter enterprise gateway pod container Have you tried restarting the pipeline session do you always get the same error

Allan Sene · Answer

Yes < Jacopo> I just noticed that `jupyter eg` Ingress is not spinning up

Allan Sene · Answer

< Rick Lamers> restarting pipeline session rebuilding our image and so does not fix it

Allan Sene · Answer

image png

Navid H · Answer

< Allan Sene> it seems the Message part of the events you shared is truncated could you share the whole Message part of that event of `jupyter eg`

Allan Sene · Answer

Here we noticed that the ingress is not up

Allan Sene · Answer

< Navid H> even using the `kubectl` it is giving just this log disappointed

Navid H · Answer

Can you run the `kubectl` command with ` o json` option

Allan Sene · Answer

same David I m going to send you the full log

Allan Sene · Answer

orchest log

Allan Sene · Answer

line 1664

Allan Sene · Answer

Every time that we try to run the kernel Kubernetes try to spin up a `kernel root` pod But it fails then dies

Navid H · Answer

Thanks for sharing these it seems Jupyter server keeps failing because `jupyter eg` can t get an endpoint let me try to see if I can reproduce it or not will comeback to you

Yannick · Answer

To shoot more ideas Could it be that you used a custom base image when building the Environment the one the Step is using for which the kernel fails to start Moreover did you explicitly install `ipykernel`

Yannick · Answer

Thank you for providing all the information so far by the way muscle

Allan Sene · Answer

Using the base python image it happens the same way < Yannick> disappointed

Allan Sene · Answer

using the quickstart project with the base image without anything extra

Navid H · Answer

< Allan Sene> Thank you very much for providing all the information it was super valuable to us I was able to reproduce your issue working on a fix

Allan Sene · Answer

Thank you guys I appreciate the effort We are very eager to start using more of Orchest here at Dadosfera

Yannick · Answer

< Allan Sene> The fix is currently in the works thanks < Navid H> and can be found here

Allan Sene · Answer

Thanks Yannick