Title
#announcements
Alexsander Pereira

Alexsander Pereira

10/05/2022, 3:01 PM
I updated to the latest version using EKS and orchest is showing this error in the controller:
Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "dbc00419165f73b0ba32733416e822c3c5c42a30787782f78475670ac50171fe" network for pod "orchest-controller-85df7b958b-9kn4c": networkPlugin cni failed to set up pod "orchest-controller-85df7b958b-9kn4c_orchest" network: add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused", failed to clean up sandbox container "dbc00419165f73b0ba32733416e822c3c5c42a30787782f78475670ac50171fe" network for pod "orchest-controller-85df7b958b-9kn4c": networkPlugin cni failed to teardown pod "orchest-controller-85df7b958b-9kn4c_orchest" network: del cmd: error received from DelNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"]
Can someone help me? It's a production cluster.
Yannick

Yannick

10/05/2022, 5:02 PM
@Navid H I haven't seen this error before, any thoughts? @Alexsander Pereira How did you go about updating Orchest to the latest version? There are some caveats involved in case you didn't use the
orchest-cli
(link) command:
orchest update
Navid H

Navid H

10/05/2022, 5:05 PM
It seems this might be related to performance improvement, I'll take a look and also tag @Jacopo here to be in the loop.
Alexsander Pereira

Alexsander Pereira

10/05/2022, 5:12 PM
It was a problem with the EKS aws-node, I managed to solve it!
Navid H

Navid H

10/05/2022, 5:13 PM
great
Jacopo

Jacopo

10/05/2022, 5:13 PM
glad to hear it šŸ˜ƒ)
Alexsander Pereira

Alexsander Pereira

10/05/2022, 5:13 PM
I tried to update the cluster by passing a custom image using the new version of orchest-controller. And he's not uploading the new image... any ideas?
5:13 PM
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.0-1.0.1
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.0-1.0.1
5:14 PM
I had to edit the pod manifest manually to update it.
Navid H

Navid H

10/05/2022, 5:19 PM
I think you should updaye the version instead of the image of each compoenent.
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
    version: your-version
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.0-1.0.1
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.0-1.0.1
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.0-1.0.1
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.0-1.0.1
Alexsander Pereira

Alexsander Pereira

10/05/2022, 5:21 PM
But I update the version already in the orchest-controller yml file.
5:22 PM
Before I updated the controller, then I updated the cluster passing the custom images, and the controller would upload the new versions of the webserver and auth-server by itself.
Navid H

Navid H

10/05/2022, 5:23 PM
have you updated the CRD?
Alexsander Pereira

Alexsander Pereira

10/05/2022, 5:23 PM
What would the CRD be? Sorry
5:24 PM
I just apply these two yaml, orchest-controller.yml and orchest-cluster.yml
5:24 PM
Before it worked
Navid H

Navid H

10/05/2022, 5:25 PM
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
    version: your-version
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.0-1.0.1
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.0-1.0.1
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.0-1.0.1
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.0-1.0.1
5:25 PM
specifically this part:
spec:
  singleNode: true
  orchest:
    version: your-version
Alexsander Pereira

Alexsander Pereira

10/05/2022, 5:26 PM
Okay, i will test
5:43 PM
It didn't work, it doesn't update the custom images.
5:44 PM
6:40 PM
@Navid H Any idea what it is?
Navid H

Navid H

10/06/2022, 7:00 AM
Let me check it out and I will comeback to you shortly
Yannick

Yannick

10/06/2022, 7:52 AM
The steps that we take to update the OrchestCluster through the
orchest-cli
are as follows (link to source code): ā€¢ Update the
orchest-controller
manifests (you can pull these from our release notes) ā€¢ Patch the
spec.orchest.version
value in the
OrchestCluster
custom object to point to the new version.
It didn't work, it doesn't update the custom images.
Unless my memory fails me, I think we settled on the fact that if a user specifies custom images then they need to manually update those custom images. The
orchest-controller
does not do that for you since we can't assume what users want to happen (everyone can have completely custom requirements here, so it is the responsibility of the user). Hope that clears some things up šŸ˜‡
Alexsander Pereira

Alexsander Pereira

10/06/2022, 1:38 PM
I understand, it's ok