https://www.orchest.io/ logo
Powered by
Title
a

Alexsander Pereira

10/05/2022, 3:01 PM
I updated to the latest version using EKS and orchest is showing this error in the controller: 
Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "dbc00419165f73b0ba32733416e822c3c5c42a30787782f78475670ac50171fe" network for pod "orchest-controller-85df7b958b-9kn4c": networkPlugin cni failed to set up pod "orchest-controller-85df7b958b-9kn4c_orchest" network: add cmd: Error received from AddNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused", failed to clean up sandbox container "dbc00419165f73b0ba32733416e822c3c5c42a30787782f78475670ac50171fe" network for pod "orchest-controller-85df7b958b-9kn4c": networkPlugin cni failed to teardown pod "orchest-controller-85df7b958b-9kn4c_orchest" network: del cmd: error received from DelNetwork gRPC call: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"]
Can someone help me? It's a production cluster.
šŸ‘€ 2
y

Yannick

10/05/2022, 5:02 PM
@Navid H I haven't seen this error before, any thoughts? @Alexsander Pereira How did you go about updating Orchest to the latest version? There are some caveats involved in case you didn't use the 
orchest-cli
(link) command: 
orchest update
n

Navid H

10/05/2022, 5:05 PM
It seems this might be related to performance improvement, I'll take a look and also tag @Jacopo here to be in the loop.
a

Alexsander Pereira

10/05/2022, 5:12 PM
It was a problem with the EKS aws-node, I managed to solve it!
šŸ™Œ 2
n

Navid H

10/05/2022, 5:13 PM
great
j

Jacopo

10/05/2022, 5:13 PM
glad to hear it :))
a

Alexsander Pereira

10/05/2022, 5:13 PM
I tried to update the cluster by passing a custom image using the new version of orchest-controller. And he's not uploading the new image... any ideas?
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.0-1.0.1
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.0-1.0.1
I had to edit the pod manifest manually to update it.
n

Navid H

10/05/2022, 5:19 PM
I think you should updaye the version instead of the image of each compoenent. 
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
    version: your-version
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.0-1.0.1
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.0-1.0.1
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.0-1.0.1
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.0-1.0.1
a

Alexsander Pereira

10/05/2022, 5:21 PM
But I update the version already in the orchest-controller yml file.
orchest-controller.yml
orchest-controller.yml
Before I updated the controller, then I updated the cluster passing the custom images, and the controller would upload the new versions of the webserver and auth-server by itself.
n

Navid H

10/05/2022, 5:23 PM
have you updated the CRD?
a

Alexsander Pereira

10/05/2022, 5:23 PM
What would the CRD be? Sorry
I just apply these two yaml, orchest-controller.yml and orchest-cluster.yml
Before it worked
n

Navid H

10/05/2022, 5:25 PM
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
    version: your-version
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.0-1.0.1
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.0-1.0.1
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.0-1.0.1
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.0-1.0.1
specifically this part: 
spec:
  singleNode: true
  orchest:
    version: your-version
a

Alexsander Pereira

10/05/2022, 5:26 PM
Okay, i will test
It didn't work, it doesn't update the custom images.
orchest-cluster.yml
orchest-cluster.yml
image.png
@Navid H Any idea what it is?
šŸ‘€ 1
n

Navid H

10/06/2022, 7:00 AM
Let me check it out and I will comeback to you shortly
y

Yannick

10/06/2022, 7:52 AM
The steps that we take to update the OrchestCluster through the 
orchest-cli
are as follows (link to source code): ā€¢ Update the 
orchest-controller
manifests (you can pull these from our release notes) ā€¢ Patch the 
spec.orchest.version
value in the 
OrchestCluster
custom object to point to the new version.
It didn't work, it doesn't update the custom images.
Unless my memory fails me, I think we settled on the fact that if a user specifies custom images then they need to manually update those custom images. The 
orchest-controller
does not do that for you since we can't assume what users want to happen (everyone can have completely custom requirements here, so it is the responsibility of the user). Hope that clears some things up šŸ˜‡
a

Alexsander Pereira

10/06/2022, 1:38 PM
I understand, it's ok
šŸ‘ 1
#announcementsJoin Slack
Powered by