Title
#announcements
Alexsander Pereira

Alexsander Pereira

09/06/2022, 3:44 PM
Good afternoon, I tried to make a backup of the EKS cluster using Velero (https://velero.io/). It turns out that when restoring the backup all containers went up normally, except for Orchest. Could you help me understand? I'm trying to implement a backup routine here at the Dadosfera company. Orchest Controller has failed:
Rick Lamers

Rick Lamers

09/06/2022, 9:00 PM
Thanks for reporting, we’ll investigate what could be the issue.
Alexsander Pereira

Alexsander Pereira

09/08/2022, 11:52 AM
Thank you, I'll wait...
Rick Lamers

Rick Lamers

09/08/2022, 2:07 PM
We're still investigating this issue, we expect to have an update tomorrow (Friday).
Yannick

Yannick

09/09/2022, 11:01 AM
Thank you for the error logs @Alexsander Pereira 🙏 @Navid H Could you take a look at https://github.com/orchest/orchest/blob/master/services/orchest-controller/pkg/controller/orchestcluster/cluster_controller.go#L783? Thank you!
Navid H

Navid H

09/12/2022, 12:09 PM
@Alexsander Pereira: I'll look into this and will comeback as soon as possible.
Yannick

Yannick

09/13/2022, 7:28 AM
@Alexsander Pereira FYI a fix will be included in the next release, thanks to @Navid H 🤩
11:59 AM
Released 🤓
Alexsander Pereira

Alexsander Pereira

09/13/2022, 5:43 PM
@Yannick That's great news, I'll do the tests!
5:52 PM
I upgraded to the new version and this is what happened... The orchest webserver and orchest api did not go up... Can you help me?
Rick Lamers

Rick Lamers

09/14/2022, 6:05 PM
We'll check out the logs and get back to you tomorrow @Alexsander Pereira At first glance it looks like it's finding that resources already exist in the cluster that the controller doesn't expect to find. Could you describe the actions you took to get to the error log you posted?
Yannick

Yannick

09/15/2022, 9:06 AM
@Navid H With the addition of deploying Nginx as part of Orchest, do you think this could have caused any of the issues? One of the logs that made me think this is the case:
E0914 17:51:53.278362       1 controller.go:172] failed to deploy helm deployment Error: rendered manifests contain a resource that already exists. Unable to continue with install: IngressClass "nginx" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>": must be set to "Helm"; annotation validation error: missing key "<http://meta.helm.sh/release-name|meta.helm.sh/release-name>": must be set to "orchest-ingress-nginx"; annotation validation error: missing key "<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>": must be set to "orchest"
I upgraded to the new version and this is what happened...
@Alexsander Pereira Did you try Velero again or did this happen just by doing a regular
orchest update
?
Navid H

Navid H

09/15/2022, 9:10 AM
Hi @Alexsander Pereira, Could you run the following command and return the result back to us, Thanks.
kubectl get ingressclass
Alexsander Pereira

Alexsander Pereira

09/15/2022, 12:39 PM
alex@alex-Legion-5-Pro-16ITH6:~/Documents/code/dadosfera$ kubectl get ingressclass
NAME    CONTROLLER             PARAMETERS   AGE
nginx   <http://k8s.io/ingress-nginx|k8s.io/ingress-nginx>   <none>       39h
12:41 PM
@Yannick @Navid H I install Nginx just like it worked before. Was that why? Our nginx is customized with ACM certificate and tags on the load balancer, I need to keep having control over it. Any way to disable nginx installation in orchest-controller?
12:42 PM
Or would it be possible to customize this in the orchest controller and not install nginx separately anymore?
Navid H

Navid H

09/15/2022, 12:46 PM
@Alexsander Pereira: Thanks for getting back to us, This is a corner case that we will fix in the next release, but in the mean time, you can add the following annotation to the OrchestCluster to disable Nginx.
<http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
After that you should also remove the nginx application from your
OrchestCluster
CRD
1:29 PM
@Alexsander Pereira: Also one more question, are you using eks, or gke cluster?
Yannick

Yannick

09/15/2022, 1:45 PM
I tried to make a backup of the EKS cluster using Velero
☝️ So I guess EKS 😉
Alexsander Pereira

Alexsander Pereira

09/16/2022, 1:47 PM
It worked! Very good! @Yannick @Navid H Thanks!
1:47 PM
@Navid H We use EKS
Navid H

Navid H

09/16/2022, 2:38 PM
@Alexsander Pereira: Thanks for letting us know 🙏