Title
#tech-support
Alexsander Pereira

Alexsander Pereira

10/13/2022, 6:46 PM
Could someone tell me why orchest-controller can't start all pods? The controller.orchest.io/deploy-ingress: "false" option is set why my nginx is configured separately.
6:56 PM
If I apply with the option to allow orchest-controller to create nginx it works.
7:12 PM
apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
  annotations:
    <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
spec:
  singleNode: true
  orchest:
    authServer:
      image: public.ecr.aws/u5k1d2l0/orchest-auth-server:v2022.10.5-1.0.0
    orchestWebServer:
      image: public.ecr.aws/u5k1d2l0/orchest-webserver:v2022.10.5-1.0.0
Yannick

Yannick

10/13/2022, 7:20 PM
A bit more info is added here: https://docs.orchest.io/en/stable/getting_started/installation.html#install-nginx. Basically the orchest-controller will, based on your k8s distro (e.g. EKS), decide whether or not to automatically deploy the Nginx Ingress controller unless you add the annotation to not deploy Nginx Ingress at all. If your cluster already has ingress configured, then there would be an error if the orchest-controller tries to set it up again. So I guess setting the annotation to β€œfalse” works for you?
Alexsander Pereira

Alexsander Pereira

10/13/2022, 7:32 PM
@Yannick Would that option?
notes:
      <http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
It is already defined and the error still occurs.
7:38 PM
I'm testing again by creating an empty cluster.
Yannick

Yannick

10/14/2022, 4:59 AM
What cluster did you create? Because based on the cluster we automatically set the annotation. Still using EKS? @Navid H Something you can quickly look at?
Navid H

Navid H

10/14/2022, 7:16 AM
After you created the cluster, controller changes the
OrchestCluster
object, can you also post that one?
Yannick

Yannick

10/14/2022, 8:00 AM
If I apply with the option to allow orchest-controller to create nginx it works.
So you have two options on EKS:1. Don't set the
<http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
annotation and let Orchest set up ingress for you. Which is easier than having to install nginx yourself (but of course allows less configuration from your side) 2. Install nginx-ingress-controller on the cluster yourself and then install Orchest with the
<http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
annotation
Alexsander Pereira

Alexsander Pereira

10/14/2022, 1:00 PM
I didn't understand. Did you see the configuration files? I already add the annotation:
<http://controller.orchest.io/deploy-ingress|controller.orchest.io/deploy-ingress>: "false"
1:02 PM
@Navid H @Yannick I install Nginx and create the cluster with this annotation, but even so, the problem indicated in the logs happens.
Yannick

Yannick

10/14/2022, 1:04 PM
@Navid H Could you try to reproduce this on EKS? So first install Nginx and then installing Orchest with the annotation? πŸ™
Alexsander Pereira

Alexsander Pereira

10/14/2022, 1:08 PM
Okay
1:24 PM
I'm changing the installation order: Old: orchest -> nginx New: nginx -> orchest
Navid H

Navid H

10/14/2022, 1:31 PM
@Alexsander Pereira: I'll try to reproduce. Will comeback here.
Alexsander Pereira

Alexsander Pereira

10/14/2022, 2:01 PM
Problem now occurs on installation:
namespace/monitoring created
daemonset.apps/node-exporter created
service/node-exporter created
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
<http://role.rbac.authorization.k8s.io/ingress-nginx|role.rbac.authorization.k8s.io/ingress-nginx> created
<http://role.rbac.authorization.k8s.io/ingress-nginx-admission|role.rbac.authorization.k8s.io/ingress-nginx-admission> created
<http://clusterrole.rbac.authorization.k8s.io/ingress-nginx|clusterrole.rbac.authorization.k8s.io/ingress-nginx> created
<http://clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission|clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission> created
<http://rolebinding.rbac.authorization.k8s.io/ingress-nginx|rolebinding.rbac.authorization.k8s.io/ingress-nginx> created
<http://rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission|rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission> created
<http://clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx|clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx> created
<http://clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission|clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission> created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
<http://ingressclass.networking.k8s.io/nginx|ingressclass.networking.k8s.io/nginx> created
<http://validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission|validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission> created
namespace/orchest created
serviceaccount/orchest-controller created
<http://customresourcedefinition.apiextensions.k8s.io/orchestclusters.orchest.io|customresourcedefinition.apiextensions.k8s.io/orchestclusters.orchest.io> created
<http://customresourcedefinition.apiextensions.k8s.io/orchestcomponents.orchest.io|customresourcedefinition.apiextensions.k8s.io/orchestcomponents.orchest.io> created
<http://clusterrole.rbac.authorization.k8s.io/orchest-controller|clusterrole.rbac.authorization.k8s.io/orchest-controller> created
<http://clusterrolebinding.rbac.authorization.k8s.io/orchest-controller|clusterrolebinding.rbac.authorization.k8s.io/orchest-controller> created
service/orchest-controller created
deployment.apps/orchest-controller created
Error from server (InternalError): error when creating "files/orchest-controller.yml": Internal error occurred: failed calling webhook "<http://validate.nginx.ingress.kubernetes.io|validate.nginx.ingress.kubernetes.io>": failed to call webhook: Post "<https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s>": no endpoints available for service "ingress-nginx-controller-admission"
Navid H

Navid H

10/14/2022, 2:06 PM
I'm looking into it.
Alexsander Pereira

Alexsander Pereira

10/17/2022, 1:11 PM
@Navid H Any news? I need to create an Orchest cluster in production.
Navid H

Navid H

10/17/2022, 1:22 PM
Unfortunately I was not able to reproduce it in EKS or Minikube cluster, maybe we can jump on a call to debug the issue together.
Alexsander Pereira

Alexsander Pereira

10/17/2022, 1:25 PM
Have you tried deployer using our yaml?
Navid H

Navid H

10/17/2022, 1:28 PM
I tried your
OrchestCluster
yaml file, it get stuck in pulling
Auth-Server
, because I don't have access to your custom image, but it does not try to deploy ingress, could you run the following command:
kubectl get ingressclass
Alexsander Pereira

Alexsander Pereira

10/17/2022, 1:33 PM
I'll send you the step by step, without the custom images.
1:36 PM
This is the Jinja2 template from nginx. In it I just modify the load balancer tags and add the ACM certificate.
1:37 PM
Steps: 1 - kubectl apply -f files/nginx.yml 2 - kubectl apply -f files/orchest-controller.yml The error happens even before deploying the yaml cluster.
Navid H

Navid H

10/17/2022, 1:41 PM
So it happens before deploying the
OrchestCluster
?
1:48 PM
the ingress and other third-party components can be enabled at controller deployment level, if you see the error before creating the
OrchestCluster
CRD, then ingress should have been enabled at controller level deployment, but I don't see this in you deployment.
Alexsander Pereira

Alexsander Pereira

10/17/2022, 1:49 PM
I will send you the outputs of each deployment.
Navid H

Navid H

10/18/2022, 9:36 AM
After investigating it for a while I think this issue is not related to orchest, it seems your cluster is deployed in a way that the nginx admission webhook is not reachable (maybe some calico configuration for port 443, I don't know), to validate that the problem is related to webhook, you can remove the admission webhook and try again (but I believe this is not a good solution, it is better to invest time in figuring out why admission webhook is not reachable) you can remove the admission webhook by following command:
kubectl delete -A validatingwebhookconfiguration ingress-nginx-admission
Alexsander Pereira

Alexsander Pereira

10/18/2022, 12:47 PM
But how do I solve it then? We at Dadosfera are blocked from using Orchest and we don't know how to solve this problem.
Navid H

Navid H

10/18/2022, 12:48 PM
the easy solution is to run that command.
Alexsander Pereira

Alexsander Pereira

10/18/2022, 12:49 PM
After installing nginx? And before orchest?
Navid H

Navid H

10/18/2022, 12:49 PM
yes