Bruno Oliveira dos Santos
05/02/2023, 1:16 PMJacopo
05/02/2023, 3:27 PMORCHEST_NAMESPACE
and ORCHEST_CLUSTER
are set correctly for the orchest-api
pod part of the namespace for which things are failing?orchest-api
• orchest-webserver
• orchest-controller
Bruno Oliveira dos Santos
05/03/2023, 10:33 AMJacopo
05/04/2023, 11:31 AMHello Jacopo. Thank you for the informations. Looking at the logs I found some inconsistencies and managed to do a reset and the mentioned problem was solved.happy to hear that!
However, when I go to build some environment I noticed that it uploads a POD called "image-build-task" and this POD is not being attached to the correct NodePool, causing an error.Take a look at
_get_image_build_manifest
in services/orchest-api/app/app/core/image_utils.py
, you'll see the pod_scheduling.modify_image_builder_pod_scheduling_behaviour(manifest)
call at the end of the function, the services/orchest-api/app/app/core/pod_scheduling.py
module is in charge of doing changes to manifests so that scheduling happens in a way we like, in this particular case, it'll look for the WORKER_PLANE_SELECTOR
env variable of the orchest-api and celery-worker
to adjust the behaviour for multi tenancy you'll likely need to do some minor changes around these variables