hi everybody we ve been using orchest a few months but we up

Question

hi everybody we ve been using orchest a few months but we updated and now we are struggling with the minikubes disk space 20 gb how can we configure a bigger disk the server has 200gb of disk

juanlu · Answer

hola < José Tomás Pérez> pinging < Jacopo> in case he can give some ideas

Rick Lamers · Answer

Hi José thanks for letting us know Are you referring to Orchest Cloud or have you deployed it yourself

José Tomás Pérez · Answer

deploy by ourself =

Jacopo · Answer

hey there < José Tomás Pérez> slightly smiling face give me a couple of minutes to think about this

José Tomás Pérez · Answer

thanks < Jacopo>

Jacopo · Answer

< José Tomás Pérez> the first thing that comes to my mind would be resizing the pvc directly in k8s The ability to do that might depend on minikube s version and other specifics but it s worth a try I guess you need the ` userdir` to increase in size right

José Tomás Pérez · Answer

yes that s right

José Tomás Pérez · Answer

let me try i have an older version

Jacopo · Answer

Do you already know how to resize it If not I ll look into the specific command you would need to issue I am deleting my local cluster to see if we can achieve a different size by simply changing a couple of values in the repo Not super clean we will likely make a release tomorrow to expand the CLI scope

Jacopo · Answer

If you are doing an install from scratch you will be able to set the userdir and builder cache size by changing the values in `deploy helm charts cluster resources charts orchest values yaml` `deploy helm charts cluster resources charts nfs values yaml`

Jacopo · Answer

then running `bash orchest install`

Jacopo · Answer

with clean install I mean either deleting the minikube cluster or deleting the `orchest` namespace

José Tomás Pérez · Answer

great help we are newbies with kubernetes

José Tomás Pérez · Answer

we ll try these commands

Jacopo · Answer

About resizing existing pvcs couple of handy commands that will allow you to check if you can resize them on the fly `kubectl edit pvc n orchest userdir pvc` `kubectl edit pvc n orchest image builder cache pvc` will open up the pvc yaml and will allow you to try and edit the pvc My guess is that it won t allow you to when using a default minikube installation so reinstalling orchest is your best bet right now

Jacopo · Answer

If you don t have `kubectl` installed this will do it `minikube kubectl edit pvc n orchest image builder cache pvc`

José Tomás Pérez · Answer

thx a lot

José Tomás Pérez · Answer

now my team is trying out

Jacopo · Answer

No problem slightly smiling face If you changed the pvc size by changing charts values you can use those same commands to confirm that it worked

José Tomás Pérez · Answer

< Jacopo> We ve tried to install from scratch and changing the ` yaml` but still the disk space in orchest is 20gb Is there a step we are missing

Navid H · Answer

< José Tomás Pérez> could you double check if you changed these values if yes could you get the pvc configs with with following command and return the response `kubectl get pvc n orchest o yaml`

Jacopo · Answer

Sorry to hear that I second what < Navid H> recommended

José Tomás Pérez · Answer

< Jacopo> < Navid H>

José Tomás Pérez · Answer

image 43 png image 42 png

Jacopo · Answer

Hey < José Tomás Pérez> thanks for coming back with this How are you verifying that the disk space is still 20 gbs

José Tomás Pérez · Answer

here in settings of orchest

Jacopo · Answer

I see that looks strange It could be that the value that is being read is wrong actually I am reinstalling with a different pvc size will get back to you

José Tomás Pérez · Answer

thanks

Jacopo · Answer

I wonder if what we are looking at is actually minikube allocated disk size Coincidentally it s 20 gigs Perhaps a different version os distribution or driver is making it act differently than what we have observed Mind stopping and then starting minikube with an additional ` disk size` flag

Jacopo · Answer

From minikube ` help` ``` disk size= 20000mb Disk size allocated to the minikube VM format lt number gt lt unit gt where unit = b k m or g ```

José Tomás Pérez · Answer

we did that < Jacopo> but with no results

José Tomás Pérez · Answer

< Max Moreno> is doing the install

Jacopo · Answer

Bit of a long shot but could it be that you need to start minikube with that size then install Orchest About the disk size that you are seeing in the settings page I think we can assume it s bugged i e don t trust it To actually verify the available disk space you could try to get into the ` userdir` of the `orchest webserver` `kubectl exec n orchest it deployment orchest webserver bash`

Jacopo · Answer

I think that getting in there and actually trying to write 20 gigs of data in ` userdir data lt whatever file gt ` will actually tell you if the pvc changes have worked

Max Moreno · Answer

I m going to try a clean install by increasing the disk space of minikube

Jacopo · Answer

Hey < Max Moreno> before doing that it might be worth `exec`ing into the webserver likely to takes less time to verify

Max Moreno · Answer

< Jacopo> I already deleted everything and I m creating the instance

Max Moreno · Answer

< Jacopo> I don t know if it s normal but it takes a long time at that installation percentage

Jacopo · Answer

It can definitively take some time due to the number of images that are being downloaded There is an issue into which we are looking where the installation is basically stuck To fix that you will need to delete the `orchest` namespace then run the installation again Most likely related to download timeouts fixing soon

Rick Lamers · Answer

Thanks for bearing with us < Max Moreno> and < José Tomás Pérez> We just released the k8s version and are streamlining the installation process

Max Moreno · Answer

< Rick Lamers> thank you for the help provided

José Tomás Pérez · Answer

raised hands

Max Moreno · Answer

< Jacopo> I ve been trying to stop the service for 1 hour any recommendations

Navid H · Answer

< Max Moreno> Thanks for coming back to us could you get the status of pods in the orchest namespace by following command and return the response ```kubectl get pods n orchest o json```

Max Moreno · Answer

finally install version 2022 03 08 it worked without problems

Jacopo · Answer

Happy to hear it < Max Moreno> are you still using kvm2 as the driver

Jacopo · Answer

We will keep looking into this issue internally

Max Moreno · Answer

this time install minikube with docker

Jacopo · Answer

I see makes sense I guess we need to cover more ground when it comes to other minikube drivers Sorry that it took so long for this to work hope you have a good time using Orchest

Max Moreno · Answer

< Jacopo> thanks for the help so far everything is working perfectly