https://www.orchest.io/ logo
#announcements
Title
# announcements
j

José Tomás Pérez

03/29/2022, 3:54 PM
hi everybody! we've been using orchest a few months, but we updated and now we are struggling with the minikubes disk space (20 gb) , how can we configure a bigger disk, the server has 200gb of disk
j

juanlu

03/29/2022, 3:55 PM
hola @José Tomás Pérez! pinging @Jacopo in case he can give some ideas
r

Rick Lamers

03/29/2022, 3:56 PM
Hi José, thanks for letting us know. Are you referring to Orchest Cloud or have you deployed it yourself?
j

José Tomás Pérez

03/29/2022, 3:56 PM
deploy by ourself =(
j

Jacopo

03/29/2022, 3:57 PM
hey there @José Tomás Pérez 🙂 , give me a couple of minutes to think about this
j

José Tomás Pérez

03/29/2022, 3:57 PM
thanks @Jacopo!
j

Jacopo

03/29/2022, 4:01 PM
@José Tomás Pérez the first thing that comes to my mind would be resizing the pvc directly in k8s. The ability to do that might depend on minikube's version and other specifics, but it's worth a try. I guess you need the
/userdir
to increase in size right?
j

José Tomás Pérez

03/29/2022, 4:02 PM
yes that's right
let me try, i have an older version
j

Jacopo

03/29/2022, 4:06 PM
Do you already know how to resize it? If not I'll look into the specific command you would need to issue. I am deleting my local cluster to see if we can achieve a different size by simply changing a couple of values in the repo. Not super clean, we will likely make a release tomorrow to expand the CLI scope
If you are doing an install from scratch you will be able to set the userdir and builder cache size by changing the values in
deploy/helm/charts/cluster-resources/charts/orchest/values.yaml
deploy/helm/charts/cluster-resources/charts/nfs/values.yaml
then running
bash orchest install
with clean install I mean either deleting the minikube cluster or deleting the
orchest
namespace
j

José Tomás Pérez

03/29/2022, 4:10 PM
great help, we are newbies with kubernetes
we'll try these commands
j

Jacopo

03/29/2022, 4:19 PM
About resizing existing pvcs, couple of handy commands that will allow you to check if you can resize them on the fly:
kubectl edit pvc -n orchest userdir-pvc
kubectl edit pvc -n orchest image-builder-cache-pvc
will open up the pvc yaml and will allow you to try and edit the pvc. My guess is that it won't allow you to when using a default minikube installation, so reinstalling orchest is your best bet right now
If you don't have
kubectl
installed this will do it:
minikube kubectl -- edit pvc -n orchest image-builder-cache-pvc
j

José Tomás Pérez

03/29/2022, 4:36 PM
thx a lot!!
now my team is trying out
j

Jacopo

03/29/2022, 4:41 PM
No problem 🙂 . If you changed the pvc size by changing charts values you can use those same commands to confirm that it worked
j

José Tomás Pérez

03/29/2022, 7:55 PM
@Jacopo We've tried to install from scratch and changing the
.yaml
but still the disk space in orchest is 20gb... Is there a step we are missing??
👀 1
n

Navid H

03/29/2022, 9:32 PM
@José Tomás Pérez: could you double check if you changed these values. https://github.com/orchest/orchest/blob/master/deploy/helm/charts/cluster-resources/charts/orchest/values.yaml#L6 https://github.com/orchest/orchest/blob/master/deploy/helm/charts/cluster-resources/charts/nfs/values.yaml#L14 if yes. could you get the pvc configs with with following command and return the response.
kubectl get pvc -n orchest -o yaml
j

Jacopo

03/30/2022, 7:50 AM
Sorry to hear that, I second what @Navid H recommended
j

José Tomás Pérez

03/30/2022, 1:04 PM
@Jacopo @Navid H
👀 1
image (43).png,image (42).png
j

Jacopo

03/30/2022, 1:06 PM
Hey @José Tomás Pérez thanks for coming back with this. How are you verifying that the disk space is still 20 gbs?
j

José Tomás Pérez

03/30/2022, 1:11 PM
here in settings of orchest
j

Jacopo

03/30/2022, 1:13 PM
I see, that looks strange. It could be that the value that is being read is wrong actually. I am reinstalling with a different pvc size, will get back to you.
j

José Tomás Pérez

03/30/2022, 1:14 PM
thanks!!
j

Jacopo

03/30/2022, 1:21 PM
I wonder if what we are looking at is actually minikube allocated disk size. Coincidentally it's 20 gigs. Perhaps a different version, os distribution or driver is making it act differently than what we have observed. Mind stopping and then starting minikube with an additional
--disk-size
flag?
From minikube
--help
Copy code
--disk-size='20000mb': Disk size allocated to the minikube VM (format: <number>[<unit>], where unit = b, k, m or g)
j

José Tomás Pérez

03/30/2022, 1:33 PM
we did that @Jacopo but with no results
@Max Moreno is doing the install
j

Jacopo

03/30/2022, 1:38 PM
Bit of a long shot but could it be that you need to start minikube with that size then install Orchest? About the disk size that you are seeing in the settings page, I think we can assume it's bugged, i.e. don't trust it. To actually verify the available disk space you could try to get into the
/userdir
of the
orchest-webserver
:
kubectl exec -n orchest -it deployment/orchest-webserver -- bash
I think that getting in there and actually trying to write 20 gigs of data in
/userdir/data/<whatever file>
will actually tell you if the pvc changes have worked
m

Max Moreno

03/30/2022, 1:41 PM
I'm going to try a clean install by increasing the disk space of minikube
j

Jacopo

03/30/2022, 1:43 PM
Hey @Max Moreno, before doing that it might be worth `exec`ing into the webserver, likely to takes less time to verify
m

Max Moreno

03/30/2022, 1:45 PM
@Jacopo I already deleted everything and I'm creating the instance
👍 3
@Jacopo I don't know if it's normal, but it takes a long time at that installation percentage
j

Jacopo

03/30/2022, 2:16 PM
It can definitively take some time due to the number of images that are being downloaded. There is an issue into which we are looking where the installation is basically stuck. To fix that you will need to delete the
orchest
namespace then run the installation again. Most likely related to download timeouts, fixing soon :))
👍 1
r

Rick Lamers

03/30/2022, 6:18 PM
Thanks for bearing with us @Max Moreno and @José Tomás Pérez. We just released the k8s version and are streamlining the installation process.
m

Max Moreno

03/30/2022, 6:23 PM
@Rick Lamers thank you for the help provided!
j

José Tomás Pérez

03/30/2022, 6:58 PM
🙌
m

Max Moreno

03/30/2022, 7:00 PM
@Jacopo I've been trying to stop the service for 1 hour, any recommendations?
n

Navid H

03/30/2022, 9:40 PM
@Max Moreno Thanks for coming back to us. could you get the status of pods in the orchest namespace by following command and return the response.
Copy code
kubectl get pods -n orchest -o json
m

Max Moreno

03/31/2022, 1:22 PM
finally install version 2022.03.08, it worked without problems
j

Jacopo

03/31/2022, 1:24 PM
Happy to hear it @Max Moreno, are you still using kvm2 as the driver?
We will keep looking into this issue internally
m

Max Moreno

03/31/2022, 1:25 PM
this time install minikube with docker
j

Jacopo

03/31/2022, 1:26 PM
I see, makes sense. I guess we need to cover more ground when it comes to other minikube drivers. Sorry that it took so long for this to work, hope you have a good time using Orchest :))
👏 1
m

Max Moreno

03/31/2022, 1:27 PM
@Jacopo thanks for the help, so far everything is working perfectly
👍 2