Kevin Schaich
02/08/2023, 10:04 PMminikube
, how can I get it to serve a set of files on my local Mac to be available for consumption within Orchest?
I tried slightly modifying the instructions for local development:
minikube delete
minikube start \
--cpus max \
--memory max \
--addons ingress metrics-server \
--mount-string="/Users/kevinschaich/shared-data:/shared-data" --mount
orchest install
sudo minikube tunnel
but there's no volume under /shared-data in the Orchest sidebar nor viewable if I do ls
from the Jupyter terminal.Jacopo
02/09/2023, 7:59 AM--dev
) is that 1) we mount the local directory (i.e. the repo) in the minikube container 2) create a volume which mounts an host path (the repo as seen from the POV of the minikube container) into the pods which make use of hot reloading. So in this case you are mounting your shared-data directory in the minikube container, but it's never exposed to any Orchest pod.
With this said, Orchest doesn't have an explicit functionality to share data between the host system and the cluster. My go to solution for this, on linux, is to move data in the local directory used by docker for the volume of the minikube container, in particular, in what will be later used as the Orchest /data
directory, i.e. /var/lib/docker/volumes/minikube/_data/hostpath-provisioner/orchest/userdir-pvc/data/
I don't have a mac within reach at the moment but depending on your installation (docker for desktop?) this might not be possible. What I would try is to mount your local directory in the path the minikube container uses for provisioning the Orchest /data
directory , through the --mount-string
or --mount
flag (note that the performance/robustness of this minikube feature can be less than optimal).
All in all, the mount string could look something like this:
--mount-string="/Users/kevinschaich/shared-data:/var/hostpath-provisioner/orchest/userdir-pvc/data/shared-data"
To verify the exact path and to verify that the mount is working correctly, you can use minikube ssh
and check out what's in /var/hostpath-provisioner/...
but there's no volume under /shared-data in the Orchest sidebar nor viewable if I doGenerally speaking Orchest works through 2 directories, the current project directory and thefrom the Jupyter terminal.ls
/data
directory. The /data
directory is shared across all projects, and will be visible from all steps, jupyter kernels etc., while the /project
directory mounted to the different steps/kernels etc. is context dependent