https://www.orchest.io/ logo
k

Kevin Schaich

02/08/2023, 10:04 PM
Hi all, if I'm hosting locally with
minikube
, how can I get it to serve a set of files on my local Mac to be available for consumption within Orchest? I tried slightly modifying the instructions for local development:
Copy code
minikube delete

minikube start \
  --cpus max \
  --memory max \
  --addons ingress metrics-server \
  --mount-string="/Users/kevinschaich/shared-data:/shared-data" --mount

orchest install

sudo minikube tunnel
but there's no volume under /shared-data in the Orchest sidebar nor viewable if I do
ls
from the Jupyter terminal.
👀 1
j

Jacopo

02/09/2023, 7:59 AM
Hi @Kevin Schaich, I'm afraid that this kind of mounting will not help in this case, what we are doing for development mode (
--dev
) is that 1) we mount the local directory (i.e. the repo) in the minikube container 2) create a volume which mounts an host path (the repo as seen from the POV of the minikube container) into the pods which make use of hot reloading. So in this case you are mounting your shared-data directory in the minikube container, but it's never exposed to any Orchest pod. With this said, Orchest doesn't have an explicit functionality to share data between the host system and the cluster. My go to solution for this, on linux, is to move data in the local directory used by docker for the volume of the minikube container, in particular, in what will be later used as the Orchest
/data
directory, i.e.
/var/lib/docker/volumes/minikube/_data/hostpath-provisioner/orchest/userdir-pvc/data/
I don't have a mac within reach at the moment but depending on your installation (docker for desktop?) this might not be possible. What I would try is to mount your local directory in the path the minikube container uses for provisioning the Orchest
/data
directory , through the
--mount-string
or
--mount
flag (note that the performance/robustness of this minikube feature can be less than optimal). All in all, the mount string could look something like this:
--mount-string="/Users/kevinschaich/shared-data:/var/hostpath-provisioner/orchest/userdir-pvc/data/shared-data"
To verify the exact path and to verify that the mount is working correctly, you can use
minikube ssh
and check out what's in
/var/hostpath-provisioner/...
but there's no volume under /shared-data in the Orchest sidebar nor viewable if I do
ls
from the Jupyter terminal.
Generally speaking Orchest works through 2 directories, the current project directory and the
/data
directory. The
/data
directory is shared across all projects, and will be visible from all steps, jupyter kernels etc., while the
/project
directory mounted to the different steps/kernels etc. is context dependent
8 Views