Orchest is an open source tool for building data pipelines.

Orchest Community

It turns out that this is true for every folder that I create. By default a hidden .ipynb_checkpoints folder gets created in all of them with or without the presence of a jupyter notebook.

image.png

Would disabling the auto save documents option solve this issue?

I think this could work. `.ipynb_checkpoints` "pollution" is indeed a bit of a troublemaker. One area I would investigate is whether Meltano has something of an "ignore paths that match this glob pattern" feature

Yes I investigated the same and came across the use of .dbtignore files. in case someone is facing a similar issue, then please have a look at this documentation <https://docs.getdbt.com/reference/dbtignore>

Is it possible to collaborate in orchest? Like can multiple users have access to the same project?

There is actually no user separation in Orchest. Every logged in user has access to the same projects. 

Hope that answers your question!

But what if I want to have people logged in with different accounts have access to a common project. Is that possible?

Oh you mean different accounts from Orchest Cloud? In that case, no.

If you want multiple people to use the same project, then 1 person needs to create an instance on Orchest Cloud and create additional accounts (username + password) through the Settings in Orchest. Then these other people can log in through the URL of the instance (so not through `<http://cloud.orchest.io|cloud.orchest.io>`).

My free orchest instance keeps on failing to start. Any ideas why?

Looks like your instance has a completely full disk, so it's failing to start

yes the status indicator is not up to date because the instance is not responding anymore

So, what should I do? Unless I cannot get into the instance, I cannot delete data

You can either increase the disk size through the cloud panel or you can tell me what you'd like to have deleted and I'll take care of that through the file system, at the moment these single node instances aren't very flexible when it comes to storage, they are free instances mostly meant for  trying out things; the multi node offering has unlimited elastic storage

Hi guys, I've created a new orchest cluster, however, the celery worker is unable to access the celery-backend-result database while trying to build new images. Any thoughts on why this may happen?

```psycopg2.OperationalError: FATAL:  database "celery_result_backend" does not exist


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/backends/database/__init__.py", line 47, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/backends/database/__init__.py", line 115, in _store_result
    session = self.ResultSession()
  File "/usr/local/lib/python3.9/site-packages/celery/backends/database/__init__.py", line 106, in ResultSession
    return session_manager.session_factory(
  File "/usr/local/lib/python3.9/site-packages/celery/backends/database/session.py", line 88, in session_factory
    self.prepare_models(engine)
  File "/usr/local/lib/python3.9/site-packages/celery/backends/database/session.py", line 72, in prepare_models
    ResultModelBase.metadata.create_all(engine)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 4864, in create_all
    bind._run_ddl_visitor(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3122, in _run_ddl_visitor
    with self.begin() as conn:
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3038, in begin
    conn = self.connect(close_with_result=close_with_result)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3210, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3289, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3259, in _wrap_pool_connect
    Connection._handle_dbapi_exception_noconnection(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2106, in _handle_dbapi_exception_noconnection
    util.raise_(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3256, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 310, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
    self.__connect()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 590, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 597, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  database "celery_result_backend" does not exist

(Background on this error at: <https://sqlalche.me/e/14/e3q8>)```

The problem was solved by creating the database. Not sure why the database `celery_result_backend` was not created on the orchest startup though.

<https://github.com/orchest/orchest/blob/20c8b1555875de4ea93ec4246714d634241ba1df/services/orchest-api/app/app/celery_app.py#L9>

The `celery_result_backend` should be created if it doesn't exist. Maybe the deployment hadn't started the celery app yet which would have attempted creation?

<@U01CBUGK657> can dig deeper. Potentially some race condition where when celery was up the DB was unavailable. Which typically shouldn't happen when `orchest-controller` manages the services.

This is fairly surprising since we automatically deploy multiple Orchest instances on different platforms on releases, e.g. eks, minikube, etc. Haven't had the issue so far.

Could you provide more details about how is the deployment being performed? As Rick mentioned this should indeed not happen

Sorry for the delay. The same problem ocurred to day in another environment.

 The deploy is being done using the following process:
1. Create EKS cluster / Node group using terraform
2. Install calico ( to overcome the EKS limitation of network interfaces )
3. Create iamserviceaccount
4. Install ebs-csi-controller
5. Create namespace orchest
6. Install nginx in the EKS cluster.
7. Deploy orchest-controller.yaml using kubectl apply
8. Deploy orchest-cluster.yaml using kubectl apply.

Anything special (i.e. any changes w.r.t. the original yaml) when it comes to the database? Or any chance that some interaction with calico is going wrong ? Both the `orchest-api` and `celery-worker` will attempt to create the database if it doesn't exist when they start (<https://github.com/orchest/orchest/blob/20c8b1555875de4ea93ec4246714d634241ba1df/services/orchest-api/app/app/__init__.py#L83|link>). The following logs would be interesting (the important part of the logs is likely the head rather than the tail given that the logic which creates the db is run on start)
• `orchest-api`
• `celery-worker`
• `orchest-database`


Hi all, if I'm hosting locally with `minikube`, how can I get it to serve a set of files on my local Mac to be available for consumption within Orchest?

I tried slightly modifying <https://docs.orchest.io/en/latest/development/development_workflow.html#cluster-for-development|the instructions for local development>:

```minikube delete

minikube start \
  --cpus max \
  --memory max \
  --addons ingress metrics-server \
  --mount-string="/Users/kevinschaich/shared-data:/shared-data" --mount

orchest install

sudo minikube tunnel```
but there's no volume under /shared-data in the Orchest sidebar nor viewable if I do `ls` from the Jupyter terminal.

Hi <@U04NAF9JDAT>, I'm afraid that this kind of mounting will not help in this case, what we are doing for development mode (`--dev`) is that 1) we mount the local directory (i.e. the repo) in the minikube container 2) create a volume which mounts an host path (the repo as seen from the POV of the minikube container)  into the pods which make use of hot reloading. So in this case you are mounting your shared-data directory in the minikube container, but it's never exposed to any Orchest pod.

With this said, Orchest doesn't have an explicit functionality to share data between the host system and the cluster. My go to solution for this, on linux, is to move data in the local directory used by docker for the volume of the minikube container, in particular, in what will be later used as the Orchest `/data` directory, i.e. `/var/lib/docker/volumes/minikube/_data/hostpath-provisioner/orchest/userdir-pvc/data/`

I don't have a mac within reach at the moment but depending on your installation (docker for desktop?) this might not be possible. What I would try is to mount your local directory in the path the minikube container uses for provisioning the Orchest `/data` directory , through the `--mount-string` or `--mount` flag (note that the performance/robustness of this minikube feature can be less than optimal).

All in all, the mount string could look something like this:
`--mount-string="/Users/kevinschaich/shared-data:/var/hostpath-provisioner/orchest/userdir-pvc/data/shared-data"`

To verify the exact path and to verify that the mount is working correctly, you can use `minikube ssh` and check out what's in `/var/hostpath-provisioner/...`

&gt; but there's no volume under /shared-data in the Orchest sidebar nor viewable if I do `ls` from the Jupyter terminal.
Generally speaking Orchest works through 2 directories, the current project directory and the `/data` directory. The `/data` directory is shared across all projects, and will be visible from all steps, jupyter kernels etc., while the `/project` directory mounted to the different steps/kernels etc. is context dependent

Hello ! I'm trying to deploy orchest on GKE without sucess. Can anyone helps me or suggest me a better way to deploy orchest ?

Hi <@U04PG7TQHEW> :wave:

Did you follow the installation instructions for GKE from our docs? (<https://docs.orchest.io/en/stable/getting_started/installation.html|link>).

The easiest way to deploy Orchest (if you don't have extensive Kubernetes experience) is to install Orchest on minikube. You can find these instructions in the docs as well (under <https://docs.orchest.io/en/stable/getting_started/installation.html|Linux > minikube&gt;). If you are planning to host Orchest on a cloud VM, then you will need to run a reverse proxy to expose it as well (<https://docs.orchest.io/en/stable/getting_started/installation.html#reverse-proxy|link to docs>).

Hope that helps!

Thank you <@U0184RP9KB8>, I'll check that out.

Hello guys, some of the jupyter-servers in our custom deployment are stucked in the following state:
```Awaiting boot lock...
Awaiting boot lock...
Awaiting boot lock...
Awaiting boot lock...
Awaiting boot lock...
Awaiting boot lock...
Awaiting boot lock...
Awaiting boot lock...
Awaiting boot lock...
Awaiting boot lock...```
```Because of this, we are unable to create sessions in those projects.

I have also noticed that by restarting the orchest cluster using the **restart** button, some of the projects that were not working are fixed, but other projects that were working starts to have the same issue.

Some important points:
- I have not found a lock in the $lockdir
- By running the /start.sh with the same parameters, I could reproduce the same result in jupyter-servers that are not working.```

Hi, could you exec into the orchest webserver and take a look at what's in the `.orchest/user-configurations/jupyterlab/lab` path of the userdir? I'd expect the lock to be there

What's in the `/usr/local/share/jupyter/lab` path of jupyter servers that are and are not failing? I.e. is the lock there for any of them?

If the lock is not there, I'd exec into the failing jupyter servers and take a look at the result of

```userdir_path=/usr/local/share/jupyter/lab
lockdir=$userdir_path/.bootlock
mkdir $lockdir```
I guess the error message will be interesting

jupyter-server-works.png

one thats not working.png

The interesting thing is that the lock is in present in both scenarios:

Logs from a server that's working properly:

Have you tried deleting the lock file? That should pretty much suffice, this snippet of logic

```release_lock() {
    rm -rf $lockdir
}```
is run once the startup logic of the jupyter server is done, perhaps something has gone wrong and a dangling lock has been left around

I haven't tried. Should I do it on the webserver or on each of the containers?

Only once, doesn't matter from which container since those directories are mounted, webserver is ok

I'll try it right now, thanks for the support, as always, Jacopo.

Have you tried restarting the jupyter server pods that are stuck? Now that the lock is there it shouldn't be a problem anymore

are they still stuck waiting on the lock? How many such pods? Could it be that's because there are many jupyter server going through the logic sequentially? I.e. one lock acquire at a time

Currently, 7 of them are stucked in CrashLoopBackoff

I'll kill the deployments and try this process one by one to see if it works

I see, I'd be curious to see what they log before crashing

By deleting the deployments and starting them one by one, I was able to fix this issue. Thank you very much, <@U01CBUGK657>.

Screenshot 2023-04-28 165330.png

As in <https://docs.orchest.io/en/stable/getting_started/installation.html#> run minikube ip, you'll get an ip to which you'll be able to reach the applicatio at ip:port, port should be 8000

You can also change your hosts file to map that ip to something easier to remember, e.g.

`192.168.49.2    <http://localorchest.io|localorchest.io>`

how to change? requesting you to please share some document

You can add such an entry to `/etc/hosts`, the ip must be the one returned by `minikube ip`

added, but it shows website not available

when visiting <http://localorchest.io|localorchest.io>

what are you getting from running `minikube ip`?

And what are you getting when going at `192.168.49.2:8000`? *http*

Screenshot 2023-04-28 171218.png

Screenshot 2023-04-28 171932.png

Even with http also it is giving the same result

could you try on port 80? :thinking_face:
`<http://192.168.49.2>`
or
`<http://192.168.49.2:8000/projects>`

Is the ingress addon enabled?  `minikube addons enable ingress`

Did you use the convenience install script in the installation docs?

Are you running through windows  + Windows Subsystem for Linux?

```{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}```


This is a response from kubernertes, what endpoint are you querying?

btw, here's some internal notes about how we proceeded when installing Orchest in windows through wsl, it's not officially supported but imo the issues you are having are solvable, although I'm not sure how much I can help without much more context around your setup

• open a terminal in windows, run `wsl --install`
• reboot
• verify wsl is working, `wsl --install -d Ubuntu`
• install docker for desktop in windows, i.e. not through the wsl terminal. Once installed you can set it to use the wsl + the distribution of your choice. Follow these instructions: <https://learn.microsoft.com/en-us/windows/wsl/tutorials/wsl-containers>
• once that’s done get into the terminal and run wsl with `wsl`
• install k9s, pip, orchest-cli and then install orchest
```# k9s
wget <https://github.com/derailed/k9s/releases/download/v0.25.18/k9s_Linux_x86_64.tar.gz>
tar -xvzf k9s_Linux_x86_64.tar.gz
sudo mv k9s /usr/bin


sudo apt-get update &amp;&amp; sudo apt-get install orchest-cli -y
pip3 install orchest-cli
orchest install
# Ingress
kubectl apply -f <https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.1/deploy/static/provider/cloud/deploy.yaml>```

pls help，<https://github.com/orchest/orchest/issues/1531>

Hi, I might have time to look into that tomorrow

At a first glance, I'd make sure something isn't going wrong with the installation when it comes to the Orchest version, you are installing with

`export VERSION="v2023.01.8"`

but you report `v2023.04.2` from the settings page. Might be worth taking a closer look into the yamls

Does the cluster have an ingress? Like nginx? <https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx>

The coredns error doesn't seem something on our end, but tomorrow I'll try to look further into this

Hi 小饭, if you could provide a a short script or summary of how you deployed kind + Orchest it'll likely help me in finding a solution

Hello guys, we're implementing some security measures in our orchest cluster.

I was not the one that have created our deployment process in the past...

We were thinking about using AWS WAF to secure it. However, since our load balancer is of type network (layer 4), WAF is not compatible. Is there a reason for orchest to create a Network Load Balancer instead of a Application Load Balancer?

Here is our manifest:
```apiVersion: v1
kind: Service
metadata:
  annotations:
    <http://service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout|service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout>: "60"
    <http://service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled|service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled>: "true"
    <http://service.beta.kubernetes.io/aws-load-balancer-ssl-cert|service.beta.kubernetes.io/aws-load-balancer-ssl-cert>: ....
    <http://service.beta.kubernetes.io/aws-load-balancer-ssl-ports|service.beta.kubernetes.io/aws-load-balancer-ssl-ports>: https
    <http://service.beta.kubernetes.io/aws-load-balancer-type|service.beta.kubernetes.io/aws-load-balancer-type>: nlb
    <http://service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags|service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags>: environment=....
  labels:
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: controller
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: ingress-nginx
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: ingress-nginx
    <http://app.kubernetes.io/part-of|app.kubernetes.io/part-of>: ingress-nginx
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: 1.3.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: tohttps
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: http
  selector:
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: controller
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: ingress-nginx
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: ingress-nginx
  type: LoadBalancer```


Hi Rafael, in principle I don't see why you couldn't go with an ALB rather than a NLB assuming the due diligence checks take place. You might have to double check the controller for any quirky behaviour around balancers (not that I know of, but I wasn't fully involved with the controller) and the logs streaming since they go through websocket

I didn't have the chance to work with WAF so I can't really help with any knowledge in that direction sadly

I was able to workaround this, it's an issue of the ingress-controller itself, it seems that it's not compatible (directly at least) with application load balancers, just classic and network load balancers. However, we were able to use a load balancer by changing the service type to NodePort and forwarding requests from an existing load balancer to the node port.

Happy  to hear you found a workaround :slightly_smiling_face: