Is it possible to execute a permanent process once you start a session in a pipeline? I have a production DB behind a bastion host, so I want to use AutoSSH to set up a permanent connection to the bastion host and forward it to the production database. Is that possible right now?

You probably want to run AutoSSH on the host (that is running Orchest). What do you mean forward it to the production database? Is the database running on the bastion or in the network the bastion is on?

In the network the bastion is on. I'm not that skillful in docker... yet 😅 . This means that I should start docker in a VM with AutoSSH running?

Is the Orchest host a Linux machine?

Right now I'm testing it locally, but I had the idea that I would deploy as a docker container later on.

AutoSSH is probably not the best way to connect to the production database in Orchest. If you really want to go that route you can make it work by running a process inside the step (remember steps run in their own container in Orchest) that needs to connect to the production database. AutoSSH is like a reverse tunnel that lets you then connect to a local port in the container like localhost:5000 to the forwarded post on the bastion. That still doesn't get you to the production database which is on the network of the bastion host right?

This is the command that I'm using:

autossh -M 0 \
        -o ServerAliveInterval=30 \
        -o ServerAliveCountMax=3 \
        -v -N -L 3306:$PROD_DATABASE:3306 \
        $SSH_USER@$SSH_BASTION_HOST \
        -i keyfile.pem \
        -o StrictHostKeyChecking=no

Then I connect to localhost on port 3306 and I'm connected. Maybe I'm wrong by saying the prod database is in the network the bastion is on.

But if AutoSSH is not the best way maybe you can point me in the right direction? I'm open to suggestions. Also, what would be the best way to deploy Orchest? A VM?

A VM would be easiest.

autossh -M 0 \

-o ServerAliveInterval=30 \

-o ServerAliveCountMax=3 \

-v -N -L 3306:$PROD_DATABASE:3306 \

$SSH_USER@$SSH_BASTION_HOST \

-i keyfile.pem \

-o StrictHostKeyChecking=no

This could work. One common alternative would be to deploy Orchest on a VM in the network that can directly reach the database.

I'm going to set it up in a VM then and let you know the results 🙂

Awesome! Let us know if you need help with anything