https://www.orchest.io/ logo
f

Fran

08/24/2021, 2:30 PM
Is it possible to execute a permanent process once you start a session in a pipeline? I have a production DB behind a bastion host, so I want to use AutoSSH to set up a permanent connection to the bastion host and forward it to the production database. Is that possible right now?
r

Rick Lamers

08/24/2021, 2:35 PM
You probably want to run AutoSSH on the host (that is running Orchest). What do you mean forward it to the production database? Is the database running on the bastion or in the network the bastion is on?
f

Fran

08/24/2021, 2:38 PM
In the network the bastion is on. I'm not that skillful in docker... yet 😅 . This means that I should start docker in a VM with AutoSSH running?
r

Rick Lamers

08/24/2021, 2:38 PM
Is the Orchest host a Linux machine?
f

Fran

08/24/2021, 2:40 PM
Right now I'm testing it locally, but I had the idea that I would deploy as a docker container later on.
r

Rick Lamers

08/24/2021, 2:44 PM
AutoSSH is probably not the best way to connect to the production database in Orchest. If you really want to go that route you can make it work by running a process inside the step (remember steps run in their own container in Orchest) that needs to connect to the production database. AutoSSH is like a reverse tunnel that lets you then connect to a local port in the container like localhost:5000 to the forwarded post on the bastion. That still doesn't get you to the production database which is on the network of the bastion host right?
can make it work by running a process inside the step (remember steps run in their own container in Orchest) that needs to connect to the production database.
For example in a pipeline step with a Notebook you can have a first cell that starts a process that runs AutoSSH:
Copy code
!autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -L 5000:localhost:3306 user@bastion.tld &
f

Fran

08/24/2021, 3:17 PM
This is the command that I'm using:
Copy code
autossh -M 0 \
        -o ServerAliveInterval=30 \
        -o ServerAliveCountMax=3 \
        -v -N -L 3306:$PROD_DATABASE:3306 \
        $SSH_USER@$SSH_BASTION_HOST \
        -i keyfile.pem \
        -o StrictHostKeyChecking=no
Then I connect to localhost on port 3306 and I'm connected. Maybe I'm wrong by saying the prod database is in the network the bastion is on.
But if AutoSSH is not the best way maybe you can point me in the right direction? I'm open to suggestions. Also, what would be the best way to deploy Orchest? A VM?
r

Rick Lamers

08/24/2021, 4:19 PM
But if AutoSSH is not the best way maybe you can point me in the right direction? I'm open to suggestions. Also, what would be the best way to deploy Orchest? A VM?
A VM would be easiest.
autossh -M 0 \
-o ServerAliveInterval=30 \
-o ServerAliveCountMax=3 \
-v -N -L 3306:$PROD_DATABASE:3306 \
$SSH_USER@$SSH_BASTION_HOST \
-i keyfile.pem \
-o StrictHostKeyChecking=no
This could work. One common alternative would be to deploy Orchest on a VM in the network that can directly reach the database.
f

Fran

08/24/2021, 5:24 PM
I'm going to set it up in a VM then and let you know the results 🙂
🙌 2
r

Rick Lamers

08/24/2021, 6:13 PM
Awesome! Let us know if you need help with anything
👍 1