https://www.orchest.io/ logo
#announcements
Title
# announcements
g

Gabriel Milan

01/27/2022, 2:47 PM
I was tinkering a bit with Orchest Cloud and it's got me wondering a few things: We're still setting things up and choosing our tools where I'm working, but some of them are kinda mature now: Prefect (on-prem), DBT (on-prem), GCS, GKE and BigQuery. This is pretty much our main stack. Our pipelines are very simple atm, they're essentially classic ETLs from SQL Server instances, APIs and BQ to BQ itself. The main challenge we face is that we'd need teams/projects/environments separation in order to achieve better cost splitting. That's why we're using Prefect, btw, for its Hybrid Execution Model (https://www.prefect.io/why-prefect/hybrid-model/) (we host our Prefect Server instance for cost optimization) Your UI would be awesome for our junior coders to develop, that's what caught my attention. Would you have any ideas on how to handle execution environments separation (like Prefect's hybrid model) on Orchest? Something like the Prefect agents, maybe
r

Rick Lamers

01/27/2022, 2:51 PM
The simplest way to run Orchest in multiple environments is to have separate deployments of Orchest (the open source project), one in each of the environments where you want to run. Orchest Cloud at this time doesn't support connecting to an agent of sorts that runs the actual pipeline processing code in a remote node using something like an agent. Good question and something we might support in the future (remote executor model of running pipelines separate from the "control pane" code).
Our pipelines are very simple atm, they're essentially classic ETLs from SQL Server instances, APIs and BQ to BQ itself.
It sounds like that usecase is a perfect fit for Orchest ; )
Installation of Orchest on a server/VM should be fairly straightforward: https://docs.orchest.io/en/latest/getting_started/installation.html
g

Gabriel Milan

01/27/2022, 2:57 PM
Installation of Orchest on a server/VM should be fairly straightforward: https://docs.orchest.io/en/latest/getting_started/installation.html
That's where I'm headed right now, but I was thinking more of deploying it in our k8s cluster. Would you have official support for it rn, something like a helm chart?
r

Rick Lamers

01/27/2022, 4:49 PM
You're about 1 month too early for the k8s deploy, it's going to be out most likely in February: https://github.com/orchest/orchest/pull/623
🚀 1
g

Gabriel Milan

01/27/2022, 4:56 PM
Awesome! Can't wait to have it! Just a quick thing I've noticed (that might be a trouble in the future) is this
.default.svc...
thing here https://github.com/orchest/orchest/pull/623/files#diff-431d03962d00cf50699fcb7d5026994d06dba27718f8770ba615da995a6adc72R27 I'm sure you're already aware, just wanted to demonstrate my worrying about the feature, as I'm expecting it to be deployed in an exclusive namespace
🙏 1
r

Rick Lamers

01/27/2022, 4:58 PM
Thanks for pointing it out, we're still in the process of defining the k8s details. It's very much our goal to integrate nicely with existing k8s clusters and be as idiomatic/'native' as possible. @Navid H @Jacopo ☝️ this is for you.
🙌 1
n

Navid H

01/27/2022, 5:11 PM
@Gabriel Milan: Thanks for pointing it out, we intend to remove the file you mentioned and use k8s native ingress for route handling.
1