Hi, everyone. Hope you're doing well. I just found Orchest via @Rick Lamers' post on LinkedIn. Currently working as a one-man DE team in my current company so I had some questions. Currently, our Django app uses a Postgres RDS instance hosted in AWS. Every day, I move the data to Redshift and S3 via Glue, very straightforward. I then use dbt Cloud to transform the data to fact tables and so on for the tables in Redshift, no issues here.
However, the final part of my workflow is running some notebooks, the output of which I have to dump back into Redshift/S3 then run some dbt Cloud jobs again. This is where I hope Orchest can do two (2) things.
1. Integrate with dbt Cloud to remove the manual part of running these notebooks in between dbt Cloud runs.
2. Ideally be able to have Orchest speak to our Redshift instance that runs behind a VPC.
Point (1), I'm exploring thanks to some docs Rick shared in LinkedIn. My problem is Point (2). I believe I need to add an inbound rule for Orchest to be allowed to communicate with Redshift. Thing is, I don't think I've seen Orchest's IP address in the docs.
Anyone with any thoughts on Point (2)?
09/30/2022, 8:47 AM
We can assign a static IP to your Orchest Cloud instance to simplify (2). Can you DM your instance URL to @Yannick and we’ll get it sorted.
09/30/2022, 9:02 AM
Hey, Rick! Thanks for the reply on LinkedIn. Been trying to teach my old dog bones new tricks. 🙂 Sure, let me DM my instance.
It's working now thanks to @Yannick, now just to get dbt Cloud and see if I can automate my workflow completely.