Hi all,
I saw an ad for
orchest.io about an hour ago and have been reading the docs since... prob need to keep reading but maybe I can get a good answer here.
I run a rather large data pipeline in R scraping some government websites, ultimately this data ends up in an AWS Aurora database. How can I securely go about getting this data to there. Also regarding idle time of my
orchest.io instances do I pay for that?
I've done 3 things in the past 1) buy a large expensive on demand EC2 instance and start and stop the instance when scripts need to execute. This works always but is too expensive. 2) move code to AWS Lambda, this works for some of my scripts but others have runtimes longer than 15mins. 3) Cloud containers using Google Cloud Build + Google Cloud Runner + Google Scheduler, this is pretty good by overly complex at times. Also have run into issues with some scripts needing very different compute / memory.
Thanks,
Sean