I have one more question apart from orchest cloud instance being run only when there is a job, are there other cost optimization solutions considered? I understand the issue that orchest is limited to 1 instance and cannot operate multiple, therefore it needs to run full one. One of use cases is api-rate-limited data gathering, which doesn't need cpu or ram but needs time, would be cool to have something cheap for those steps(but I understand that it might be not a target use case for you)

Good question and example. In your case I would schedule short-lived jobs that trigger every 30 minutes and might only execute 3 minutes to saturate the API rate-limit. I'm pleased to announce that auto stop and starting is in the works. We're calling it JIT execution internally for the time being 😄

Cool that its in the works 🙂 My case is complicated as third party is using rate limits per different intervals, even per min, 10 sec. So I can do only well-defined load. AWS Fargate does an awesome job for this giving full customization. But joining up and re-joining ecs tasks running on fargate is such a pain 😞 So I guess if using orchest i would need to somehow combine jobs executions so the cpu / ram is utilized fully

For such high granularity I think something like a pure serverless framework would be a better fit, is my guess. Perhaps AWS Lambda?

I think it will not work the same. Right now I will try to adjust pipelines to squeeze out time as much as possible

Awesome! Thanks for sharing. It's always fun to read. Let us know how it goes 🙂