Couple questions here:
Is it possible to externalize the execution of the code from the instance running Orchard ?
What if you have multiple hundreds/thousands of tasks that you want to execute in parallel ? Can we connect that with an external kurbernetes cluster or lambda funtions for example?
Basically how to have unlimited resources ?
Also what’s the scheduler running under orchest ? Celery ? is it able to handle huge amount of tasks ?
02/25/2022, 6:48 AM
hello Louis, it's Juan Luis here from Orchest. while I gather some answers from the team, I wanted to ask some follow-up questions: when you talk about tasks to execute in parallel, are you thinking of a large number of independent batch jobs? or rather, complex task graphs with interdependent steps?
our scheduled tasks run on Celery and the steps are isolated, so it should work just fine for a considerable amount of tasks. if you find a bottleneck here we would be happy to take a closer look.
about the resources, we are undergoing a migration to Kubernetes, and when we are done (hopefully end of Q1) you will have as much computational power as you want to pay for.
02/25/2022, 3:23 PM
Thanks for your answer @juanlu.
This is good news ! I’ll give it a try 🙂