hi everyone here JT from Santiago Chile! I'm very excited with Orchest and I would love to know more about how it manages cron jobs, cause now I have a EC2 that explodes cause python once a week xD

Hi @José Tomás Pérez it is great to have you! What specifically would you like to know about the way in which we manage cron jobs? I'd be happy to answer any questions you might have 😄

hi @Yannick!! thanks for your welcome

That is a great question. Depending on the value you have set for

MAX_JOB_RUNS_PARALLELISM

in the global settings (docs) multiple pipelines run in parallel, including for the same job. So to answer your question: yes, the same job will overlap with the next one (if the duration is longer than time between schedules). Of course you can set

MAX_JOB_RUNS_PARALLELISM

equal to one to have them be queued instead. But do note that this would impact other jobs as well such that only one pipeline can run at any time (across all jobs in all projects). Hopefully that answers your question.

i would like to know if orchest let the same job overlap with the next one

What would you like to be the behavior?

Welcome @José Tomás Pérez! It would be great to see if the current model of cron jobs suits your needs, we're actively looking for feedback from the community 🙂!

Thanks for your clear answer @Yannick ! @Rick Lamers don't doubt about it 😉

I would like to have a parameter that blocks overlap between the same job

That's a good idea. Could you turn it into a GitHub issue? https://github.com/orchest/orchest/issues

Thanks for your feedback ❤️

🙌

Thanks for creating the issue as well :)