I just found Orchest, and I love it! I have a question about triggering a pipeline, though. Can it be done remotely? is there an API for talking to the orchest server to automatically run DAG jobs? I couldn't find anything like that in our documentation.
10/17/2020, 8:34 AM
Thanks for the input!
We could combine this together with the cron-line feature that @Edwin asked about. I definitely see the potential there :)
10/17/2020, 10:31 AM
Long story short, I would not recommend using this. But perhaps if you really want to you could trigger a pipeline execution manually.
In the file I've attached you see the four requests that are executed to: create an experiment (in both the scheduler back-end and the Orchest web-application) and trigger the execution of pipeline runs (along with the parameterization of each run).
I'd definitely consider these steps low level, and without proper coordination it leads to invalid application state. But I'll let you decide whether you want to tinker with it.
Just know that in the future we'll have a simple, clean and stable API to trigger DAG runs.
As a side note, if you've enabled authentication make sure you send the auth cookies along (two cookies created on login: auth_token, auth_username).