https://www.orchest.io/ logo
Title
k

Kay Kayyy

10/01/2021, 9:37 PM
is there a concept of "streaming" with orchest? it seems like everything is being run in batch, correct? the use case in the quickstart would be i never want the
get-data.py
step 1 to stop running. it would run endlessly (keeps streaming data in) and we could program a "trigger" of some type that then would kick off step 2
pre-processing.ipynb
r

Rick Lamers

10/01/2021, 9:45 PM
It’s a batch oriented paradigm. But it’s fairly trivial to consume from a message stream in batch pipeline to handle streaming style workloads
k

Kay Kayyy

10/01/2021, 9:55 PM
^^ would be great to see an example
r

Rick Lamers

10/01/2021, 10:14 PM
Great idea. I’m adding it to my todo for adding it to https://github.com/orchest/orchest-examples (these automatically become visible on the examples page in Orchest itself). Expect to see it next week