is there a concept of "streaming" with orchest? it seems like everything is being run in batch, correct? the use case in the quickstart would be i never want the

get-data.py

step 1 to stop running. it would run endlessly (keeps streaming data in) and we could program a "trigger" of some type that then would kick off step 2

pre-processing.ipynb

It’s a batch oriented paradigm. But it’s fairly trivial to consume from a message stream in batch pipeline to handle streaming style workloads

^^ would be great to see an example

Great idea. I’m adding it to my todo for adding it to https://github.com/orchest/orchest-examples (these automatically become visible on the examples page in Orchest itself). Expect to see it next week