Hi, Guys! I was wandering: does make sense to use Spark inside Orchest? I mean, not just connect a Spark Session and submit jobs to other cluster, but to have an infrastructure of containers inside the same k8s and highly coupled with Orchest administration

does make sense to use Spark inside Orchest?

I would say yes, but for the foreseeable future, only if those Spark jobs run outside of Orchest. Similar to how you can instruct

dbt

from inside Orchest, you can instruct Spark jobs. In other words, we are not planning to have Orchest be responsible to manage a Spark cluster. Of course these things can change in the future. I do like your thinking though 🙌

Makes sense, @Yannick. Many of our potencial customers does have tons of PySpark code that they don't want to re-write to SQL or anything else. I was think about how is the best way to co-exist with Spark when we are deploying Orchest

It just so happens that @Rick Lamers created an example showcasing PySpark in Orchest 🙌 https://github.com/ricklamers/orchest-hello-spark