Title
#announcements
Allan Sene

Allan Sene

10/06/2022, 12:09 AM
Hi, Guys! I was wandering: does make sense to use Spark inside Orchest? I mean, not just connect a Spark Session and submit jobs to other cluster, but to have an infrastructure of containers inside the same k8s and highly coupled with Orchest administration
Yannick

Yannick

10/06/2022, 7:58 AM
does make sense to use Spark inside Orchest?
I would say yes, but for the foreseeable future, only if those Spark jobs run outside of Orchest. Similar to how you can instruct
dbt
from inside Orchest, you can instruct Spark jobs. In other words, we are not planning to have Orchest be responsible to manage a Spark cluster. Of course these things can change in the future. I do like your thinking though 🙌
Allan Sene

Allan Sene

10/10/2022, 6:54 PM
Makes sense, @Yannick. Many of our potencial customers does have tons of PySpark code that they don't want to re-write to SQL or anything else. I was think about how is the best way to co-exist with Spark when we are deploying Orchest
Yannick

Yannick

10/10/2022, 7:33 PM
It just so happens that @Rick Lamers created an example showcasing PySpark in Orchest 🙌 https://github.com/ricklamers/orchest-hello-spark