Allan Sene
10/06/2022, 12:09 AMYannick
does make sense to use Spark inside Orchest?I would say yes, but for the foreseeable future, only if those Spark jobs run outside of Orchest. Similar to how you can instruct
dbt
from inside Orchest, you can instruct Spark jobs.
In other words, we are not planning to have Orchest be responsible to manage a Spark cluster. Of course these things can change in the future.
I do like your thinking though 🙌Allan Sene
10/10/2022, 6:54 PMYannick