@Rick Lamers Thanks for the reply. I’m still new to this field. But I assume your tool is similar to AirFlow? I checked AirFlow once but it seems like an overkill for my task.
Basically, I want to make an automated pipeline of getting data from hadoop, make some preprocessing, train a model, make an evaluation, and output a report. Nothing special.
Since most of our base code is written in python and jupyter notebook, I feel like your tool is very fit to our situation.
Are you aware of other tools that might suit my situation?