set the channel topic: Ask us anything about Orchest! Check out the GitHub project here: https://github.com/orchest/orchest

Hi @rohit sohlot! Awesome to have you join first 💪. Please feel free to share any thoughts or questions you have and we will get back to you ASAP :)

Hi @Yannick, thank you :). I will try to share the content to help Orchest grow and improve. I personally liked it a lot.

Welcome @Derek Chia. Glad to have you! Let us know if you need anything

Hi, I was looking for tools to automate my ml pipeline and found Orchest which looks exactly like what I needed. However, I'm still in the stage of comparing several tools and make a report to convince my team. Do you know if there is any articles that compare Orchest to other tools available? like pros and cons of each one would be very appreciated.

Hi @Natthawute Saelim thanks for joining! Happy we made a good first impression. I'd like to point out that at this point we are still very early stage and I think that should be considered in your comparison. What other tools did you have in mind? Maybe I can contrast for you to help you understand the key differences.

At this time we also don't know of any articles comparing tools. Although we'll soon be included in pt. 2 of https://towardsdatascience.com/25-hot-new-data-tools-and-what-they-dont-do-31bf23bd8e56

@Rick Lamers Thanks for the reply. I’m still new to this field. But I assume your tool is similar to AirFlow? I checked AirFlow once but it seems like an overkill for my task. Basically, I want to make an automated pipeline of getting data from hadoop, make some preprocessing, train a model, make an evaluation, and output a report. Nothing special. Since most of our base code is written in python and jupyter notebook, I feel like your tool is very fit to our situation. Are you aware of other tools that might suit my situation?

@Natthawute Saelim it does indeed share similarities to AirFlow. In essence you're managing a DAG to run your code. However, we run each DAG step in its own container. Allowing you to more easily combine multiple codebases/projects if they have different running requirements

We also have Jupyter notebook integration which I don't really see in AirFlow (could be missing it). And we allow you to interactively edit and execute parts of your DAG while you're constructing it

It sounds like your use case is supported by Orchest today. Is model training dependent on GPU(s - multiple)?

@Rick Lamers we train our current model using only CPU for now. We do have a plan to change to more complicated modes that may require GPU though but its' not in the near future. Does Orchest have any issue with GPU? By the way, I quickly tried installing Orchest on my server (centos) but got 500 internal server error. Any way to check what I did wrong?

Hi @Natthawute Saelim, this internal server error should be fixed now. To get the latest version of Orchest you can run

./orchest.sh update

Similarly to running

./orchest.sh start

that starts the application. And we definitely plan on supporting GPUs, which should not be too much of a hassle due to our use of containerization. Hope that helps. Let me know if you need further assistance.

@Yannick Thanks ! I'll try when I'm back to my office.

Indeed, the fix should now be deployed to Docker Hub which will get pulled in with

./orchest.sh update

Hi @Stelios Gisdakis! Great to see you here :)

@Luciano Resende hi welcome! Happy to have you. Love your work on Jupyter.

@howie hu Welcome!🖐️

Hi @Rick Lamers, happy to see you 🤗.

Hello! Just discovered your project and looks interesting! I have a question though, which I couldn't find an answer for in the documentation: is it possible to schedule the execution of a pipeline? One thing I struggle with today is that I often need to run my notebooks daily, and I wonder if Orchest could be of any help on this?

Hi @Edwin! You can already schedule to run notebooks once (with experiments). It allows you to set the execution date and time. For repeated execution we're working on a 'cron jobs' feature. But that hasn't shipped yet.

I recorded a Loom to demonstrate how to schedule using experiments: https://www.loom.com/share/7a8e1a2a0b584aacb4e2d18a08e474b5

Cool thanks! And do you already have an approximate release date for the cron-like feature or not yet?

Hi everyone! We just shipped a major critical new feature: Custom Images. Easily define your own dependencies and use the resulting image as the base of your pipeline steps. Simply run

./orchest.sh update

and click on the 'Images' menu item on the left.

Hi everyone! How do you pass parameters from one step to another one?

Hello everyone, today marks an important release for Orchest. As we're introducing a super simple update mechanism that you can run directly from your browser. Find it and many other improvements on our GitHub https://github.com/orchest/orchest As always, if you run into any issues just ping us here.

Hi everyone! A quick update about our latest release to

master

. We've improved some important areas of Orchest: • Pipeline step log outputs are prettier and faster with Xterm.js based websocket streaming of logs • Pipeline session control directly from the pipeline list view. • Updating styling of the included File Manager • Ability to view interactive pipeline execution status after exiting pipeline editor • Removal of unnecessary eviction of JupyterLab <iframe> resulting in less load time when moving in & out of the pipeline editor

I just found Orchest, and I love it! I have a question about triggering a pipeline, though. Can it be done remotely? is there an API for talking to the orchest server to automatically run DAG jobs? I couldn't find anything like that in our documentation.