Hi everyone! We have some exciting news to share. The first stable release of Orchest is live 🎉 (0.3.0)! We can't wait to see what you'll build with it.

Check out this quickstart walkthrough to see what it's all about

The most important new features are: • Projects: share code between pipelines, add pipelines to existing projects, and simply version Orchest projects using

git

. • Environments (simple container environments to make dependencies actually easy!) • Directly import

git

projects (from GitHub/GitLab) • New pipeline editor renderer (try zooming in and out!)

Thankful for being invited to check this out. Just what is needed because spreadsheets solvers are crashing with too many nonlinear equations and putting them into pandas with SciPy proves too difficult because I'm unable to visualize it.

Hello everyone! I'm a software engineer and co-founder of a civic tech startup with a small DS team. I'm interested in open source and involved in some related projects in the open data community. I checked out Orchest after the HN post a couple of weeks ago and liked what I saw. Looking forward to getting more experience in the platform, and thanks for developing this inspiring product in the open!

We just shared Orchest on /r/datascience. Feel free to join the discussion! https://www.reddit.com/r/datascience/comments/k61tby/orchest_an_open_source_tool_for_data_science/

Why can't I resize you say? Now you can!

Hi Rick

Saw your post on reddit and wanted to try out your software

When I do data = orchest.get_inputs()

Are you storing all of the previous data in RAM

If I am running it on a large dataset

Is this the way to be writing the source or is there any other option?

Hi there! Yes, calling

orchest.get_inputs()

will store the data of the previous step in RAM (more specifically the RAM of the pipeline step in which the function is invoked). When making the data available from the previous step you have the ability to use

orchest.transfer.output_to_disk()

instead of

orchest.output()

. If you are working with large datasets and still want to pass them through memory, then it is recommended to enable

auto_eviction

(https://orchest.readthedocs.io/en/latest/user_guide/other.html#pipeline-level-configurations) otherwise it will be kept in memory until you reboot the interactive session. Hope that answers your question :)

On another note, we are currently improving on the serialization that is happening when passing data from one step to another. What is the size of the datasets we are talking about?

In addition, for data set sizes that are very large you could also manually manage read/write by just writing to /data. That mounted directory is shared between all pipeline steps. You could use the data passing of

orchest.output(some_path)

to pass the path to the next step (to not hardcode where the files can be found).

We're getting ready to welcome Julia to the list of supported languages for Orchest. Stay tuned! (https://github.com/orchest/orchest/pull/85/files)

You can now try Julia in the latest release (v0.3.6)! Run

./orchest update

to check it out. Would love to get some feedback on our first attempt

Hi all! I'm Karolina, Data Engineer and colleague of @Oleg L 🙂 Come here for more insights

Glad to have you Karolína! If you have any questions, don't hesitate to ask 🙂!

We just pushed an exciting new release: 0.3.9! What does it bring? • Passing data between pipeline steps now supports named data passing

Heads up: this requires updating your old code. In short: 

orchest.output(data)

 becomes 

orchest.output(data, name)

.

• File viewer now contains a refresh button (see your Notebook fill up as it runs on the scheduler) • File viewer now allows you to easily navigate to the parent and child pipeline steps • Log output / build output to Xterm cleaned up to support loading style line rollback (\r) (edited screenhots)

image (3).png,image (2).png,Group 1.png

hi everyone, i was searching to smoothen my workflow on Jupyter kernels and more efficient work with it an then i stumbled upon, orchest.io glad to contribute in any way possible.

Hi there! That’s great. Let us know if you have any questions. Make sure to check the docs https://orchest.readthedocs.io/en/stable/ and import some of the sample projects https://github.com/orchest/awesome-orchest-pipelines

Glad to have you!

thanks rick

was going through the repo

and did some minor changes in documentation

if you still reviewing PL's would be glad to have reviews.