We received a question about how you can use Orchest with R and utilize the data passing features. Check out this sample GitHub project (simply import Projects -> Import Project in Orchest) to see how to use R in Orchest. https://github.com/ricklamers/orchest-pipeline-r-python-mix For the curious, it utilizes the reticulate package for Python library interop.

Is there a way to add a private custom image from Docker Hub?

Is it possible to execute a permanent process once you start a session in a pipeline? I have a production DB behind a bastion host, so I want to use AutoSSH to set up a permanent connection to the bastion host and forward it to the production database. Is that possible right now?

I've been playing a lot with Orchest lately and I gotta say I'm hooked with the potential it has. I set it up yesterday morning, build a simple pipeline to run an ETL process to get the finance team an MRR and just like that it's already providing value 🙌 I know the framework is still in its alpha version (which to me it's impressive how well it works in this stage). I feel like I'll be working more and more with this, but I have a few questions: • What's the best way to provide feedback? • Do you have a public roadmap? • Where can I read the updates? I hope I'm not asking too many questions that are very obvious to everyone 🙂

I took the liberty of implementing an example Orchest pipeline that demonstrates how to use the Orchest SDK with Julia. ➡️ https://github.com/orchest-examples/julia-orchest-sdk/tree/main If you hadn’t seen the one that demonstrates the same in R, check out: ➡️ https://github.com/orchest-examples/orchest-pipeline-r-python-mix

Is

orchest.output()

the same as

orchest.transfer.output()

?

This is how the pipeline is going so far 😄 🛠️ It extracts information from our prod database, transform it using pandas and then pushing it to a Snowflake database. Besides passing dataframes from step to step using

orches.output()

it also stores the information in the

/data

folder to be accesible by other projects. I’m planning in adding a ML step to augment this information (like churn detection or sentiment analysis). What’s been difficult for me so far is: • Moving around the “map”. As you can see the pipeline is too big for my screen and can’t move it around. • Hiding sensitive environment variables is something desired • More documentation on what’s required to set up a custom environment would be great. An example would be useful. • Turn off a session after some time to avoid wasting resources. • Documentation on how to push projects into a a GitHub repositories. It was not obvious to me, and I want to be mindful not to break anything. • Sometimes when creating a new step I can’t type the letter

H

in Title or File path. • Snapping ability Those things aside, it’s been really a pleasure working with it and none of the complexities I mention before have stopped me from enjoying the framework in it’s alpha stage 🙂

After 3 weeks of hardwork and testing here is the complete tutorial on mlops using Orchest. It's beginner friendly too. https://pub.towardsai.net/a-new-way-of-building-machine-learning-pipelines-54700ed1aded

A cool demo pipeline I've been working on (and got contributions for from @Abid!) is a TTS pipeline that is based on an awesome open source TTS project called coqui. You can find more info in the README. But basically it generates speech from text using a state of the art deep learning model called VITS and sends a message containing the audio snippet to a Slack or Discord channel. https://github.com/ricklamers/orchest-coqui-tts Listen to the sample file to get an idea of the quality! 🎼

I wish to contribute more in upcoming weeks.

Hi! I'm trying to run the quickstart example, and i'm getting a

PermissionError

when the

get-data.py

step is downloading the data.

Hello Guys , Im getting issue while installing orchest on windows 10

When I trying to setup it , It says orchest is not internal or external command. How can install it on windows 10 ?

Thanks

Hey Guys! I want to make a theme for orchest. I have experience in wordpress theme development and have done frontend dev for some years just editing php, css and html template files. Im looking at orchest and see that it uses something called stitches. (never heard of it before now) I was wondering how to setup up my dev environment if was to make a theme. Kinda feel like a complete noob atm. 🙂

Hello! I'm having trouble with the installation. Can someone help me? ./orchest install 'docker' is not accessible for the current user: 'mahan' Automatically prepending 'sudo' for 'docker' commands... [sudo] password for mahan: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. See 'docker run --help'.

Hi! Has anyone implemented the streamlit service? I’m not able to install plotly 1. I installed the package in the environment but it seems to only affect the notebook, and not the streamlit service. 2. I went to Services > Edit Services > Streamlit and replaced the image to the one above (I modified the build to include streamlit and plotly) but I only managed to break the service 🥲 Can anyone tell me how to install packages for the streamlit service?

I’ve got a little 🐞 to report: The step parameters are not being saved, and there’s weird bit where I wrote parameter and it got saved sequentially letter by letter

Hi! while using vscode on Orchest cloud, I am not able to see option for running the cell.

Is there a specific channel to report issues?

Hi all! － We have some cool news to share! It means we’ll be able to make Orchest even better 🥳 https://techcrunch.com/2021/10/01/orchest-raises-3-5m-to-provide-a-simpler-way-to-build-data-pipelines/

hi - congrats on your funding! i saw it and decided to try it out, however the standard installation is failing. running ec2 ubuntu and your required version of docker. after it starts via CLI i hit the webapp and the home

/projects

is broken stuck in loading loop. chrome console showing a bunch of 500 errors. for example

/async/projects?session_counts=true&job_counts=true 500 (INTERNAL SERVER ERROR)

GET <http://IP:1122/catch/api-proxy/api/sessions/> 500 (INTERNAL SERVER ERROR)

Unable to fetch sessions Response {type: 'basic', url: '<http://IP:1122/catch/api-proxy/api/sessions/>', redirected: false, status: 500, ok: false, …}

when running the quickstart step 1...

PermissionError: [Errno 13] Permission denied: '/data/cal_housing.tgz'

where is that dir? it's not on the local machine

is there a concept of "streaming" with orchest? it seems like everything is being run in batch, correct? the use case in the quickstart would be i never want the

get-data.py

step 1 to stop running. it would run endlessly (keeps streaming data in) and we could program a "trigger" of some type that then would kick off step 2

pre-processing.ipynb

is there a concept of conditional steps? the use case in the quickstart would be after step 1, only run step 2

explore-data.ipynb

if some condition X is true, else if condition Y is true run

pre-processing.ipynb

I am curious - from a pure pipeline workflow POV, how is this conceptually unique and better? for example, for your quickstart example, i could take each one of those files and combine it into 1 large single notebook and run and have the same outcome, no? a jupyter notebook is all batch and runs sequentially too. your homepage says your main features are 1. Visual pipeline editor (sure node-based is better UX but doesn't change workflow itself) 2. Code in Notebooks (same as regular jupyter) 3. Jobs (can run ipynb with shell scripts with cron) and 4. Environments (for local usage not much need). anything major i'm missing?

Hi! Is possible to expose the streamlit service externally? My goal is to have users access the website without needing to log in

if i create job for every day on cloud do I have to keep running instance or it will start instance and run the pipeleine and then close.

Hi everyone! There's a way to integrate git with orchest using a ssh key ? Here we can't use HTTPS...