https://www.orchest.io/ logo
Docs
Join the conversationJoin Slack
Channels
announcements
be-shameless
hacktoberfest-22
introduce-yourself
random
tech-support
welcome
Powered by Linen
announcements
  • r

    Rick Lamers

    08/14/2021, 7:25 AM
    We received a question about how you can use Orchest with R and utilize the data passing features. Check out this sample GitHub project (simply import Projects -> Import Project in Orchest) to see how to use R in Orchest. https://github.com/ricklamers/orchest-pipeline-r-python-mix For the curious, it utilizes the reticulate package for Python library interop.
  • a

    Abid

    08/21/2021, 11:18 AM
    Environment Friendly Pipelines
    💯 1
    🔥 3
    r
    y
    • 3
    • 3
  • f

    Fran

    08/23/2021, 6:24 PM
    Is there a way to add a private custom image from Docker Hub?
    r
    • 2
    • 2
  • f

    Fran

    08/24/2021, 2:30 PM
    Is it possible to execute a permanent process once you start a session in a pipeline? I have a production DB behind a bastion host, so I want to use AutoSSH to set up a permanent connection to the bastion host and forward it to the production database. Is that possible right now?
    r
    • 2
    • 11
  • f

    Fran

    08/25/2021, 4:31 PM
    I've been playing a lot with Orchest lately and I gotta say I'm hooked with the potential it has. I set it up yesterday morning, build a simple pipeline to run an ETL process to get the finance team an MRR and just like that it's already providing value 🙌 I know the framework is still in its alpha version (which to me it's impressive how well it works in this stage). I feel like I'll be working more and more with this, but I have a few questions: • What's the best way to provide feedback? • Do you have a public roadmap? • Where can I read the updates? I hope I'm not asking too many questions that are very obvious to everyone 🙂
    ❤️ 1
    💯 1
    🔥 3
    r
    y
    • 3
    • 3
  • r

    Rick Lamers

    08/25/2021, 9:03 PM
    I took the liberty of implementing an example Orchest pipeline that demonstrates how to use the Orchest SDK with Julia. ➡️ https://github.com/orchest-examples/julia-orchest-sdk/tree/main If you hadn’t seen the one that demonstrates the same in R, check out: ➡️ https://github.com/orchest-examples/orchest-pipeline-r-python-mix
    🙌 1
  • f

    Fran

    08/26/2021, 2:16 PM
    Is
    orchest.output()
    the same as
    orchest.transfer.output()
    ?
    j
    r
    • 3
    • 8
  • f

    Fran

    08/27/2021, 6:33 PM
    This is how the pipeline is going so far 😄 🛠️ It extracts information from our prod database, transform it using pandas and then pushing it to a Snowflake database. Besides passing dataframes from step to step using
    orches.output()
    it also stores the information in the
    /data
    folder to be accesible by other projects. I’m planning in adding a ML step to augment this information (like churn detection or sentiment analysis). What’s been difficult for me so far is: • Moving around the “map”. As you can see the pipeline is too big for my screen and can’t move it around. • Hiding sensitive environment variables is something desired • More documentation on what’s required to set up a custom environment would be great. An example would be useful. • Turn off a session after some time to avoid wasting resources. • Documentation on how to push projects into a a GitHub repositories. It was not obvious to me, and I want to be mindful not to break anything. • Sometimes when creating a new step I can’t type the letter
    H
    in Title or File path. • Snapping ability Those things aside, it’s been really a pleasure working with it and none of the complexities I mention before have stopped me from enjoying the framework in it’s alpha stage 🙂
    r
    • 2
    • 2
  • a

    Abid

    08/30/2021, 5:23 PM
    After 3 weeks of hardwork and testing here is the complete tutorial on mlops using Orchest. It's beginner friendly too. https://pub.towardsai.net/a-new-way-of-building-machine-learning-pipelines-54700ed1aded
    👏 1
    🎉 6
    r
    • 2
    • 1
  • r

    Rick Lamers

    09/02/2021, 6:12 PM
    A cool demo pipeline I've been working on (and got contributions for from @Abid!) is a TTS pipeline that is based on an awesome open source TTS project called coqui. You can find more info in the README. But basically it generates speech from text using a state of the art deep learning model called VITS and sends a message containing the audio snippet to a Slack or Discord channel. https://github.com/ricklamers/orchest-coqui-tts Listen to the sample file to get an idea of the quality! 🎼
    speaker.wav
    🤠 3
  • a

    Abid

    09/03/2021, 4:12 AM
    I wish to contribute more in upcoming weeks.
    🙌 2
  • c

    Camilo Valenzuela

    09/07/2021, 4:00 PM
    Hi! I'm trying to run the quickstart example, and i'm getting a
    PermissionError
    when the
    get-data.py
    step is downloading the data.
    y
    r
    • 3
    • 11
  • v

    Vinay

    09/09/2021, 8:30 AM
    Hello Guys , Im getting issue while installing orchest on windows 10
  • v

    Vinay

    09/09/2021, 8:31 AM
    When I trying to setup it , It says orchest is not internal or external command. How can install it on windows 10 ?
    r
    t
    • 3
    • 8
  • v

    Vinay

    09/09/2021, 8:32 AM
    Thanks
  • p

    Philip

    09/20/2021, 7:30 PM
    Hey Guys! I want to make a theme for orchest. I have experience in wordpress theme development and have done frontend dev for some years just editing php, css and html template files. Im looking at orchest and see that it uses something called stitches. (never heard of it before now) I was wondering how to setup up my dev environment if was to make a theme. Kinda feel like a complete noob atm. 🙂
    r
    y
    • 3
    • 3
  • m

    Mahan

    09/22/2021, 7:06 AM
    Hello! I'm having trouble with the installation. Can someone help me? ./orchest install 'docker' is not accessible for the current user: 'mahan' Automatically prepending 'sudo' for 'docker' commands... [sudo] password for mahan: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. See 'docker run --help'.
    👋 1
    r
    • 2
    • 8
  • t

    Tadeo

    09/24/2021, 1:27 PM
    Hi! Has anyone implemented the streamlit service? I’m not able to install plotly 1. I installed the package in the environment but it seems to only affect the notebook, and not the streamlit service. 2. I went to Services > Edit Services > Streamlit and replaced the image to the one above (I modified the build to include streamlit and plotly) but I only managed to break the service 😒miling_face_with_tear: Can anyone tell me how to install packages for the streamlit service?
    y
    r
    • 3
    • 10
  • t

    Tadeo

    09/25/2021, 2:08 PM
    I’ve got a little :ladybug: to report: The step parameters are not being saved, and there’s weird bit where I wrote parameter and it got saved sequentially letter by letter
    Screen Recording 2021-09-25 at 15.19.39.mov
    🐛 1
    y
    r
    • 3
    • 8
  • s

    sumit upadhye

    09/25/2021, 4:28 PM
    Hi! while using vscode on Orchest cloud, I am not able to see option for running the cell.
    y
    r
    • 3
    • 5
  • e

    Eliezer de Souza da Silva

    10/01/2021, 6:56 AM
    Is there a specific channel to report issues?
    r
    • 2
    • 3
  • r

    Rick Lamers

    10/01/2021, 1:41 PM
    Hi all! - We have some cool news to share! It means we’ll be able to make Orchest even better 🥳 https://techcrunch.com/2021/10/01/orchest-raises-3-5m-to-provide-a-simpler-way-to-build-data-pipelines/
    👏 5
    🙌 5
    a
    k
    • 3
    • 2
  • k

    Kay Kayyy

    10/01/2021, 7:48 PM
    hi - congrats on your funding! i saw it and decided to try it out, however the standard installation is failing. running ec2 ubuntu and your required version of docker. after it starts via CLI i hit the webapp and the home
    /projects
    is broken stuck in loading loop. chrome console showing a bunch of 500 errors. for example
    /async/projects?session_counts=true&job_counts=true 500 (INTERNAL SERVER ERROR)
    GET <http://IP:1122/catch/api-proxy/api/sessions/> 500 (INTERNAL SERVER ERROR)
    Unable to fetch sessions Response {type: 'basic', url: '<http://IP:1122/catch/api-proxy/api/sessions/>', redirected: false, status: 500, ok: false, …}
    🙌 1
    y
    r
    • 3
    • 11
  • k

    Kay Kayyy

    10/01/2021, 9:27 PM
    when running the quickstart step 1...
    PermissionError: [Errno 13] Permission denied: '/data/cal_housing.tgz'
    where is that dir? it's not on the local machine
    r
    • 2
    • 7
  • k

    Kay Kayyy

    10/01/2021, 9:37 PM
    is there a concept of "streaming" with orchest? it seems like everything is being run in batch, correct? the use case in the quickstart would be i never want the
    get-data.py
    step 1 to stop running. it would run endlessly (keeps streaming data in) and we could program a "trigger" of some type that then would kick off step 2
    pre-processing.ipynb
    r
    • 2
    • 3
  • k

    Kay Kayyy

    10/01/2021, 9:57 PM
    is there a concept of conditional steps? the use case in the quickstart would be after step 1, only run step 2
    explore-data.ipynb
    if some condition X is true, else if condition Y is true run
    pre-processing.ipynb
    r
    • 2
    • 5
  • k

    Kay Kayyy

    10/03/2021, 7:19 PM
    I am curious - from a pure pipeline workflow POV, how is this conceptually unique and better? for example, for your quickstart example, i could take each one of those files and combine it into 1 large single notebook and run and have the same outcome, no? a jupyter notebook is all batch and runs sequentially too. your homepage says your main features are 1. Visual pipeline editor (sure node-based is better UX but doesn't change workflow itself) 2. Code in Notebooks (same as regular jupyter) 3. Jobs (can run ipynb with shell scripts with cron) and 4. Environments (for local usage not much need). anything major i'm missing?
    r
    t
    • 3
    • 12
  • t

    Tadeo

    10/05/2021, 12:08 PM
    Hi! Is possible to expose the streamlit service externally? My goal is to have users access the website without needing to log in
    r
    • 2
    • 5
  • a

    Abid

    10/05/2021, 3:34 PM
    if i create job for every day on cloud do I have to keep running instance or it will start instance and run the pipeleine and then close.
    y
    r
    • 3
    • 2
  • a

    Altieris Peixoto

    10/06/2021, 5:16 PM
    Hi everyone! There's a way to integrate git with orchest using a ssh key ? Here we can't use HTTPS...
    r
    • 2
    • 3
Powered by Linen
Title
a

Altieris Peixoto

10/06/2021, 5:16 PM
Hi everyone! There's a way to integrate git with orchest using a ssh key ? Here we can't use HTTPS...
r

Rick Lamers

10/06/2021, 5:55 PM
echo "chmod 400 /data/id_rsa && ssh-agent /bin/bash" >> ~/.profile
echo "ssh-add /data/id_rsa 2>/dev/null" >> ~/.bashrc
You can upload an SSH key to the
/data
folder in "File Manager" and add the above to the JupyterLab Config (in the general Settings -> Configure JupyterLab) view. This is sort of a temporary workaround. We're working on a nicer integration of private git repositories. Make sure to save and build to update JupyterLab
🙌 1
🚀 1
a

Altieris Peixoto

10/06/2021, 5:59 PM
uow, thank you so much @Rick Lamers!
r

Rick Lamers

10/06/2021, 8:28 PM
👍 Happy to help
View count: 1