Release - v2022.08.7 New release published by yannickperrenet What's Changed Improvements 👷 • Add endpoint to trigger a job run "manually" by @fruttasecca in #1186 • Fix: peer dependency issue when building web server by @mausworks in #1192 Full Changelog: v2022.08.6...v2022.08.7 orchest/orchest

Guys, it is possible to change the

orchest-webserver

(or other ones) image to use a custom one? I know that there is a property

orchestWebServer

in the orchest-controller, but I'm not sure how is the best way to change this value.

Hi guys, I work with @Allan Sene. I tried to send these parameters to EKS cluster using kubectl, but without success.

apiVersion: <http://orchest.io/v1alpha1|orchest.io/v1alpha1>
kind: OrchestCluster
metadata:
  name: cluster-1
  namespace: orchest
spec:
  singleNode: true
  orchest:
    orchestWebServer:
      image: alexsanderdadosfera/orchest-webserver:latest

How to send a custom image to Orchest?

Hi guys, is it possible to use other kubternets distributions instead of minikube or microk8s like k3s or k3d?

if you have an instance created before July 5th this year, it's probably running a very old version of Orchest (specifically

v2022.03.6

). the good news is that you can now migrate it to the latest version! To perform the migration, go to your Cloud dashboard, click on the "..." three dots to the right of the instance, and select "Migrate to the latest version". After the migration, the upgrades will continue working as usual and you will be able to enjoy the new File Manager, the improved Pipeline Editor view, the context menu, and much more! check out your Cloud Dashboard 😮rchest:

i have an issue starting orchest

orchest: command not found

and do we have a guided document to setup ochest on AWS fragate

Release - v2022.08.8 New release published by yannickperrenet What's Changed Improvements 👷 • Feature: Generate parameter forms from jsonschema by @ricklamers in #1166 • Add instructions on setting up a reverse proxy by @astrojuanlu in #1200 • Change docs to PyData theme by @ricklamers in #1202 • Allow using custom image for orchest components by @nhaghighat in #1205 Full Changelog: v2022.08.7...v2022.08.8 orchest/orchest

Hello, Is there a way to trigger a pipeline via webhook? Ive not started playing with this yet but a key requirement of what I need is to listen to be able to listen to webhooks and trigger a pipeline to do something when a webhook is recieved.

New blog post! 🔥 Make the most of your Google Analytics data with Orchest and Meltano 📈 https://orchest.io/blog/google-analytics-data-with-orchest-and-meltano

quick newbie question: this is my first time looking at a distributed/kubernetes-based codebase. When working on the code of the different services, do you create a python virtualenv for each service?

@Robert Sun has joined the channel

I've been trying to setup orchest on microk8s - multi node, with NFS storage backing. I'm getting to one part that is really flaky, doesn't seem to work - and it's the environment building step.

Starting worker...
Starting image build.............
Copying context.....There was a problem building the image. The building script had a non 0 exit code, build failed.

The logs from the image-build-task container are :

init time="2022-08-24T22:11:21.922Z" level=info msg="Starting Workflow Executor" executorType │
│ init time="2022-08-24T22:11:21.924Z" level=info msg="Creating a emissary executor"            │
│ init time="2022-08-24T22:11:21.924Z" level=info msg="Executor initialized" deadline="0001-01- │
│ init time="2022-08-24T22:11:21.952Z" level=info msg="Start loading input artifacts..."        │
│ init time="2022-08-24T22:11:21.952Z" level=info msg="Alloc=4910 TotalAlloc=9295 Sys=74065 Num │
│ main STEP 1/6: FROM <http://docker.io/orchest/base-kernel-py:v2022.08.8|docker.io/orchest/base-kernel-py:v2022.08.8>                     │
│ main STEP 2/6: LABEL _orchest_project_uuid=f5f62dca-b5ad-40e3-a522-a29220d966f2               │
│ main --> Using cache 438dc990cbb2581f4d42e4c852b0630acc0ee6c016865ea762e37eb332af237b         │
│ main --> 438dc990cbb                                                                          │
│ main STEP 3/6: LABEL _orchest_environment_uuid=1bfd8c08-5b24-4a66-9a84-a50dd5e16652           │
│ main --> Using cache 823aad6fe66e4bcfba5da3946e8acae6a8eb36df47922c7eb49b46cc7461e816         │
│ main --> 823aad6fe66                                                                          │
│ main STEP 4/6: WORKDIR /project-dir                                                           │
│ main --> Using cache 1593801dc92911eadd946ad8cf53027fe64a7d893896a251bd70307b6f49bd0a         │
│ main --> 1593801dc92                                                                          │
│ main STEP 5/6: COPY . .                                                                       │
│ main error committing container for step {Env:[PATH=/opt/conda/bin:/usr/local/sbin:/usr/local │
│ main Error: exit status 125                                                                   │
│ Stream closed EOF for orchest/image-build-task-4a159dc2-a4b3-4050-816e-976dfbe7858e (init)    │
│ Stream closed EOF for orchest/image-build-task-4a159dc2-a4b3-4050-816e-976dfbe7858e (main)    │
│ wait time="2022-08-24T22:11:22.437Z" level=info msg="Starting Workflow Executor" executorType │
│ wait time="2022-08-24T22:11:22.440Z" level=info msg="Creating a emissary executor"            │
│ wait time="2022-08-24T22:11:22.440Z" level=info msg="Executor initialized" deadline="0001-01- │
│ wait time="2022-08-24T22:11:22.440Z" level=info msg="Starting deadline monitor"               │
│ wait time="2022-08-24T22:11:26.440Z" level=info msg="Main container completed"                │
│ wait time="2022-08-24T22:11:26.441Z" level=info msg="No Script output reference in workflow.  │
│ wait time="2022-08-24T22:11:26.441Z" level=info msg="No output parameters"                    │
│ wait time="2022-08-24T22:11:26.441Z" level=info msg="No output artifacts"                     │
│ wait time="2022-08-24T22:11:26.441Z" level=info msg="Killing sidecars []"                     │
│ wait time="2022-08-24T22:11:26.441Z" level=info msg="Alloc=4742 TotalAlloc=9333 Sys=74065 Num │
│ Stream closed EOF for orchest/image-build-task-4a159dc2-a4b3-4050-816e-976dfbe7858e (wait)

Has anyone run into this? Any help or clue as to where the configuration is wrong?

@Tech Account has joined the channel

Hi, I was interested in finding out if Orchest provided any form of API integration.

For example triggering a pipeline

Release - v2022.08.9 New release published by yannickperrenet What's Changed Improvements 👷 • Fix unable to save pipeline parameters when clicking outside of the field. by @iannbing in #1234 Full Changelog: v2022.08.8...v2022.08.9 orchest/orchest

Release - v2022.08.10 New release published by yannickperrenet What's Changed Improvements 👷 • Disable builder cache by @fruttasecca in #1238 Full Changelog: v2022.08.9...v2022.08.10 orchest/orchest

Any way to use a Postgres in RDS? We use orchest on EKS. If yes, how do I pass this in yml?

Release - v2022.08.11 New release published by yannickperrenet Full Changelog: v2022.08.10...v2022.08.11 orchest/orchest

⚠️ Quick PSA: if you're on v2022.08.10 upgrade to v2022.08.11 since v2022.08.10 contains a pod scheduling bug that will prevent container starts for crucial behaviors like pipeline sessions.

@Nelson Silva has joined the channel

<!channel> Check out Data Overload Vol. 13, the latest edition of our monthly newsletter! ⚡ 🎨 Big changes are coming to the Orchest UI 💞 We are having our first Community Event 🔄 Manually trigger jobs 🏗️ Generate parameter forms from a JSON schema ...and lots more! Read the full post online 😮rchest: Do you want to receive the next edition directly in your inbox? Subscribe here!

@Krishna Sangeeth KS has joined the channel

Hey Folks, Thanks for creating this project. It looks very interesting and i am excited to try it out. I am trying out orchest for the first time today and I have gone for the self hosted mode of setting up in K8s. I have used the

orchest install

option and installation was clean. I am trying to run the sample pipeline from

global-key-value

. It fails for some reason and i couldn't find any logs associated with it. In the k8s namespace, i briefly saw this pod coming up and then i think it got cleaned up. Any pointers on how to debug this further ?

pipeline-run-task-b9ce091c-baad-4477-915a-77f46dc4c192-12230460   0/2     Init:1/2   0          18s

New blog post! 🔥 Automate your anomaly detection pipelines with Orchest and Clarify https://orchest.io/blog/automate-your-anomaly-detection-pipelines-with-orchest-and-clarify You can download the Orchest project and inspect the notebooks here https://github.com/astrojuanlu/orchest-timeseries-clarify happy hacking!

Good afternoon, I tried to make a backup of the EKS cluster using Velero (https://velero.io/). It turns out that when restoring the backup all containers went up normally, except for Orchest. Could you help me understand? I'm trying to implement a backup routine here at the Dadosfera company. Orchest Controller has failed: