Any Docker gotchas I should be aware of? Trying to run

./orchest install

but I’m getting stuck at pulling images. Just updated and restarted Docker too 🤔 Do I need to sign in to bypass the DockerHub rate limit?

Trying to

apt install

something in my environment, but it says I don’t have privileges. I even tried

apt install vim

like the docs suggest, but same issue. Am I missing something?

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

For those of you who haven't read it yet, we wrote a bit about why we believe Notebooks are useful in the context of data science. https://hackernoon.com/why-jupyter-notebooks-are-the-future-of-data-science-e7133uv7 If you have any ideas or opinions about the use of Notebooks would love to open a bit of discussion 🙂!

A new release! v0.8.0 We introduce the concept of Environment variables. Read more about in the release notes below:

This release introduces the concept of Environment variables 📇 to Orchest! With its introduction we're deprecating and removing Data sources.

We've been testing Orchest and listening to feedback 📣. As a result we discovered everybody handles data sources slightly differently in their workflow. We want Orchest to be helpful, but allow for flexibility in for example the choice of the data source connector (e.g. Python offers various Postgres database connectors).

Environment variables allow for simple and effective secret passing (e.g. hostnames, passwords, usernames) to make it easy to connect and configure your favorite data sources.

In addition, this release includes many UI/UX improvements. Such as a dedicated project settings pane, and a cleaned up ✨ job pane.

https://github.com/orchest/orchest/releases/tag/v0.8.0

Hi community, our friends at Data Council have asked to share their OSS survey. Feel free to share your opinion (and potentially give Orchest some love!) https://twitter.com/DataCouncilAI/status/1359905254205247491

Hello all, Thank you for the recent updates! After upgrading from 0.6.4 to 0.8.0 (yup, I got distracted), I am getting an error when switching to JupyterLab:

Error attempting to connect to Gateway server url '<http://jupyter-EG-someid:8888>'.  Ensure gateway url is valid and the Gateway instance is running.

I can run the scripts from the pipeline view, and I have successfully tried

curl <http://jupyter-EG-someid:8888>

from orchest-api container to check connectivity. Do you have pointers to help me out?

Hello everyone! I'm trying to get started with this amazing project in a volunteer job, which I participate in. And I'd also like to congrats every contributor because the project is getting increadible! Well, I'm trying to install the Orchest in a Ubuntu 20.04 Amazon EC2. First, I cloned the repo and run

./orchest install

,

./orchest update

and

./orchest start

. But it always return this message for me:

Before starting Orchest, make sure Orchest is installed. Run:
	orchest install

I repeated the cycle many times, installing it again but it always returns the same massage. However, when I run

./orchest version

it returns to me the version v0.8.0. I'd appreciate a lot if someone can help me with this. Below is the execution cycle in my terminal.

ubuntu@ip-:~/orchest$ ./orchest start
Before starting Orchest, make sure Orchest is installed. Run:
	orchest install

ubuntu@ip-172-31-53-217:~/orchest$ ./orchest install
Some images have been pulled before. Don't forget to run:
	orchest update
after the installation is finished to ensure that all images are running
the same version of Orchest.
Installing Orchest...
Pulling images: 6/6|###################################################|

ubuntu@ip-:~/orchest$ ./orchest update
latest: Pulling from orchest/orchest-ctl
Digest: sha256:9c6a3beb21f4fbb6f244881ec6643bfd07306e18c86729b9ee4fe385f5cf0af3
Status: Image is up to date for orchest/orchest-ctl:latest
<http://docker.io/orchest/orchest-ctl:latest|docker.io/orchest/orchest-ctl:latest>
Updating...
From <https://github.com/orchest/orchest>
 * branch              HEAD       -> FETCH_HEAD
Already up to date.
HEAD is up to date.
From <https://github.com/orchest/orchest>
 * branch              HEAD       -> FETCH_HEAD
Pulling images: 8/8|###################################################|
Don't forget to restart Orchest for the changes to take effect:
	orchest restart

ubuntu@ip-:~/orchest$ ./orchest start
Before starting Orchest, make sure Orchest is installed. Run:
	orchest install

ubuntu@ip-:~/orchest$ ./orchest version
Orchest version: v0.8.0

Hi everyone, a new release is waiting for you all! v0.9.0 Most importantly, we finally have proper support for URL based navigation in Orchest. Read all about in the release notes:

We are very proud of this release and we call it: The Big One 🔥.

This release introduces a massive new feature to Orchest: full URL deep linking support. Easy back/forward browser navigation to open the right views in your browser. Reloading the URL and directly opening the right view. Furthermore, environment build dependent actions (e.g. running a job, or interactive pipeline editing) are more automated: no more hard choices to make when builds complete.

In addition, we now support running an exhaustive debug dump using the simple 

orchest debug

 command. This will make troubleshooting much easier for both the users & developers. Many tests were added to the 

orchest-api

 test suite for much improved reliability thanks mainly to @fruttasecca.

Finally, many smaller bug fixes and performance improvements were added. Have fun building in Orchest!

https://github.com/orchest/orchest/releases/tag/v0.9.0

Hi everyone! Another week, another release. This time it's a pretty big overhaul of the user interface of Orchest. The result is a more intuitive and consistent user experience with minimal visual distraction. More room for data, plots, and code! Check it out by running

./orchest update

or update directly from the web UI settings pane. P.S. you might have noticed we switched to CalVer (calver.org). This sets better expectations about what the version means, and doesn't mean. P.P.S. you might not have seen it yet, but we now have full support for JupyterLab extensions, including server extensions. Test it out, and let us know which extensions are you favorite. We are big fans of https://pypi.org/project/jupyterlab-git/!

Hi all! We added a cool new feature for your GitHub repos that contain Orchest pipelines. ✅ You can now add a dynamically updating image of your pipeline to your markdown files! All you have to do is add this line to a README.md:

![My pipeline](<https://pviz.orchest.io/?pipeline=https://github.com/orchest/quickstart/blob/main/california_housing.orchest|https://pviz.orchest.io/?pipeline=https://github.com/orchest/quickstart/blob/main/california_housing.orchest>)

Check it out in our quickstart repo https://github.com/orchest/quickstart

Hi

Another release! Happy data coding everybody 🥳

This release brings a UI refactor for managing your pipeline sessions. We simplify the session management between the JupyterLab and Pipeline view by moving it to the top header bar.

In addition, we have a number of quality of life, performance and bug fixes. Enjoy!

https://github.com/orchest/orchest/releases/tag/v2021.03.6

Hello, how does Orchest compare to Dagaster/Prefect/MLFlow?

This repo contains an Orchest pipeline that features the workshop notebooks for the Ray Core workshop that will be held later today: https://github.com/ricklamers/ray_tutorial_orchest Fork of: https://github.com/DerwenAI/ray_tutorial Workshop: https://anyscale.zoom.us/j/98407189275 https://www.eventbrite.com/e/anyscale-academy-ray-core-hands-on-tutorial-tickets-146678245581

Hi everyone! We're super excited to announce that today we completed final testing of the first iteration of Orchest Cloud. A fully scalable architecture based on AWS for running independent Orchest instances at your command. We're opening up priority access to the Orchest Cloud private beta for members of this Slack community. If you're interested sign up here and shoot me a message so I can prioritize your onboarding.

Hi, joined as I’m interested in the development progress

Hey y'all! I created a couple of videos to explain some of the core concepts in Orchest. Check them out here: https://www.orchest.io/knowledge-base

Find out the latest about the data ecosystem & Orchest in Data Overload ⚡ Vol. 4! https://mailchi.mp/a471def8d2eb/orchest-presents-data-overload-vol-6527425

This new release brings an exciting new feature to Orchest: real-time collaboration 🤝 in JupyterLab. It's made possible by the awesome work from the Jupyter core team and yjs.

In addition we have done some internal refactoring to improve the extensibility of the codebase, and as a result there are numerous improvements to stability and performance.

We can't wait to see you build more awesome data pipelines with this release!

https://github.com/orchest/orchest/releases/tag/v2021.05.0 Hi all! Today we have a new release. Check it out if you want to see "spooky action at a distance" (real-time colab in JupyterLab baked into Orchest that is).

./orchest update

Should be all you need. Or use the web updater in Orchest itself (Global settings -> Update).

We just crossed 1K stars on GitHub 🤩 Thanks everyone for your enthusiasm about what we’re building! Even though GitHub stars is just one metric, we’re glad the community is growing. https://github.com/orchest/orchest If you ever feel like contributing a small piece of code, just reach out to me and I’ll gladly assist you in the process 🤓

I am seeing an issue while running "pnpm run dev".. I posted on git issue but thats not the right place. I figured the slack channel today. Your help is appreciated.

Quick announcement: a big new feature is coming up. It's called: Services. For the impatient: the alpha release is currently available on this development branch. More details will follow! Yes, you're seeing it right. That's VS Code, TensorBoard, and Streamlit running directly in Orchest. 🤯

Super exciting news! Services are now available on the latest stable release. Read more about this new feature 👇

📣 This release introduces a major new feature to Orchest: Services ✨

Services allow you to easily integrate your favorite data science applications without any hassle. Simply use one of the existing service templates to add to your Orchest pipeline:

- TensorBoard

- Streamlit

- PostgreSQL

- Redis

- and even VS Code!

After adding a service to a pipeline it will show up in your pipeline editor (see attached image).

If you want to start exploring services, simply import the 

<https://github.com/ricklamers/orchest-language-classifier>

 project to Orchest. Make sure you upgrade Orchest to the latest version by running: 

./orchest update

.

We can't wait to see what ideas you come up with 🤩!

You can find more details in the Services docs, including how to set up your own custom services.

https://github.com/orchest/orchest/releases/tag/v2021.06.0

I have upgraded to the latest version, but I still get the following error

Hi everyone! Our most stable and performant release to date just went up: https://github.com/orchest/orchest/releases/tag/v2021.06.6 Update by running

./orchest update

or update through the web UI. One of the core improvements is the conversion of the job scheduler to a managed daemon process to support auto restarting of the worker processes. ➡️ In short: more robust job execution 🔥

Hello! I am new here. Thanks for all the work you guys are doing. Any plans to integrate this with Google Colab?

Hi! Glad to be here! 👋 I'm looking to sync Python scripts/notebooks with GCP or AWS. Can Orchest help?

Hey guys, Im trying to run orchest on AWS i did docker installation and everything seems to be fine but can't start orchest for some reason