sorry my bad PR. did some minor changes to the documentation

Will gladly review

thanks rick.

i got in touch with you on that tweet of orchest.

really looking forward to interact more and get insights on your project work.

Hello! I am getting an error I cannot explain while passing data. Here is the output call for "Load current logs":

orchest.output({'df': df}, name="current")

And for "Process data":

orchest.output({'df': df,
                'processedlist': processedlist,
                'ignoredlist': ignoredlist,
                'messageslist': messageslist},
               name='processed')

I get the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-b9a2b7cd25cd> in <module>
      5 print("Getting current and pending logs from Orchest pipeline...", end=" ")
      6 
----> 7 data = orchest.get_inputs()
      8 
      9 display(data)

/orchest/orchest-sdk/python/orchest/transfer.py in get_inputs(ignore_failure, verbose)
    859         parent_uuid = parent.properties["uuid"]
    860         get_output_method, args, kwargs, metadata = _resolve(
--> 861             parent_uuid, consumer=step_uuid
    862         )
    863 

/orchest/orchest-sdk/python/orchest/transfer.py in _resolve(step_uuid, consumer)
    753                 method_info = method(step_uuid, consumer=consumer)
    754             else:
--> 755                 method_info = method(step_uuid)
    756 
    757         except OutputNotFoundError:

/orchest/orchest-sdk/python/orchest/transfer.py in _resolve_disk(step_uuid)
    349         with open(head_file, "r") as f:
    350             timestamp, serialization, name = f.read().split(
--> 351                 Config.__METADATA_SEPARATOR__
    352             )
    353 

ValueError: not enough values to unpack (expected 3, got 1)
ValueError: not enough values to unpack (expected 3, got 1)

The error only happens if the "Load current logs" block is connected to the "Merge logs" one.

i think the there is a specific function which is changed to return only and the other to not return. that might be causing the problem.

Hey folks. Saw your post from /r/SelfHosted, and very keen to give it a whirl. Looks like the local "installer" and whatnot is very docker-focused at the moment. Any support for podman/kubernetes available yet?

@James Read awesome! At the moment it's Docker only. We want to be more agnostic in the future though.

Okay. I'm quite keen to try it out, but on Fedora I've had to disable cgroups and SELinux so far to start the "installer" - to pull images FYI.

Some of the containers require access to the Docker sock in order to dynamically create containers with the appropriate mounts w.r.t. the host system. We've mostly focused on Debian based systems, and having the launching user be in the

docker

group is sufficient. Alternatively, the user can invoke

./orchest <command>

commands where

sudo

is used when we detect the user has no access to the Docker

.sock

. We're very open to improving this to be more compatible with e.g. Fedora Linux and security features like SELinux. Would gladly pick your brain sometime if you see any particular areas for improvement.

Yeah that's fine. Found all the interesting stuff in orchest-ctl/app/config to see what's actually done.

Just looking for the code that starts/stop images - is that abstracted enough so that a "kubernetes driver" could be written?

We should be able to eventually abstract out the Docker specific calls, it's mostly start/stop and mounting. But for features like environments, we also have some Docker 'image building' specific code (in orchest-api), in addition to various image/container cleanup and networking. Furthermore, the Jupyter Enterprise Gateway would also need to support the Kubernetes backend - which it has support for - but our integration with it should be made compatible (and the various language specific container images - such as the Julia kernel image).

Sure. I'll poke around with it at some point when I have a bit more time 🙂 Thanks.

Hi everyone! We just released another big version. This time, most of the changes are under the hood. But it should improve performance by a lot across the board. A quick overview: • Rework of permissions across containers for the 

userdir/

 - massive increase in performance (no more daemon ). • Individual SQLite databases merged into central Postgres service - improved concurrency across services. • Introduced migrations for schema changes to improve updating experience. • Improved Orchest SDK robustness in error handling. • Various small fixes and changes. Update to v0.4.1 Because of the changes, some of you might experience permission issues with the userdir/. To fix them please run the following command (from the

orchest/

repo directory):

sudo chown -R $USER:$USER userdir/
sudo chmod -R g+rwxs userdir/

Happy holidays everyone! 🎄

Awesome contrib from https://github.com/obulat: https://github.com/orchest/orchest/pull/108 We now have proper backspace + delete keyboard support. How convenient!

Hey Rick and Orchest team, very cool project finally getting a little time to dive in. I'm having trouble moving around the pipeline. I can zoom in and out just fine, but I can't scroll like on a maps app. Is there a way to move around an Orchest pipeline?

Holding the space bar should do the trick.

Let me know if that doesn’t work 🤓.

Hi everyone, we’re considering refactoring data sources to become environment variables. Our intuition has been that it’s already mostly functioning as a credential store. But we’re glad to get input on this. Reply to this thread if you have thoughts to share 🙂

@Rick Lamers Hi, https://github.com/orchest/orchest/issues/110

Do you have any thoughts on this issue?

In my opinion, the advantage of workflow is that any script can be freely combined. Now that the freedom of scripts is limited to the project, can you consider letting the project developers decide which scripts can be shared with others’ projects? @Rick Lamers

@Rick Lamers Hi, is there any documentation about building custom images, I need to build a python 3.6 base-kernel.

My current build method is based on build_container.sh. This method is a bit cumbersome. Is there an easier way?

When it's not a bad time, the most challenging problem in machine learning that I've seen. How to handle multiple dependent variables with Orchest. In-case anyone wants to take this impossible challenge, I added 15 second sample MOBA video game win/loss/tie ratio optimization. VariablesToGuess sheet = Dependent Variables KnownVariables sheet = Independent Variables Not that we have to use this specific example. I just made it because I wanted it to be a little more fun for you.

this will be a little irrelevant, but i need to address it here, as there is problem im facing with orchest docker setup. i tried installing dockers for orchest to run, then while setting it up, there is an option to setup WSL 2 for working of orchest without a trouble, which links you to the microsoft store. now while i was setting up orchest through dockers and when it gave me the option to choose the linux distro for the application to run, docker crashed my microsoft store, now i can't access it. it crashed while setting up dockers for orchest so is there any way i could solve it or any issue i might be missing ? please help!

Hi everyone! I'm trying to get started with this cool looking project but the UI doesn't seem to update. I've cloned the repo and run

./orchest install

./orchest update

and

./orchest start

. When I try to create a project nothing shows up in the UI and I can't select the default

Project

. The projects are being made and put into the

userdir/projects

directory but I can't interact with them. Any tips would be greatly appreciated

Hi all, we have another exciting release. This time it includes a feature proposed by @howie hu. In addition we added JupyterLab config persistence, to make sure you can safely customize your code/notebook editor to your liking. Here's the release note:

This release brings some great quality of life 💎 features!

For those of you that prefer configuring your IDEs to your liking (🕶️ dark mode anyone?) - with this release we make JupyterLab configuration persistent. In addition to user config (settings you modify in the JupyterLab Settings pane) we also make client side JupyterLab extensions persistent. Support for server side extensions will be added later.

@howie6879 pointed out that it would be nice to have pipeline parameters. It would alleviate the need to repeat yourself in pipeline steps for global pipeline level parameterization. Hence, you can now configure pipeline parameters 🥳. Thanks @howie6879! Pipeline parameters can also be overridden for scheduled jobs.

In addition, some minor logging, performance, and UI improvements were made.

https://github.com/orchest/orchest/releases/tag/v0.7.0