https://www.orchest.io/ logo
Docs
Join the conversationJoin Slack
Channels
announcements
be-shameless
hacktoberfest-22
introduce-yourself
random
tech-support
welcome
Powered by Linen
announcements
  • v

    Vivan

    12/16/2020, 9:04 AM
    sorry my bad PR. did some minor changes to the documentation
  • r

    Rick Lamers

    12/16/2020, 9:05 AM
    Will gladly review
  • v

    Vivan

    12/16/2020, 9:07 AM
    thanks rick.
  • v

    Vivan

    12/16/2020, 9:07 AM
    i got in touch with you on that tweet of orchest.
  • v

    Vivan

    12/16/2020, 9:07 AM
    really looking forward to interact more and get insights on your project work.
    🙏 1
    💯 2
    ❤️ 1
  • m

    Médéric Descazeaux

    12/17/2020, 3:24 PM
    Hello! I am getting an error I cannot explain while passing data. Here is the output call for "Load current logs":
    orchest.output({'df': df}, name="current")
    And for "Process data":
    orchest.output({'df': df,
                    'processedlist': processedlist,
                    'ignoredlist': ignoredlist,
                    'messageslist': messageslist},
                   name='processed')
    I get the following error:
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-3-b9a2b7cd25cd> in <module>
          5 print("Getting current and pending logs from Orchest pipeline...", end=" ")
          6 
    ----> 7 data = orchest.get_inputs()
          8 
          9 display(data)
    
    /orchest/orchest-sdk/python/orchest/transfer.py in get_inputs(ignore_failure, verbose)
        859         parent_uuid = parent.properties["uuid"]
        860         get_output_method, args, kwargs, metadata = _resolve(
    --> 861             parent_uuid, consumer=step_uuid
        862         )
        863 
    
    /orchest/orchest-sdk/python/orchest/transfer.py in _resolve(step_uuid, consumer)
        753                 method_info = method(step_uuid, consumer=consumer)
        754             else:
    --> 755                 method_info = method(step_uuid)
        756 
        757         except OutputNotFoundError:
    
    /orchest/orchest-sdk/python/orchest/transfer.py in _resolve_disk(step_uuid)
        349         with open(head_file, "r") as f:
        350             timestamp, serialization, name = f.read().split(
    --> 351                 Config.__METADATA_SEPARATOR__
        352             )
        353 
    
    ValueError: not enough values to unpack (expected 3, got 1)
    ValueError: not enough values to unpack (expected 3, got 1)
    The error only happens if the "Load current logs" block is connected to the "Merge logs" one.
    r
    j
    • 3
    • 12
  • v

    Vivan

    12/17/2020, 3:29 PM
    i think the there is a specific function which is changed to return only and the other to not return. that might be causing the problem.
    r
    • 2
    • 2
  • j

    James Read

    12/22/2020, 12:26 PM
    Hey folks. Saw your post from /r/SelfHosted, and very keen to give it a whirl. Looks like the local "installer" and whatnot is very docker-focused at the moment. Any support for podman/kubernetes available yet?
  • r

    Rick Lamers

    12/22/2020, 12:27 PM
    @James Read awesome! At the moment it's Docker only. We want to be more agnostic in the future though.
  • j

    James Read

    12/22/2020, 12:40 PM
    Okay. I'm quite keen to try it out, but on Fedora I've had to disable cgroups and SELinux so far to start the "installer" - to pull images FYI.
  • r

    Rick Lamers

    12/22/2020, 12:44 PM
    Some of the containers require access to the Docker sock in order to dynamically create containers with the appropriate mounts w.r.t. the host system. We've mostly focused on Debian based systems, and having the launching user be in the
    docker
    group is sufficient. Alternatively, the user can invoke
    ./orchest <command>
    commands where
    sudo
    is used when we detect the user has no access to the Docker
    .sock
    . We're very open to improving this to be more compatible with e.g. Fedora Linux and security features like SELinux. Would gladly pick your brain sometime if you see any particular areas for improvement.
  • j

    James Read

    12/22/2020, 12:45 PM
    Yeah that's fine. Found all the interesting stuff in orchest-ctl/app/config to see what's actually done.
  • j

    James Read

    12/22/2020, 12:46 PM
    Just looking for the code that starts/stop images - is that abstracted enough so that a "kubernetes driver" could be written?
  • r

    Rick Lamers

    12/22/2020, 12:53 PM
    We should be able to eventually abstract out the Docker specific calls, it's mostly start/stop and mounting. But for features like environments, we also have some Docker 'image building' specific code (in orchest-api), in addition to various image/container cleanup and networking. Furthermore, the Jupyter Enterprise Gateway would also need to support the Kubernetes backend - which it has support for - but our integration with it should be made compatible (and the various language specific container images - such as the Julia kernel image).
  • j

    James Read

    12/22/2020, 12:54 PM
    Sure. I'll poke around with it at some point when I have a bit more time 🙂 Thanks.
    👍 1
  • r

    Rick Lamers

    12/23/2020, 6:26 PM
    Hi everyone! We just released another big version. This time, most of the changes are under the hood. But it should improve performance by a lot across the board. A quick overview: • Rework of permissions across containers for the 
    userdir/
     - massive increase in performance (no more daemon ). • Individual SQLite databases merged into central Postgres service - improved concurrency across services. • Introduced migrations for schema changes to improve updating experience. • Improved Orchest SDK robustness in error handling. • Various small fixes and changes. Update to v0.4.1 Because of the changes, some of you might experience permission issues with the userdir/. To fix them please run the following command (from the
    orchest/
    repo directory):
    sudo chown -R $USER:$USER userdir/
    sudo chmod -R g+rwxs userdir/
    Happy holidays everyone! 🎄
    🙌 1
  • r

    Rick Lamers

    01/08/2021, 8:51 PM
    Awesome contrib from https://github.com/obulat: https://github.com/orchest/orchest/pull/108 We now have proper backspace + delete keyboard support. How convenient!
    😀 1
  • f

    Foren Power

    01/13/2021, 10:12 PM
    Hey Rick and Orchest team, very cool project finally getting a little time to dive in. I'm having trouble moving around the pipeline. I can zoom in and out just fine, but I can't scroll like on a maps app. Is there a way to move around an Orchest pipeline?
  • r

    Rick Lamers

    01/13/2021, 10:24 PM
    Holding the space bar should do the trick.
    🙌 1
    f
    • 2
    • 2
  • r

    Rick Lamers

    01/13/2021, 10:24 PM
    Let me know if that doesn’t work 🤓.
  • r

    Rick Lamers

    01/14/2021, 2:07 PM
    Hi everyone, we’re considering refactoring data sources to become environment variables. Our intuition has been that it’s already mostly functioning as a credential store. But we’re glad to get input on this. Reply to this thread if you have thoughts to share 🙂
    f
    • 2
    • 1
  • h

    howie hu

    01/15/2021, 12:51 AM
    @Rick Lamers Hi, https://github.com/orchest/orchest/issues/110
  • h

    howie hu

    01/15/2021, 12:51 AM
    Do you have any thoughts on this issue?
  • h

    howie hu

    01/15/2021, 1:05 AM
    In my opinion, the advantage of workflow is that any script can be freely combined. Now that the freedom of scripts is limited to the project, can you consider letting the project developers decide which scripts can be shared with others’ projects? @Rick Lamers
    r
    • 2
    • 5
  • h

    howie hu

    01/16/2021, 2:19 PM
    @Rick Lamers Hi, is there any documentation about building custom images, I need to build a python 3.6 base-kernel.
  • h

    howie hu

    01/16/2021, 2:23 PM
    My current build method is based on build_container.sh. This method is a bit cumbersome. Is there an easier way?
    r
    y
    • 3
    • 24
  • f

    Foren Power

    01/20/2021, 12:50 AM
    When it's not a bad time, the most challenging problem in machine learning that I've seen. How to handle multiple dependent variables with Orchest. In-case anyone wants to take this impossible challenge, I added 15 second sample MOBA video game win/loss/tie ratio optimization. VariablesToGuess sheet = Dependent Variables KnownVariables sheet = Independent Variables Not that we have to use this specific example. I just made it because I wanted it to be a little more fun for you.
    2021_01_20_006_MOBA_guess.xls
    r
    • 2
    • 3
  • v

    Vivan

    01/22/2021, 6:34 AM
    this will be a little irrelevant, but i need to address it here, as there is problem im facing with orchest docker setup. i tried installing dockers for orchest to run, then while setting it up, there is an option to setup WSL 2 for working of orchest without a trouble, which links you to the microsoft store. now while i was setting up orchest through dockers and when it gave me the option to choose the linux distro for the application to run, docker crashed my microsoft store, now i can't access it. it crashed while setting up dockers for orchest so is there any way i could solve it or any issue i might be missing ? please help!
    r
    f
    • 3
    • 9
  • m

    Michael Duncan

    01/29/2021, 6:40 PM
    Hi everyone! I'm trying to get started with this cool looking project but the UI doesn't seem to update. I've cloned the repo and run
    ./orchest install
    ./orchest update
    and
    ./orchest start
    . When I try to create a project nothing shows up in the UI and I can't select the default
    Project
    . The projects are being made and put into the
    userdir/projects
    directory but I can't interact with them. Any tips would be greatly appreciated
    y
    r
    j
    • 4
    • 70
  • r

    Rick Lamers

    02/02/2021, 1:01 PM
    Hi all, we have another exciting release. This time it includes a feature proposed by @howie hu. In addition we added JupyterLab config persistence, to make sure you can safely customize your code/notebook editor to your liking. Here's the release note:
    This release brings some great quality of life 💎 features!
    For those of you that prefer configuring your IDEs to your liking (🕶️ dark mode anyone?) - with this release we make JupyterLab configuration persistent. In addition to user config (settings you modify in the JupyterLab Settings pane) we also make client side JupyterLab extensions persistent. Support for server side extensions will be added later.
    @howie6879 pointed out that it would be nice to have pipeline parameters. It would alleviate the need to repeat yourself in pipeline steps for global pipeline level parameterization. Hence, you can now configure pipeline parameters 🥳. Thanks @howie6879! Pipeline parameters can also be overridden for scheduled jobs.
    In addition, some minor logging, performance, and UI improvements were made.
    https://github.com/orchest/orchest/releases/tag/v0.7.0
    🙌 1
Powered by Linen
Title
r

Rick Lamers

02/02/2021, 1:01 PM
Hi all, we have another exciting release. This time it includes a feature proposed by @howie hu. In addition we added JupyterLab config persistence, to make sure you can safely customize your code/notebook editor to your liking. Here's the release note:
This release brings some great quality of life 💎 features!
For those of you that prefer configuring your IDEs to your liking (🕶️ dark mode anyone?) - with this release we make JupyterLab configuration persistent. In addition to user config (settings you modify in the JupyterLab Settings pane) we also make client side JupyterLab extensions persistent. Support for server side extensions will be added later.
@howie6879 pointed out that it would be nice to have pipeline parameters. It would alleviate the need to repeat yourself in pipeline steps for global pipeline level parameterization. Hence, you can now configure pipeline parameters 🥳. Thanks @howie6879! Pipeline parameters can also be overridden for scheduled jobs.
In addition, some minor logging, performance, and UI improvements were made.
https://github.com/orchest/orchest/releases/tag/v0.7.0
🙌 1
View count: 1