https://www.orchest.io/ logo
Docs
Join the conversationJoin Slack
Channels
announcements
be-shameless
hacktoberfest-22
introduce-yourself
random
tech-support
welcome
Powered by Linen
announcements
  • a

    Altieris Peixoto

    10/08/2021, 6:04 PM
    Hi everyone! I've increased the
    MAX_JOB_RUNS_PARALLELISM =3
    but i cannot run parallel pipelines using web tool yet. There's another configuration i should do?
    y
    r
    • 3
    • 13
  • j

    juanlu

    10/14/2021, 12:23 PM
    the UI won't let me finish the initial survey 🙂
    Screencast 2021-10-14 14:23:13.mp4
    👀 1
    r
    • 2
    • 3
  • r

    Richard Pelgrim

    10/15/2021, 2:54 PM
    perhaps a known issue, but it seems https://cloud.orchest.io/onboarding doesn't work on Brave? switching to Chrome solved it for me [EDIT] I spoke too soon 🙊 the main page loads on chrome but nothing happens when I click on "New Instance". Is this website down?
    r
    • 2
    • 5
  • a

    Abid

    10/19/2021, 2:02 PM
    A Guide to Machine Learning Pipelines and Orchest https://www.analyticsvidhya.com/blog/2021/10/a-guide-to-machine-learning-pipelines-and-orchest/
    🚀 2
  • a

    Abid

    10/19/2021, 2:03 PM
    https://dagshub.com/kingabzpro/ML-Pipeline-Disaster-tweets
  • p

    Pruthvi Shetty

    10/21/2021, 11:30 PM
    Hello, all. Not sure if this is the right channel to post. I saw a couple of issues while trying Orchest: • It adds empty cells to the top of the notebook after each run. • Any print statement such as ‘Success’ etc. seems to be printing multiples times in the logs Any ideas on how to resolve this?
    y
    r
    • 3
    • 5
  • m

    Maicon Faria

    10/22/2021, 6:04 PM
    Hello, is there recommendation for deploy Orchest on a HPC environment?
    y
    r
    • 3
    • 5
  • f

    Felix Becker

    10/22/2021, 9:46 PM
    Hey guys - I am having issues with transferring a dataframe from one step to another. orchest.output((predictions_df_all), name="predictions") when reading the dataframe: data = orchest.get_inputs() predictions_df_all = data['predictions'] I receive this error:
    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in <module>
          1 import orchest
    ----> 2 data = orchest.get_inputs()
          3 predictions_df_all = data['predictions']
    
    /orchest/orchest-sdk/python/orchest/transfer.py in get_inputs(ignore_failure, verbose)
       1016         # continue with other steps.
       1017         try:
    -> 1018             incoming_step_data = get_output_method(*args, **kwargs)
       1019         except error.OutputNotFoundError as e:
       1020             if not ignore_failure:
    
    /orchest/orchest-sdk/python/orchest/transfer.py in _get_output_memory(step_uuid, consumer)
        727     obj_id = _convert_uuid_to_object_id(step_uuid)
        728     try:
    --> 729         obj = _deserialize_output_memory(obj_id, client)
        730 
        731     except error.ObjectNotFoundError:
    
    /orchest/orchest-sdk/python/orchest/transfer.py in _deserialize_output_memory(obj_id, client)
        696         # Can load the buffer directly because its a bytes-like-object:
        697         # <https://docs.python.org/3/library/pickle.html#pickle.loads>
    --> 698         return pickle.loads(buffer)
        699     else:
        700         raise ValueError("Object was serialized with an unsupported serialization")
    
    AttributeError: Can't get attribute 'new_block' on <module 'pandas.core.internals.blocks' from '/opt/conda/lib/python3.7/site-packages/pandas/core/internals/blocks.py'>
    I already reloaded the session + sent "abcdef" from one step to another. There must be somethign in the dataframe that messes up the transfer. The problem is between my pipeline step: Inf_MakePrediction -> Inf_sendToFB Thank you guys in advance and keep up the great work!
    r
    • 2
    • 4
  • a

    Arnzok

    10/25/2021, 8:34 AM
    hi, i'm trying to use orchest on windows but i have multiple problems. Docker & wsl 2 are well working, it has to do with the "orchest" file to install & start : • when launching orchest install, i had an error (wrong folder used) so i changed the orchest file with //var/run instead of /var/run (i'm on win10 pro 64 bits) • after that the installation worked, but when i use orchest start i get the following error : "FileNotFoundError: [Errno 2] No such file or directory: '/orchest-host/orchest'" any idea on how to solve this please ?
    j
    y
    r
    • 4
    • 40
  • a

    Arnzok

    10/25/2021, 8:35 AM
    Traceback (most recent call last):
      File "/usr/local/bin/orchest", line 8, in <module>
        sys.exit(__entrypoint())
      File "/usr/local/lib/python3.8/site-packages/app/cli/main.py", line 84, in __entrypoint
        typer_app()
      File "/usr/local/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
        return get_command(self)(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1637, in invoke
        super().invoke(ctx)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
        return __callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/typer/main.py", line 500, in wrapper
        return callback(**use_params)  # type: ignore
      File "/usr/local/lib/python3.8/site-packages/app/cli/start.py", line 50, in _default
        reg(port, log_level, cloud, dev)
      File "/usr/local/lib/python3.8/site-packages/app/cli/start.py", line 93, in reg
        container_config = get_container_config(port, cloud, dev, log_level)
      File "/usr/local/lib/python3.8/site-packages/app/spec.py", line 131, in get_container_config
        config = get_reg_container_config(port, env)
      File "/usr/local/lib/python3.8/site-packages/app/spec.py", line 168, in get_reg_container_config
        env = utils.get_env()
      File "/usr/local/lib/python3.8/site-packages/app/utils.py", line 50, in get_env
        env["ORCHEST_HOST_GID"] = str(os.stat("/orchest-host/orchest").st_gid)
    FileNotFoundError: [Errno 2] No such file or directory: '/orchest-host/orchest'
  • e

    Eliezer de Souza da Silva

    10/25/2021, 8:46 AM
    The validation rules for the GitHub repo URLs is not working properly. I have a valid URL but it is still not allowing me to add the project (maybe because of the
    -
    in the name, although in theory it should accept that as well)
    j
    r
    • 3
    • 7
  • e

    Eliezer de Souza da Silva

    10/25/2021, 12:37 PM
    Is it possible to import new project using only subfolders of a github repository?
    r
    • 2
    • 3
  • r

    Rick Lamers

    10/28/2021, 11:10 AM
    ⚡ A New Orchest release: v2021.10.2 This one is special to me, as it ships our auto layout feature. Something I've been wanting to build for Orchest for a long time. It works through the Sugiyama graph layout algorithm devised by Kozo Sugiyama. Furthermore, it enables GPU passthrough for Orchest Cloud. Which should become enabled on Orchest Cloud this week or next week. Thanks for the hard work from @Huang-Ming @Jacopo @Yannick!
    auto-layout.mp4
    🛸 1
    🙌 4
    • 1
    • 1
  • r

    Rick Lamers

    10/28/2021, 9:51 PM
    Quick piece of advice! If you want to try speeding up your
    pandas
    code, give https://github.com/modin-project/modin a shot!
    # import pandas as pd
    import modin.pandas as pd
  • s

    Serhii

    10/29/2021, 12:17 PM
    Hi everyone! I wonder how can i limit concurrent number of steps executed? or better place them one after one to ensure they are not executed together?
    r
    j
    • 3
    • 2
  • s

    Serhii

    10/29/2021, 1:43 PM
    I have one more question apart from orchest cloud instance being run only when there is a job, are there other cost optimization solutions considered? I understand the issue that orchest is limited to 1 instance and cannot operate multiple, therefore it needs to run full one. One of use cases is api-rate-limited data gathering, which doesn't need cpu or ram but needs time, would be cool to have something cheap for those steps(but I understand that it might be not a target use case for you)
    r
    • 2
    • 6
  • s

    Serhii

    11/01/2021, 9:27 AM
    There definitely a need for a copy button for a step and maybe for a pipeline as well 🙂
    r
    • 2
    • 3
  • s

    Serhii

    11/01/2021, 10:23 AM
    Another stuff I see missing - I can see there is "monitoring" on paid instance. But I don't really know where to learn what exactly is it about. Its crucial to understand memory / cpu consumption of your jobs with time to place them right, so I guess (hope) it's that?
    r
    • 2
    • 1
  • r

    Rick Lamers

    11/01/2021, 6:01 PM
    How can we see the forest for the trees? Today I'm launching https://alldatatools.com. An open-source initiative to develop the most comprehensive collection of data tools for data practitioners. Found a tool you love? Want to update a description? Just open a PR!
    👏 1
    🤠 3
    a
    • 2
    • 7
  • s

    Serhii

    11/03/2021, 8:13 AM
    Weird flow for out of memory: 1. instance becomes inaccessible "Instance undergoing restart or update" 2. instance recovers in half hour + and seems the step in pipeline is shown as running but apparently it should have been crashed
    r
    j
    • 3
    • 15
  • s

    Serhii

    11/04/2021, 8:48 AM
    Question - is there a way to transfer instance to another payment account somehow?
    r
    • 2
    • 1
  • s

    Serhii

    11/04/2021, 12:52 PM
    When I run jupiter notebook as "run selected steps" it is show as ready and thats it - no success. Not very useful. Or maybe its not being run at all
    👍 1
    y
    • 2
    • 3
  • r

    Rick Lamers

    11/09/2021, 6:30 PM
    ⚡Update! Orchest Cloud (cloud.orchest.io) now supports GPU instances ⚡ Get your PyTorch and TensorFlow on to train deep learning models. You can simply stop instances and start them only when you need the GPU compute (job based auto start and stopping is coming soon).
    🙌🏼 1
    🙌 3
    👍 1
    • 1
    • 1
  • s

    Serhii

    11/10/2021, 8:52 AM
    Another feedback from non-pro 🙂 I see 2 things left for orchest io to be recommendable to everyone even now (maybe these are already there, as I`m not proficient with jupyter, more a VS Code user (it has some support of Jupyter though)). Maybe we could add VS Code and then all the problems would have been solved (or give access to instance - then we can install it) 1. Work with git (I`m not sure how to do that right now with my private repos so just thrown a directory and working with it and backing it up - super unsafe) 2. Code completions - idk what is the best way to make them working and if its even possible, missing multiline select in VS Code, especially needed to adjust python lines 🙂
    r
    • 2
    • 2
  • s

    Serhii

    11/12/2021, 12:33 PM
    Can there a be an option of keeping instance auto clean? (like keeping only last 2 jobs data) I mean we have to manually remove job files from the instance if I understand correctly? Or setup a job to clean previous jobs? Otherwise I need always keep monitoring instance memory, and if I have some cron jobs running it might be very hard
    j
    r
    • 3
    • 4
  • s

    Serhii

    11/13/2021, 8:42 AM
    One more crucial. Literally no ability to run visual step from mobile. Wanted to do data deployment on the go and realized that in mobile there is no space and in desktop mode graph is not draggable, and there is no list to select non interactively. Maybe replace graph with list? Or make desktop mode on mobile draggable Or make zoom out center the view (it leaves blank space now
    r
    • 2
    • 6
  • a

    Abid

    11/16/2021, 6:55 PM
    Learn how machine learning pipelines are used in productions and design your first pipeline using simple steps on disaster tweets classification datasets. -Repost https://pub.towardsai.net/a-guide-to-machine-learning-pipelines-and-orchest-80f3e3a49994
    💯 3
    r
    • 2
    • 1
  • s

    Serhii

    11/23/2021, 7:49 AM
    Morning guys! I often see jupyter hang eternally, what can I do with it? Might be related to occasionally it is not saving files. "Saving started" and kernel not initialized Additional question - how to run python file separately with environment I built? (not through step -> run) Can VS Code service reuse environment? (Just trying to figure out correct expected workflow working with orchest)
    r
    • 2
    • 26
  • a

    Abid

    11/25/2021, 11:44 AM
    @Rick Lamers the guide got first award:
    🙌 5
    r
    • 2
    • 2
  • s

    Serhii

    11/30/2021, 8:57 AM
    Wanted to ask - sometimes logs are not being fetch for the job into the ui, how can I improve on that?
    r
    • 2
    • 8
Powered by Linen
Title
s

Serhii

11/30/2021, 8:57 AM
Wanted to ask - sometimes logs are not being fetch for the job into the ui, how can I improve on that?
r

Rick Lamers

11/30/2021, 8:58 AM
Hi Nem, thanks for reporting the issue. That should not happen. Is there a particular set of steps you know that could let us reproduce the issue?
On which version are you, if I may ask?
s

Serhii

11/30/2021, 9:10 AM
I will write it down in detail once I see it again
r

Rick Lamers

11/30/2021, 9:55 AM
👍
Thanks!
s

Serhii

12/02/2021, 4:33 PM
@Rick Lamers it happened for python files mostly now we have one
That is not showing logs dynamically
r

Rick Lamers

12/03/2021, 12:44 AM
Alright, thanks for reporting. I’ll investigate 🙌🏻
View count: 1