orchest #announcements

perhaps a known issue, but it seems https://cloud.orchest.io/onboarding doesn't work on Brave? switching to Chrome solved it for me [EDIT] I spoke too soon 🙊 the main page loads on chrome but nothing happens when I click on "New Instance". Is this website down?

Abid

10/19/2021, 2:02 PM

A Guide to Machine Learning Pipelines and Orchest https://www.analyticsvidhya.com/blog/2021/10/a-guide-to-machine-learning-pipelines-and-orchest/

🚀 2

Abid

10/19/2021, 2:03 PM

https://dagshub.com/kingabzpro/ML-Pipeline-Disaster-tweets

Pruthvi Shetty

10/21/2021, 11:30 PM

Hello, all. Not sure if this is the right channel to post. I saw a couple of issues while trying Orchest: • It adds empty cells to the top of the notebook after each run. • Any print statement such as ‘Success’ etc. seems to be printing multiples times in the logs Any ideas on how to resolve this?

Maicon Faria

10/22/2021, 6:04 PM

Hello, is there recommendation for deploy Orchest on a HPC environment?

Felix Becker

10/22/2021, 9:46 PM

Hey guys - I am having issues with transferring a dataframe from one step to another. orchest.output((predictions_df_all), name="predictions") when reading the dataframe: data = orchest.get_inputs() predictions_df_all = data['predictions'] I receive this error:

Copy code

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in <module>
      1 import orchest
----> 2 data = orchest.get_inputs()
      3 predictions_df_all = data['predictions']

/orchest/orchest-sdk/python/orchest/transfer.py in get_inputs(ignore_failure, verbose)
   1016         # continue with other steps.
   1017         try:
-> 1018             incoming_step_data = get_output_method(*args, **kwargs)
   1019         except error.OutputNotFoundError as e:
   1020             if not ignore_failure:

/orchest/orchest-sdk/python/orchest/transfer.py in _get_output_memory(step_uuid, consumer)
    727     obj_id = _convert_uuid_to_object_id(step_uuid)
    728     try:
--> 729         obj = _deserialize_output_memory(obj_id, client)
    730 
    731     except error.ObjectNotFoundError:

/orchest/orchest-sdk/python/orchest/transfer.py in _deserialize_output_memory(obj_id, client)
    696         # Can load the buffer directly because its a bytes-like-object:
    697         # <https://docs.python.org/3/library/pickle.html#pickle.loads>
--> 698         return pickle.loads(buffer)
    699     else:
    700         raise ValueError("Object was serialized with an unsupported serialization")

AttributeError: Can't get attribute 'new_block' on <module 'pandas.core.internals.blocks' from '/opt/conda/lib/python3.7/site-packages/pandas/core/internals/blocks.py'>

I already reloaded the session + sent "abcdef" from one step to another. There must be somethign in the dataframe that messes up the transfer. The problem is between my pipeline step: Inf_MakePrediction -> Inf_sendToFB Thank you guys in advance and keep up the great work!

Arnzok

10/25/2021, 8:34 AM

hi, i'm trying to use orchest on windows but i have multiple problems. Docker & wsl 2 are well working, it has to do with the "orchest" file to install & start : • when launching orchest install, i had an error (wrong folder used) so i changed the orchest file with //var/run instead of /var/run (i'm on win10 pro 64 bits) • after that the installation worked, but when i use orchest start i get the following error : "FileNotFoundError: [Errno 2] No such file or directory: '/orchest-host/orchest'" any idea on how to solve this please ?

Arnzok

10/25/2021, 8:35 AM

Copy code

Traceback (most recent call last):
  File "/usr/local/bin/orchest", line 8, in <module>
    sys.exit(__entrypoint())
  File "/usr/local/lib/python3.8/site-packages/app/cli/main.py", line 84, in __entrypoint
    typer_app()
  File "/usr/local/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1637, in invoke
    super().invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/app/cli/start.py", line 50, in _default
    reg(port, log_level, cloud, dev)
  File "/usr/local/lib/python3.8/site-packages/app/cli/start.py", line 93, in reg
    container_config = get_container_config(port, cloud, dev, log_level)
  File "/usr/local/lib/python3.8/site-packages/app/spec.py", line 131, in get_container_config
    config = get_reg_container_config(port, env)
  File "/usr/local/lib/python3.8/site-packages/app/spec.py", line 168, in get_reg_container_config
    env = utils.get_env()
  File "/usr/local/lib/python3.8/site-packages/app/utils.py", line 50, in get_env
    env["ORCHEST_HOST_GID"] = str(os.stat("/orchest-host/orchest").st_gid)
FileNotFoundError: [Errno 2] No such file or directory: '/orchest-host/orchest'

Eliezer de Souza da Silva

10/25/2021, 8:46 AM

The validation rules for the GitHub repo URLs is not working properly. I have a valid URL but it is still not allowing me to add the project (maybe because of the

in the name, although in theory it should accept that as well)

Eliezer de Souza da Silva

10/25/2021, 12:37 PM

Is it possible to import new project using only subfolders of a github repository?

Rick Lamers

10/28/2021, 11:10 AM

⚡ A New Orchest release: v2021.10.2 This one is special to me, as it ships our auto layout feature. Something I've been wanting to build for Orchest for a long time. It works through the Sugiyama graph layout algorithm devised by Kozo Sugiyama. Furthermore, it enables GPU passthrough for Orchest Cloud. Which should become enabled on Orchest Cloud this week or next week. Thanks for the hard work from @Huang-Ming @Jacopo @Yannick!

auto-layout.mp4

🛸 1

🙌 4

Rick Lamers

10/28/2021, 9:51 PM

Quick piece of advice! If you want to try speeding up your

pandas

code, give https://github.com/modin-project/modin a shot!

Copy code

# import pandas as pd
import modin.pandas as pd

Serhii

10/29/2021, 12:17 PM

Hi everyone! I wonder how can i limit concurrent number of steps executed? or better place them one after one to ensure they are not executed together?

Serhii

10/29/2021, 1:43 PM

I have one more question apart from orchest cloud instance being run only when there is a job, are there other cost optimization solutions considered? I understand the issue that orchest is limited to 1 instance and cannot operate multiple, therefore it needs to run full one. One of use cases is api-rate-limited data gathering, which doesn't need cpu or ram but needs time, would be cool to have something cheap for those steps(but I understand that it might be not a target use case for you)

Serhii

11/01/2021, 9:27 AM

There definitely a need for a copy button for a step and maybe for a pipeline as well 🙂

Serhii

11/01/2021, 10:23 AM

Another stuff I see missing - I can see there is "monitoring" on paid instance. But I don't really know where to learn what exactly is it about. Its crucial to understand memory / cpu consumption of your jobs with time to place them right, so I guess (hope) it's that?

Rick Lamers

11/01/2021, 6:01 PM

How can we see the forest for the trees? Today I'm launching https://alldatatools.com. An open-source initiative to develop the most comprehensive collection of data tools for data practitioners. Found a tool you love? Want to update a description? Just open a PR!

👏 1

🤠 3

Serhii

11/03/2021, 8:13 AM

Weird flow for out of memory: 1. instance becomes inaccessible "Instance undergoing restart or update" 2. instance recovers in half hour + and seems the step in pipeline is shown as running but apparently it should have been crashed

Serhii

11/04/2021, 8:48 AM

Question - is there a way to transfer instance to another payment account somehow?

Serhii

11/04/2021, 12:52 PM

When I run jupiter notebook as "run selected steps" it is show as ready and thats it - no success. Not very useful. Or maybe its not being run at all

👍 1

Rick Lamers

11/09/2021, 6:30 PM

⚡Update! Orchest Cloud (cloud.orchest.io) now supports GPU instances ⚡ Get your PyTorch and TensorFlow on to train deep learning models. You can simply stop instances and start them only when you need the GPU compute (job based auto start and stopping is coming soon).

🙌🏼 1

🙌 3

👍 1

Serhii

11/10/2021, 8:52 AM

Another feedback from non-pro 🙂 I see 2 things left for orchest io to be recommendable to everyone even now (maybe these are already there, as I`m not proficient with jupyter, more a VS Code user (it has some support of Jupyter though)). Maybe we could add VS Code and then all the problems would have been solved (or give access to instance - then we can install it) 1. Work with git (I`m not sure how to do that right now with my private repos so just thrown a directory and working with it and backing it up - super unsafe) 2. Code completions - idk what is the best way to make them working and if its even possible, missing multiline select in VS Code, especially needed to adjust python lines 🙂

Serhii

11/12/2021, 12:33 PM

Can there a be an option of keeping instance auto clean? (like keeping only last 2 jobs data) I mean we have to manually remove job files from the instance if I understand correctly? Or setup a job to clean previous jobs? Otherwise I need always keep monitoring instance memory, and if I have some cron jobs running it might be very hard

Serhii

11/13/2021, 8:42 AM

One more crucial. Literally no ability to run visual step from mobile. Wanted to do data deployment on the go and realized that in mobile there is no space and in desktop mode graph is not draggable, and there is no list to select non interactively. Maybe replace graph with list? Or make desktop mode on mobile draggable Or make zoom out center the view (it leaves blank space now

Abid

11/16/2021, 6:55 PM

Learn how machine learning pipelines are used in productions and design your first pipeline using simple steps on disaster tweets classification datasets. -Repost https://pub.towardsai.net/a-guide-to-machine-learning-pipelines-and-orchest-80f3e3a49994

💯 3

Serhii

11/23/2021, 7:49 AM

Morning guys! I often see jupyter hang eternally, what can I do with it? Might be related to occasionally it is not saving files. "Saving started" and kernel not initialized Additional question - how to run python file separately with environment I built? (not through step -> run) Can VS Code service reuse environment? (Just trying to figure out correct expected workflow working with orchest)

Abid

11/25/2021, 11:44 AM

@Rick Lamers the guide got first award:

🙌 5

Serhii

11/30/2021, 8:57 AM

Wanted to ask - sometimes logs are not being fetch for the job into the ui, how can I improve on that?