https://www.orchest.io/ logo
m

Médéric Descazeaux

12/17/2020, 3:24 PM
Hello! I am getting an error I cannot explain while passing data. Here is the output call for "Load current logs":
orchest.output({'df': df}, name="current")
And for "Process data":
Copy code
orchest.output({'df': df,
                'processedlist': processedlist,
                'ignoredlist': ignoredlist,
                'messageslist': messageslist},
               name='processed')
I get the following error:
Copy code
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-b9a2b7cd25cd> in <module>
      5 print("Getting current and pending logs from Orchest pipeline...", end=" ")
      6 
----> 7 data = orchest.get_inputs()
      8 
      9 display(data)

/orchest/orchest-sdk/python/orchest/transfer.py in get_inputs(ignore_failure, verbose)
    859         parent_uuid = parent.properties["uuid"]
    860         get_output_method, args, kwargs, metadata = _resolve(
--> 861             parent_uuid, consumer=step_uuid
    862         )
    863 

/orchest/orchest-sdk/python/orchest/transfer.py in _resolve(step_uuid, consumer)
    753                 method_info = method(step_uuid, consumer=consumer)
    754             else:
--> 755                 method_info = method(step_uuid)
    756 
    757         except OutputNotFoundError:

/orchest/orchest-sdk/python/orchest/transfer.py in _resolve_disk(step_uuid)
    349         with open(head_file, "r") as f:
    350             timestamp, serialization, name = f.read().split(
--> 351                 Config.__METADATA_SEPARATOR__
    352             )
    353 

ValueError: not enough values to unpack (expected 3, got 1)
ValueError: not enough values to unpack (expected 3, got 1)
The error only happens if the "Load current logs" block is connected to the "Merge logs" one.
r

Rick Lamers

12/17/2020, 3:27 PM
Thanks for sharing. We’re looking into it. Probably there’s something wrong with the output file written by Load current logs. Offtopic: cool mod of the pipeline color scheme. Did you use custom CSS?
If you want, could you share a .zip of the orchest/userdir/projects/<your project> ? It could contain some state that can help us find the issue.
j

Jacopo

12/17/2020, 3:31 PM
Thanks for reaching out! I am on it
m

Médéric Descazeaux

12/17/2020, 3:31 PM
I have just realized my company would not allow me to send the whole code, sorry... But I can try to replicate the error in another project (it may help me understand what it is happening)
Offtopic: I am using the Dark Reader extension in my browser, it has greatly reduced my eye strain 🙂
r

Rick Lamers

12/17/2020, 3:33 PM
No worries, no need to send the project
We can most likely find the root cause without
Your example was quite clear
m

Médéric Descazeaux

12/17/2020, 3:49 PM
FYI I failed to replicate it in another project, so I deleted the blocks and readded them, and it seems to be working now. I had several times moved them, moved connections, renamed notebooks in the past hour, so maybe the bug lays over there. 😉
r

Rick Lamers

12/17/2020, 3:56 PM
I think we found the cause: there existed output of that step on disk that was written by an older version of the Orchest SDK. In a way it was ‘cached’ in the project directory. By creating a new step the UUID of the step changes which causes the old output to be ignored. Hence, solving your problem. We can’t guarantee that this actually caused your error. But in case it comes up again in another project, you can always fix it by recreating the step (triggering a new UUID) or by calling
orchest.output_to_disk(data, name=my_name)
from the step that outputs the data.
m

Médéric Descazeaux

12/17/2020, 4:01 PM
Nice explanation, thanks! I indeed updated Orchest at some point today or yesterday. Thanks for working on it, I am eager to see where you go with Orchest 🙂
🎉 1
😄 1
r

Rick Lamers

12/17/2020, 4:01 PM
FYI we’ve created an internal issue to increase the robustness of the Orchest SDK handling older versions of cached data. We could have gracefully handled this edge case - and we will in the future ; )