Hello! I am getting an error I cannot explain while passing data. Here is the output call for "Load current logs":

orchest.output({'df': df}, name="current")

And for "Process data":

orchest.output({'df': df,
                'processedlist': processedlist,
                'ignoredlist': ignoredlist,
                'messageslist': messageslist},
               name='processed')

I get the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-b9a2b7cd25cd> in <module>
      5 print("Getting current and pending logs from Orchest pipeline...", end=" ")
      6 
----> 7 data = orchest.get_inputs()
      8 
      9 display(data)

/orchest/orchest-sdk/python/orchest/transfer.py in get_inputs(ignore_failure, verbose)
    859         parent_uuid = parent.properties["uuid"]
    860         get_output_method, args, kwargs, metadata = _resolve(
--> 861             parent_uuid, consumer=step_uuid
    862         )
    863 

/orchest/orchest-sdk/python/orchest/transfer.py in _resolve(step_uuid, consumer)
    753                 method_info = method(step_uuid, consumer=consumer)
    754             else:
--> 755                 method_info = method(step_uuid)
    756 
    757         except OutputNotFoundError:

/orchest/orchest-sdk/python/orchest/transfer.py in _resolve_disk(step_uuid)
    349         with open(head_file, "r") as f:
    350             timestamp, serialization, name = f.read().split(
--> 351                 Config.__METADATA_SEPARATOR__
    352             )
    353 

ValueError: not enough values to unpack (expected 3, got 1)
ValueError: not enough values to unpack (expected 3, got 1)

The error only happens if the "Load current logs" block is connected to the "Merge logs" one.

Thanks for sharing. We’re looking into it. Probably there’s something wrong with the output file written by Load current logs. Offtopic: cool mod of the pipeline color scheme. Did you use custom CSS?

Thanks for reaching out! I am on it

I have just realized my company would not allow me to send the whole code, sorry... But I can try to replicate the error in another project (it may help me understand what it is happening)

No worries, no need to send the project

FYI I failed to replicate it in another project, so I deleted the blocks and readded them, and it seems to be working now. I had several times moved them, moved connections, renamed notebooks in the past hour, so maybe the bug lays over there. 😉

I think we found the cause: there existed output of that step on disk that was written by an older version of the Orchest SDK. In a way it was ‘cached’ in the project directory. By creating a new step the UUID of the step changes which causes the old output to be ignored. Hence, solving your problem. We can’t guarantee that this actually caused your error. But in case it comes up again in another project, you can always fix it by recreating the step (triggering a new UUID) or by calling

orchest.output_to_disk(data, name=my_name)

from the step that outputs the data.

Nice explanation, thanks! I indeed updated Orchest at some point today or yesterday. Thanks for working on it, I am eager to see where you go with Orchest 🙂

FYI we’ve created an internal issue to increase the robustness of the Orchest SDK handling older versions of cached data. We could have gracefully handled this edge case - and we will in the future ; )