I’m having a weird issue where python libraries are successfully identified and run within JupyterLab, but in pipeline view, the libraries are not identified. (cc: @Shaleen Bengani)

Did you

pip install pandas

or similar in the environment?

I think it might be an case of https://github.com/orchest/orchest/issues/425 - @Eshwaran Venkat could you share your setup script (where you installed the dependencies) so we can try to reproduce the problem?

The way it works internally, is that running code in JupyterLab (inside a kernel) is different from running code when doing a pipeline run; a different code path is invoked (link to source code). Like Juanlu alluded to with his comment, your issue is likely to be caused by something "custom" in your environments set-up script which leads to a wrong use/initialization of the underlying

conda

environment.

could you share your setup script

This would indeed be a great start for us to debug your problem :)

Hey guys! Thanks for the quick turnaround. Yes, we have a custom setup for Git and also an internal git package we use within the org. Let me share the setup script

Thank you for sharing! What version of Orchest are you running on?

How do I check the version?

@Eshwaran Venkat you can go to the

/settings

page, there should be something like this:

Ah apologies, here you go!

I'll try to reproduce without installing your private package

Sure, thanks!

I tried this setup script:

#!/bin/bash
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install -y git

pip install orchest
# pip install git+https://{ACCESS_KEY}@github.com/dotlas/swarm.git -q
pip install geoalchemy2

and everything seems to work fine on JupyterLab ✔️ , interactive pipeline runs ✔️ , and one-off jobs ✔️ the only explanation is that some dependency of

swarm

is causing some disruption. let's take this privately.

Got it. Thanks!

I can reproduce the issue with this setup script:

#!/bin/bash
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install -y git

pip install orchest
pip install "ipython==8.2.0" "ipykernel==6.12.1" "jupyter-client==7.2.1" "numpy==1.22.3" "SQLAlchemy==1.4.34"
pip install geoalchemy2

That partially makes sense to me 🤔 I don't understand why running inside a Jupyter kernel works, but inside a pipeline run it does not. Could it be that

nbformat

(which we use to execute Notebooks in pipeline runs) has an incompatibility with that specific

ipykernel

version?

for completeness, the latest Orchest version is also affected by this issue

Thank you @juanlu, @Shaleen Bengani appears to have found out that using pipreqs instead of pip freeze works for our private package.

Hi @Eshwaran Venkat, we noticed that adding

python -m ipykernel install --sys-prefix

at the end fixes the problem, in case you want to try it out (if you have already simplified the dependencies and have a working solution, no need to do anything then)