Hello guys some of the jupyter servers in our custom deploym

Question

Hello guys some of the jupyter servers in our custom deployment are stucked in the following state ```Awaiting boot lock Awaiting boot lock Awaiting boot lock Awaiting boot lock Awaiting boot lock Awaiting boot lock Awaiting boot lock Awaiting boot lock Awaiting boot lock Awaiting boot lock ``` ```Because of this we are unable to create sessions in those projects I have also noticed that by restarting the orchest cluster using the restart button some of the projects that were not working are fixed but other projects that were working starts to have the same issue Some important points I have not found a lock in the $lockdir By running the start sh with the same parameters I could reproduce the same result in jupyter servers that are not working ```

Jacopo · Answer

Hi could you exec into the orchest webserver and take a look at what s in the ` orchest user configurations jupyterlab lab` path of the userdir I d expect the lock to be there What s in the ` usr local share jupyter lab` path of jupyter servers that are and are not failing I e is the lock there for any of them

Jacopo · Answer

If the lock is not there I d exec into the failing jupyter servers and take a look at the result of ```userdir path= usr local share jupyter lab lockdir=$userdir path bootlock mkdir $lockdir``` I guess the error message will be interesting

Rafael Rodrigues Santana · Answer

Yea it s there

Rafael Rodrigues Santana · Answer

The interesting thing is that the lock is in present in both scenarios

Rafael Rodrigues Santana · Answer

Logs from a server that s working properly

Jacopo · Answer

Have you tried deleting the lock file That should pretty much suffice this snippet of logic ```release lock rm rf $lockdir ``` is run once the startup logic of the jupyter server is done perhaps something has gone wrong and a dangling lock has been left around

Rafael Rodrigues Santana · Answer

I haven t tried Should I do it on the webserver or on each of the containers

Jacopo · Answer

Only once doesn t matter from which container since those directories are mounted webserver is ok

Rafael Rodrigues Santana · Answer

Makes sense

Rafael Rodrigues Santana · Answer

I ll try it right now thanks for the support as always Jacopo

Jacopo · Answer

no problem +1

Rafael Rodrigues Santana · Answer

Unfortunately it didn t work =x

Jacopo · Answer

Have you tried restarting the jupyter server pods that are stuck Now that the lock is there it shouldn t be a problem anymore

Rafael Rodrigues Santana · Answer

yea I killed those pods

Jacopo · Answer

are they still stuck waiting on the lock How many such pods Could it be that s because there are many jupyter server going through the logic sequentially I e one lock acquire at a time

Rafael Rodrigues Santana · Answer

Currently 7 of them are stucked in CrashLoopBackoff

Rafael Rodrigues Santana · Answer

I ll kill the deployments and try this process one by one to see if it works

Jacopo · Answer

I see I d be curious to see what they log before crashing

Rafael Rodrigues Santana · Answer

By deleting the deployments and starting them one by one I was able to fix this issue Thank you very much < Jacopo>