Seems auto migration didnt work and no clue what to do next

Question

Seems auto migration didnt work and no clue what to do next in orchest cloud

Jacopo · Answer

Hi < Serhii> your old instance should still be functional and its state should not be affected in case of a failure during the migration Was this not the case I ve taken a look at the logs and it looks like the failure is due to some inconsistent state in the old orchest instance I should be able to fix it tomorrow

Serhii · Answer

Okay seems my environment stopped building in old version for some reason and job step reported failure even though there was nothing that could fail I gave one more try to migrate hoping new version will fix those If not hope it can be done tomorrow

Jacopo · Answer

gt Okay seems my environment stopped building in old version for some reason and job step reported failure even though there was nothing that could fail The instance to be migrated gets stopped and all ongoing runs are cancelled as far as I remember this information is part of the warnings that the orchest cloud pops up before migrating if that s not the case I ll make sure that s fixed tomorrow gt I gave one more try to migrate hoping new version will fix those If not hope it can be done tomorrow I doubt that will work but if it does it s definitively a welcome surprise I can pretty much promise that tomorrow this will be fixed slightly smiling face

Jacopo · Answer

tomorrow I can take care of migrating the instance as well although as mentioned ongoing runs are going to be cancelled

Jacopo · Answer

but if you prefer to migrate it at your own time that s perfectly okay of course

Serhii · Answer

Sure sure Just if im lucky to see it autofixed i can make some work today as i planned slightly smiling face If not well then probably tomorrow Neither old nor new version is in working state anymore anyway slightly smiling face they broke some time before not due to migration guess due to some orcest change when so environment got built successfully but didnt install what I needed didnt notice that

Serhii · Answer

< Jacopo> okay it didnt work so waiting for you to help me out with migration

Jacopo · Answer

Hi < Serhii> today we will be looking into it I might have to start the instance and or try to migrate it myself to make things quicker would that be ok

Serhii · Answer

< Jacopo> sure you can event ping me for questions

Jacopo · Answer

alright will keep you posted

Jacopo · Answer

< Serhii> the instance has been migrated please let me know if everything is in order Note that for the existing project I had to create a temporary pipeline in order to fix the inconsistent state w r t some old existing jobs runs feel fee to delete it

Serhii · Answer

thanks checking out memory used on machine is suspiciously low

Serhii · Answer

0 o your python image has node js 0 p

Serhii · Answer

< Jacopo> sorry to bother again seems my setup got broken due to some package versions python versions i guess What is the best way to set python version on your image so I can try if one of them works for me

Jacopo · Answer

< Serhii> you are not bothering at all on the contrary I m sorry that you are meeting post migration issues During the migration from docker to k8s our base images have lost some weight and while the cloud migration logic does add some packages back through the environment setup script in an attempt to avoid breakage it could very well be that some package or program that you depended on is not there anymore so adding that to the setup script could fix the issue What kind of error are you getting Is it something you can share Here or through a PM