https://www.orchest.io/ logo
Title
s

Serhii

10/02/2022, 3:21 PM
Seems auto-migration didnt work and no clue what to do next in orchest cloud
j

Jacopo

10/02/2022, 3:44 PM
Hi @Serhii, your old instance should still be functional and its state should not be affected in case of a failure during the migration. Was this not the case? I've taken a look at the logs and it looks like the failure is due to some inconsistent state in the old orchest instance, I should be able to fix it tomorrow
s

Serhii

10/02/2022, 3:45 PM
Okay, seems my environment stopped building in old version for some reason and job step reported failure even though there was nothing that could fail. I gave one more try to migrate hoping new version will fix those. If not, hope it can be done tomorrow
j

Jacopo

10/02/2022, 3:50 PM
Okay, seems my environment stopped building in old version for some reason and job step reported failure even though there was nothing that could fail.
The instance to be migrated gets stopped and all ongoing runs are cancelled, as far as I remember this information is part of the warnings that the orchest cloud pops up before migrating, if that's not the case I'll make sure that's fixed tomorrow
I gave one more try to migrate hoping new version will fix those. If not, hope it can be done tomorrow
I doubt that will work but if it does it's definitively a welcome surprise. I can pretty much promise that tomorrow this will be fixed 🙂
tomorrow I can take care of migrating the instance as well although as mentioned ongoing runs are going to be cancelled
but if you prefer to migrate it at your own time that's perfectly okay of course
s

Serhii

10/02/2022, 3:53 PM
Sure, sure. Just if im lucky to see it autofixed, i can make some work today as i planned 🙂 If not - well, then probably tomorrow... Neither old nor new version is in working state anymore anyway 🙂 (they broke some time before not due to migration, guess due to some orcest change when so environment got "built successfully" but didnt install what I needed, didnt notice that)
👀 1
@Jacopo okay, it didnt work so waiting for you to help me out with migration...
j

Jacopo

10/03/2022, 7:36 AM
Hi @Serhii , today we will be looking into it, I might have to start the instance and/or try to migrate it myself to make things quicker, would that be ok?
s

Serhii

10/03/2022, 9:02 AM
@Jacopo sure you can event ping me for questions
j

Jacopo

10/03/2022, 9:03 AM
alright, will keep you posted!
@Serhii the instance has been migrated, please let me know if everything is in order. Note that for the existing project I had to create a temporary pipeline in order to fix the inconsistent state w.r.t. some old existing jobs/runs, feel fee to delete it
s

Serhii

10/03/2022, 2:23 PM
thanks, checking out, memory used on machine is suspiciously low
✅ 1
0_o your python image has node js 0_p
@Jacopo sorry to bother again, seems my setup got broken due to some package versions / python versions i guess. What is the best way to set python version on your image so I can try if one of them works for me?
j

Jacopo

10/04/2022, 7:40 AM
@Serhii you are not bothering at all, on the contrary, I'm sorry that you are meeting post migration issues. During the migration from docker to k8s our base images have lost some weight, and while the cloud migration logic does add some packages back through the environment setup script in an attempt to avoid breakage it could very well be that some package or program that you depended on is not there anymore, so adding that to the setup script could fix the issue What kind of error are you getting? Is it something you can share? Here or through a PM