Project removed from microclimate

[arjunsbabu]

Issue type: Bug/
Reported in: 1.10 installed on ICp
Description of issue: We have noticed project got removed from user. We have couple of build failures and hence after understanding the reason for build failure (changes in cluster image policy , so jenkins not able to pull image ) we terminated an ongoing build and logged out of microclimate and checked option to stop the user pods. When we logged back again we can see the project got missed from microclimate UI and the user pod went to terminating loop. We have checked the folders under user folder in microclimate PV and we can see no project folders exist..i request the microclimate team to check on this as its a serious problem as i never deleted the project manually.
In the same infra i faced a similar issue ie dissassociation of pipeline and deployment before 1 week. Not sure why that happened. Ie dissociation of pipeline from project and it will ask us to link and existing pipeline to project
Now for solving the issue i recreated the project and linked the existing pipeline.

Workaround (if possible): no

Logs are located at https://ibm.ent.box.com/folder/77474648200

[root@dohvoricpap001s ~]# k get po -n devops
NAME                                                              READY     STATUS        RESTARTS   AGE
microclimate-ibm-microclimate-94b99f9c7-mkw7r                     1/1       Running       0          51d
microclimate-ibm-microclimate-atrium-686f76d7df-mhn2f             1/1       Running       0          51d
microclimate-ibm-microclimate-devops-7c7df79b7-qjvkn              1/1       Running       0          51d
microclimate-ibm-microclimate-dohadminnday-editor-57684967998qm   0/2       Terminating   0          7d
microclimate-ibm-microclimate-dohadminnday-editor-57684967h5ff9   2/2       Running       0          20h
microclimate-ibm-microclimate-dohadminnday-filewatcher-6cc586p5   0/1       Terminating   0          7d
microclimate-ibm-microclimate-dohadminnday-filewatcher-6ccxnb86   1/1       Running       0          20h
microclimate-ibm-microclimate-dohadminnday-loadrunner-7cc97qw2n   1/1       Running       0          20h
microclimate-ibm-microclimate-dohadminnday-loadrunner-7cc9wmhd4   0/1       Terminating   0          7d
microclimate-jenkins-74f8fc6cb7-2cssg                             1/1       Running       0          20d

[sishida]

Hello! Could you please try to update to the newest version of Microclimate?
https://github.com/IBM/charts/blob/master/stable/ibm-microclimate/RELEASENOTES.md
If problems persist after the update, please let us know! Thank you!

[arjunsbabu]

@sishida we cant do it because of microclimate limitation. We can only use ICp 3.1.0 because that is a requirement for another chart ( Watson assistant 1.1.0 ) . Microclimate 1.10 is the highest supported on ICp 3.1.0 . So we cannot proceed with highest version of MC without updating ICp
So let us know how to avoid such issues in MC 1.10.
Is it a known issue ?

[sishida]

I have notified my team of this situation. I will let you know what we find out.

[arjunsbabu]

@sishida Thanks for your reply.
I have updated my first comment. Earlier i told like its not impacting us as we are using pipeline. But its impacting us .
ie the build is successful and the notification is send to devops for deployment as per successful build log. But the deployment is not happening. The devops pod is not getting any notification when i checked its log. So i thought its because of not existing the three user pod. So i logged into microclimate as a user so the three pod ( filewatcher , load runner , editor ) got created and ran the build again but deployment never happened. Then i created new project with the same name as old ( auto deleted one ) and linked the existing pipeline and ran a build again , and the deployment happended successfully after build. ie devops pod got the notification.
So i believe the creation of the project and linking the existing pipeline solved the issue but not 100% sure it could be also the logging to MC portal in the previous step solved the issue. (10% ) as per my understanding. Please help me understand the resolution

[arjunsbabu]

@sishida any update ?

[jagraj]

Hi Arjun,

We delivered new feature in Microclimate 19.05 release and chart version is v1.13.0. With this feature users can create pipeline with out the project from the portal UI. To create pipeline in the version v10.0 you need to create project first and then create pipeline. Do you still have dependency with Watson assistant team where they do not support latest version of ICP 3.1.x and 3.2.0.?

[arjunsbabu]

We still have dependancy for 3.1.0.
We are perfectly ok with creating project and then pipeline. Our concern is about project deletion.
ie our build was not working ( due to docker pull issue from our side ) . So i terminated the build and then logged out of microclimate after asking to shutdown pods.
But the projects are removed from microclimate due to that. I think its a bug and like the microclimate team to check on it.
To fix issue i have recreated the project with the same old name and associated the existing pipeline and issue is fixed.

[jagraj]

There was one issue from the user where ClusterImagePolicypolicy repos was removed accidentally and this caused Jenkins slaves not to come up and hence user terminated running build and logged out of MC portal with shutdown option.

After fixing cluster image policy user loggedin back and found project was missing. User also verified in persistence volume and the workspace projects folder was missing.

I and Arjun had web-ex call and we tried to reproduce on one of my cluster and we could not reproduce this problem.

Arjun is going to watch this in POC cluster and try to reproduce this problem again.