Clone repos from da-server

[robobenklein]

Need some way that allows workers to clone from WoC repos

Perhaps WoC could use a token, or we have one ssh key we make for all CA workers?

[agrant16]

One ssh key should probably work, though I don't know if they have rate limiting setup which might interfere with our efforts. I still don't have access to WoC. I've emailed Dr. Mockus to see if he can go ahead and get my access setup this week.

[robobenklein]

Sounds like an SSH key would work, but I think it would be better practice to get a separate account for the workers to use since there doesn't seem to be any protection on the mongodb databases on da servers.

[robobenklein]

ok new plan from today:

build a repo list based on the thruMaps for each target language:

• get blob
• find repo for commit that had that blob
• export repo information

export to json, cp from server, import into db using a model changer

[agrant16]

We can get repos directly from the output of the thruMaps. We can just cut the repo out of that output directly and filter it through uniq. Or is there some advantage to collecting blobs from the thruMaps and going blob->commit->repo that I'm missing?

[robobenklein]

we can get the list of repos for a language, but I think there's also some useful metadata stored elsewhere that we'd otherwise want to pull from the github api, and for non-GH (GL, BB, etc) repos we should come up with some kind of common set of stats

[agrant16]

That's true. I'm not sure what all metadata we can pull from WoC, but I'll look into it more and see. If we can get stuff we want from there instead that would make things easier.