-
Notifications
You must be signed in to change notification settings - Fork 3
Dev #12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Dev #12
Conversation
getVersion func modified to remove loop
Code Review changes made, VersionParser Interface introduced
parseVersion() fixed
joelanford
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like a good set of improvements from last time. Here's another round or comments and suggestions. We're getting there! 👍
cmd/sdkstats/main.go
Outdated
| } else if _, ok := err.(*github.RateLimitError); ok { | ||
| log.Println("Rate Limit has reached.") | ||
| } else if err != nil { | ||
| fmt.Println("Failed to get Stats for Query", r, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a great opportunity to use a type switch. I'd suggest using this instead of if .. else if blocks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will Do.
cmd/sdkstats/main.go
Outdated
| _ = ioutil.WriteFile(fileName, file, 0644) | ||
| fmt.Println("Results are written in ", fileName) | ||
| if _, ok := err.(*github.AcceptedError); ok { | ||
| log.Println("Job is scheduled on GitHub side") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this error something we can recover from? If something is scheduled on the Github side, can we make a follow-up request to check to see if the results are ready?
I'd also suggest including the underlying error string in the message since it might help users understand the issue more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought about it, again we need to keep process running till we wait. Also, log.Println("Job is scheduled on GitHub side"), this string is coming from Library example of handling the error., Please see this README, https://github.com/google/go-github
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From the github client README about the Accepted Error:
Methods known to behave like this are documented specifying this behavior.
We should check to see if the methods we're calling can emit this error. If so, we should look into what it means and what (if any) follow-up calls we can make to get the data one the processing on the github side has completed.
cmd/sdkstats/main.go
Outdated
| if _, ok := err.(*github.AcceptedError); ok { | ||
| log.Println("Job is scheduled on GitHub side") | ||
| } else if _, ok := err.(*github.RateLimitError); ok { | ||
| log.Println("Rate Limit has reached.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto about printing the error string from the rate limit error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, log.Println("Rate Limit has reached."), this string is coming from Library example of handling the error. Please see below link https://github.com/google/go-github README
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think those are meant to be simple examples. You can see the actual error text in the code here.
When printing an error, it is conventional to add some context and then print the underlying error, like this:
fmt.Printf("Could not fetch GitHub stats: %v\n", err)
Until we look into how to handle the AcceptedError, we should do something similar for that, IMO.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, Will include the original ERR. Also, On that note, I am still getting the same error. Will investigate more.
cmd/sdkstats/main.go
Outdated
| fileName := r.ProjectType + ".json" | ||
| fmt.Println("Total count: ", len(stats)) | ||
| file, _ := json.MarshalIndent(stats, "", " ") | ||
| _ = ioutil.WriteFile(fileName, file, 0644) | ||
| fmt.Println("Results are written in ", fileName) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than doing a file per project type, it would be nice to combine everything and print it to standard out. Then, if users want it in a file, they can always pipe it into one.
Maybe we update GetStats() to accept the entire slice of queries and return the combined results as a flat slice. Then we can iterate over the returned results to compute our final stats. How does that sound?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As fas as main.go goes, I took our requirements into consideration, as we were talking about ProjectType based breakdown, I have written into corresponding files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please see my above note, and let me know what you think
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think we miscommunicated a little about ProjectType breakdown. I just meant that one of the stats that would be interesting is the number of projects of each project type, not that we should split stats into separate files. Sorry about that!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No Problem, this has been evolving anyways, I can make that change.
cmd/sdkstats/main.go
Outdated
| } | ||
|
|
||
| //Parse the given Code result to search Text Matches for Version number. | ||
| func (p projectVersionParser) ParseVersion(codeResults github.CodeResult, projectType string) (string, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think you need projectType in the function signature, since you're passing an implementation of this into GetStats as part of a RepoMetadataQuery that already includes the project type.
I would suggest first writing a separate implementation of this interface for each query, and then looking to see if there are ways to re-use.
For example, the helm and ansible version parser implementations are basically identical other than a certain anchor you're looking for (e.g. the image name). So maybe you end up with something like this:
type baseImageVersionParser struct {
baseImage string
}
ansibleVersionParser := baseImageVersionParser{baseImage: "quay.io/operator-framework/ansible-operator"}
helmVersionParser := baseImageVersionParser{baseImage: "quay.io/operator-framework/helm-operator"}For the Go projects, maybe it makes sense to have a goModVersionParser and a gopkgTomlVersionParser, where you would end up making separate queries to use each of them. Then you could use your existing implementation for go.mod and then have the Gopkg.toml version parser just return "unknown", nil or similar.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure., I completely understand this. Since "Gopkg.toml" results , would not need ParseVersion, as they do not have defined search Query, I am defaulting sdkVersion to "N/A". Hence, I thought its unnecessary to make function calls to ParseVersion from sdkstats.go for below condition , if strings.Contains(q, "Gopkg.toml") {
skipVersion = 1
}
"I don't think you need projectType in the function signature, since you're passing an implementation of this into GetStats as part of a RepoMetadataQuery that already includes the project type."
So, You are saying its better to have separate ParseFunction defined, rather than passing "ProjectType", from sdkstats? Or May be I use which is available in Main.go, Not sure if that thread will still give me current ProjectType value passed into sdkstats.go.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Think of it from a library perspective and pretend you don't know what queries other users will be sending in. In that case, all the library can do is use the provided version parser.
So then from our perspective in main.go, think of things in terms of the API we have available, which is the VersionParser interface. In the case of Gopkg.toml, we're unable to parse a version so we define a version parser whose sole job is to return "unknown", nil.
type unknownVersionParser struct{}
func (_ unknownVersionParser) ParseVersion(_ github.CodeResult) (string, error) {
return "unknown", nil
}Since that means we have different implementations for parsing go.mod and Gopkg.toml, it probably means we need to split our ProjectType: "go" queries into two ProjectType: "go" queries like this:
queries := []sdkstats.RepoMetadataQuery{
sdkstats.RepoMetadataQuery{
ProjectType: "ansible",
Queries: []string{"filename:Dockerfile quay.io/operator-framework/ansible-operator"},
VersionParser: &baseImageVersionParser{
baseImage: "quay.io/operator-framework/ansible-operator",
},
},
sdkstats.RepoMetadataQuery{
ProjectType: "helm",
Queries: []string{"filename:Dockerfile quay.io/operator-framework/helm-operator"},
VersionParser: &baseImageVersionParser{
baseImage: "quay.io/operator-framework/helm-operator",
},
},
sdkstats.RepoMetadataQuery{
ProjectType: "go",
Queries: []string{
"filename:go.mod github.com/operator-framework/operator-sdk",
},
VersionParser: &goModVersionParser{
module: "github.com/operator-framework/operator-sdk",
},
},
sdkstats.RepoMetadataQuery{
ProjectType: "go",
Queries: []string{
"filename:Gopkg.toml github.com/operator-framework/operator-sdk",
},
VersionParser: &unknownVersionParser{},
},
}There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with separating the queries for "go.mod", and "Gopkg.toml". I was having them as separate queries in the beginning as well, to make VersionParse easier. Will make the changes.
pkg/sdkstats/sdkstats.go
Outdated
| for j := 0; j < len(searchOp); j++ { | ||
| for i := 0; i < len(searchOp[j].CodeResults); i++ { | ||
| if skipVersion == 1 { | ||
| sdkVersion = "N/A" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is starting to come into shape as a general purpose library, so should we use version as the variable name?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dont think I understood this, Can you please elaborate?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I'm just saying, lets change the variable name from "sdkVersion" to "version" now that this library his become much more extensible and usable for totally different projects.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will do.
pkg/sdkstats/sdkstats.go
Outdated
|
|
||
| repoMap := map[string]string{} | ||
|
|
||
| //GetRepoDetails Returns []RepoMetaData with Stars,CreatedAt,PushedAt, and TotalCommits details for a given owner, and repo |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A couple of nits, not super important, but good to instill good habits (this is as much a reminder for me as is it for anyone else reading this 🙂)
- Include a space after the comment marker
// - The first word of a comment that describes an identifier should be the identifier, so good to go there 👍
- Use full sentences that end with a period
- It's documentation for humans, so write what seems most natural and helpful.
With that in mind, perhaps this:
| //GetRepoDetails Returns []RepoMetaData with Stars,CreatedAt,PushedAt, and TotalCommits details for a given owner, and repo | |
| // GetRepoDetails uses client to look up repository metadata about repo. | |
| // The returned RepoMetadata struct includes details such as the | |
| // repository's stars, timestamps when it was created and most recently | |
| // pushed to, and the total number of commits to the repo. If an error | |
| // occurs, *RepoMetadata will be nil, and the error will be non-nil. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds reasonable, will include.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another point is that it is most important to pay attention to the doc conventions on exported functions. Since that's what users of the library will see.
Comments on unexported functions are still nice, but are only helpful to other developers and maintainers of the library.
pkg/sdkstats/sdkstats.go
Outdated
| ctx := context.Background() | ||
|
|
||
| repos, _, err := client.Repositories.Get(ctx, owner, name) | ||
| repos, _, err := client.Repositories.Get(ctx, repoDetails.Owner, repoDetails.Name) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Should repos be plural or singular? It seems like this function returns one repo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
:D Yes, It seems good to name it repo.
pkg/sdkstats/sdkstats.go
Outdated
|
|
||
| //GetRepoDetails Returns []RepoMetaData with Stars,CreatedAt,PushedAt, and TotalCommits details for a given owner, and repo | ||
| func GetRepoDetails(client *github.Client, repoDetails *RepoMetadata) (*RepoMetadata, error) { | ||
| ctx := context.Background() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In library code, it's best to let users pass their own context. The convention is to make the first argument in exported functions be the context, so this would be:
func GetRepoDetails(ctx context.Context, client *github.Client, repoDetails *RepoMetadata) (*RepoMetadata, error) {Since we call GetRepoDetails from another function in this library, we should also update that function's signature as well, and so on, until we reach a function that is not called anywhere else.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will do.
pkg/sdkstats/sdkstats.go
Outdated
| return nil, err | ||
| } | ||
| //GetStats returns []RepoMetaData populated with Code results from Github Search Code API, for a given search string | ||
| func GetStats(client *github.Client, rq RepoMetadataQuery) ([]RepoMetadata, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the only function used in main.go, so that's probably a good indication that the other functions (e.g. Search and GetRepoDetails) do not need to be exported and made public. To unexport the other functions, simply make the the first letter of their name lowercase.
An extremely important aspect of maintaining libraries is to be very careful and deliberate about designing and changing their public API surface. Any breaking changes to a library make life more difficult for users of the library, so it's best to keep the API surface as small and intuitive as possible.
Later versions of libraries can always add new functionality and expose more parts of the API surface without breaking existing users, but going the opposite direction and making already public APIs private is a breaking change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason I made these exportable, is that, In case other users have the need to let say only call "Search()", or "GetRepoDetails", It would be simple enough to call them. WDYT. This also makes case, NOT to pass "CodeResult" to GetRepoDetails(). WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will make them unexportable based on this comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See my comment here
|
Looks like this PR has some conflicts with |
Code review changes for https://github.com/learnoperators/go-github-m…
Exception handling added in sdkstats.go
cmd/sdkstats/main.go
Outdated
| "log" | ||
| "strings" | ||
|
|
||
| "github.com/go-github-metrics-1/pkg/sdkstats" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line needs to match the actual repo path:
| "github.com/go-github-metrics-1/pkg/sdkstats" | |
| "github.com/learnoperators/go-github-metrics/pkg/sdkstats" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am guessing I cant refer to this until I merge DEV with MASTER on learnoperators/go-github-metrics
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you can. In your local environment, you can get this working in one of two ways:
- Move the project directory into
$GOPATH/src/github.com/learnoperators/go-github-metrics - Create
go.modfile in the project root withgo mod init github.com/learnoperators/go-github-metrics
I suggest option 2 since the SDK uses Go Modules and it is now the official dependency manager supported by Go's built-in tooling.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
cmd/sdkstats/main.go
Outdated
| fmt.Println(err) | ||
| stats, err := sdkstats.GetStats(ctx, tc, r) | ||
| if err != nil { | ||
| fmt.Printf("Failed to get stats for query %q %v\n", r, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
%q requires a string, not a struct. Now that r is a struct and the queries is a slice of strings, it would be better to use %v and r.Queries. Also the go convention is to always separate the context from the underlying error with a colon:
| fmt.Printf("Failed to get stats for query %q %v\n", r, err) | |
| fmt.Printf("Failed to get stats for queries %v: %v\n", r.Queries, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
cmd/sdkstats/main.go
Outdated
| if err != nil { | ||
| fmt.Printf("Failed to get stats for query %q %v\n", r, err) | ||
| } | ||
| collectStats = append(collectStats, stats) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reason that we need a slice of slices? I think we can use this:
collectStats := []sdkstats.RepoMetadata{}
...
collectStats = append(collectStats, stats...)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cannot use stats (type []sdkstats.RepoMetadata) as type sdkstats.RepoMetadata in appendgo
I see this error, when I try to make the change as suggested.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that means you need the ellipses after stats. This doc has some good examples: https://programming.guide/go/append-explained.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
cmd/sdkstats/main.go
Outdated
| } | ||
| fileName := "Search_Results.json" | ||
| file, _ := json.MarshalIndent(collectStats, "", " ") | ||
| _ = ioutil.WriteFile(fileName, file, 0644) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ioutil.WriteFile returns an error. We need to make sure it returns nil or handle the error if not nil.
Ditto for MarshalIndent on the line above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
cmd/sdkstats/main.go
Outdated
| ) | ||
| tc := oauth2.NewClient(ctx, ts) | ||
| client := github.NewClient(tc) | ||
| //client := github.NewClient(tc) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think a better solution here would be:
// Defined in pkg/sdkstats/sdkstats.go
type Client struct {
client *github.Client
}
func (c Client) GetStats(ctx context.Contect, queries []RepoMetadataQuery) ([]RepoMetadata, error) {
// c.search(ctx, q)
}
func (c Client) search(ctx context.Context, query string) ([]*github.CodeSearchResult, error) {
// c.Client.Search.Code(ctx, query, opts)
}And then in this file, you could do:
client := sdkstats.Client{Client: github.NewClient(tc)}
stats, err := client.GetStats(ctx, queries)Re-creating the client each time in search() and getRepoDetails() might actually have a negative performance impact.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
cmd/sdkstats/main.go
Outdated
| "filename:go.mod github.com/operator-framework/operator-sdk", | ||
| }, | ||
| VersionParser: &gomodVersionParser{ | ||
| searchQ: "replace github.com/operator-framework/operator-sdk => github.com/operator-framework/operator-sdk v", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should not assume the SDK version will be in it's own replace statement. It could look like any of the following:
# Option 1:
replace github.com/operator-framework/operator-sdk => github.com/operator-framework/operator-sdk <version>
# Option 2
replace (
...
github.com/operator-framework/operator-sdk => github.com/operator-framework/operator-sdk <version>
...
)
# Option 3
require (
...
github.com/operator-framework/operator-sdk <version>
...
)
With these three options in mind, the pattern is:
<repo><whitespace><version><newline>
We can probably craft a regex for this as well.
cmd/sdkstats/main.go
Outdated
|
|
||
| // Parse the given Code result to search Text Matches for Version number. | ||
| func (p unknownVersionParser) ParseVersion(codeResults github.CodeResult) (string, error) { | ||
| version := "N/A" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: This isn't really "N/A" so much as "unknown" right? Gopkg.toml files DO have the SDK version in them, but we're just not able to parse it.
Also, there's no need to declare a variable and then return it. It's probably better to do return "unknown", nil.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
pkg/sdkstats/sdkstats.go
Outdated
| TotalCommits: TotalCommits, | ||
| func GetStats(ctx context.Context, tc *http.Client, rq RepoMetadataQuery) ([]RepoMetadata, error) { | ||
| var repoList []RepoMetadata | ||
| var repoDetails *RepoMetadata |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
repoDetails doesn't need to be declared at this scope since it is only used in the loop.
In the loop, you can use the following to declare and initialize in a single statement:
repoDetails, err := getRepoDetails(...)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
pkg/sdkstats/sdkstats.go
Outdated
| for i := 0; i < len(searchOp[j].CodeResults); i++ { | ||
| repoDetails, err = getRepoDetails(ctx, tc, searchOp[j].CodeResults[i], rq) | ||
| if err != nil { | ||
| return repoList, err |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an example of a situation where we may not want to exit the loop if one of the queries fails. If this function is a struct method on a new Client struct, we could add a Log function field to the struct and use that to log a message here:
type Client struct {
...
Log func(format string, fields ...interface{}) (int, error)
...
}
repoDetails, err := getRepoDetails(...)
if err != nil {
c.Log("Failed to get repo details for %q: %v\n", searchOp[j].CodeResults[i].GetRepository().GetFullName(), err)
// Keep processing the other results
continue
}| } | ||
| if projectType == "go.mod" { | ||
| searchQ = "github.com/operator-framework/operator-sdk v0." | ||
| for _, r := range commits { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To make it clearer, declare totalCommits := 0 right above this loop rather than at the beginning of the function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
pkg/sdkstats/sdkstats.go
Outdated
| } | ||
| for j := 0; j < len(searchOp); j++ { | ||
| for i := 0; i < len(searchOp[j].CodeResults); i++ { | ||
| repoDetails, err = getRepoDetails(ctx, tc, searchOp[j].CodeResults[i], rq) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like we're calling getRepoDetails() more than once per repository, which unnecessarily hits the Github API, driving up the query count and making things hit the rate limit faster.
We should refactor things so that we only call getRepoDetails once per repo.
Since a repo might have more than one operator built into it, maybe we call need RepoMetadata to have a []string for Versions, and then we coul call getRepoDetails() once, and iterate the code results calling rq.VersionParser.ParseVersion() and appending to the Versions field.
WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/hold
joelanford
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It sounds like you've made some changes (based on your "done" comments), but I don't see them in the review. Can you push these updates to the dev branch?
cmd/sdkstats/main.go
Outdated
| "log" | ||
| "strings" | ||
|
|
||
| "github.com/go-github-metrics-1/pkg/sdkstats" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you can. In your local environment, you can get this working in one of two ways:
- Move the project directory into
$GOPATH/src/github.com/learnoperators/go-github-metrics - Create
go.modfile in the project root withgo mod init github.com/learnoperators/go-github-metrics
I suggest option 2 since the SDK uses Go Modules and it is now the official dependency manager supported by Go's built-in tooling.
cmd/sdkstats/main.go
Outdated
| // Parse the given Code result to search Text Matches for Version number. | ||
| func (p baseVersionParser) ParseVersion(codeResults github.CodeResult) (string, error) { | ||
| baseImageRegex := regexp.QuoteMeta(p.searchQ) | ||
| versionRegex := strings.Trim(`(:([^s]+\n))?`, "\n") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you want the \n newline character to be appended after the version capture group, basically after the very end after the ?. Also don't think you need strings.Trim() here.
Added regex for gomodversionparser
| "golang.org/x/oauth2" | ||
| ) | ||
|
|
||
| type helmVersionParser struct{} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As a good practice, we should have basic info in the README in order to allow any person knows how to configure using the project. It could be another PR, however, how we can review this one without the knowledge to test it locally?
| queries = append(queries, "filename:go.mod github.com/operator-framework/operator-sdk") | ||
| queries = append(queries, "filename:Gopkg.toml github.com/operator-framework/operator-sdk") | ||
| queries := []sdkstats.RepoMetadataQuery{ | ||
| /*sdkstats.RepoMetadataQuery{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the code is not useful, so could be removed. WDYT?
| searchLatest string | ||
| } | ||
| type gomodVersionParser struct { | ||
| searchQ string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, would be great have in the README an overview. As for example, describing the purpose of it. After I checked the code shows that we are looking for getting metrics over projects published in GitHub which were done using Operator-SDK. Am I right?
The starts, commits and etc are nor over operator-sdk repo/project. Am I right?
Code review changes made, by adding below
--RepoMetadataQuertry struct, and Version Parsser interface.
--Added ERR check for GetStats func call in main.go
--Added skiVersion Flag to skip ParseVersion func calls when Query contains
Gopkg.toml"