-
Notifications
You must be signed in to change notification settings - Fork 21
Update feature added for headers command. #181
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
a9cf724
85701d1
23c1a5b
8d9e651
4f00f8a
ca5b9f9
29b6786
55011d0
a5c98c7
df33765
6eaaa52
43b901c
10b8e97
35676e0
8c3dad3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -3,8 +3,11 @@ | |||||||||
| This repo provides utilities for managing copyright headers and license files | ||||||||||
| across many repos at scale. | ||||||||||
|
|
||||||||||
| You can use it to add or validate copyright headers on source code files, add a | ||||||||||
| LICENSE file to a repo, report on what licenses repos are using, and more. | ||||||||||
| Features: | ||||||||||
| - Add or validate copyright headers on source code files | ||||||||||
| - Add and/or manage LICENSE files with git-aware copyright year detection | ||||||||||
| - Report on licenses used across multiple repositories | ||||||||||
| - Automate compliance checks in CI/CD pipelines | ||||||||||
|
|
||||||||||
| ## Getting Started | ||||||||||
|
|
||||||||||
|
|
@@ -33,7 +36,7 @@ Usage: | |||||||||
| copywrite [command] | ||||||||||
|
|
||||||||||
| Common Commands: | ||||||||||
| headers Adds missing copyright headers to all source code files | ||||||||||
| headers Adds missing copyright headers and updates existing headers' year information. | ||||||||||
| init Generates a .copywrite.hcl config for a new project | ||||||||||
| license Validates that a LICENSE file is present and remediates any issues if found | ||||||||||
|
|
||||||||||
|
|
@@ -62,8 +65,18 @@ scan all files in your repo and copyright headers to any that are missing: | |||||||||
| copywrite headers --spdx "MPL-2.0" | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| You may omit the `--spdx` flag if you add a `.copywrite.hcl` config, as outlined | ||||||||||
| [here](#config-structure). | ||||||||||
| The `copywrite license` command validates and manages LICENSE files with git-aware copyright years: | ||||||||||
|
|
||||||||||
| ```sh | ||||||||||
| copywrite license --spdx "MPL-2.0" | ||||||||||
| ``` | ||||||||||
|
|
||||||||||
| **Copyright Year Behavior:** | ||||||||||
| - **Start Year**: Auto-detected from config file and if not found defaults to repository's first commit | ||||||||||
| - **End Year**: Set to current year when an update is triggered (git history only determines if update is needed) | ||||||||||
| - **Update Trigger**: Git detects if source code file was modified since the copyright end year | ||||||||||
|
|
||||||||||
| You may omit the `--spdx` flag if you add a `.copywrite.hcl` config, as outlined [here](#config-structure). | ||||||||||
|
|
||||||||||
| ### `--plan` Flag | ||||||||||
|
|
||||||||||
|
|
@@ -72,6 +85,24 @@ performs a dry-run and will outline what changes would be made. This flag also | |||||||||
| returns a non-zero exit code if any changes are needed. As such, it can be used | ||||||||||
| to validate if a repo is in compliance or not. | ||||||||||
|
|
||||||||||
| ## Technical Details | ||||||||||
|
|
||||||||||
| ### Copyright Year Logic | ||||||||||
|
|
||||||||||
| **Source File Headers:** | ||||||||||
| - End year: Set to current year when file's source code is modified | ||||||||||
| - Git history determines if update is needed (compares file's last commit year to copyright end year) | ||||||||||
| - When triggered, end year updates to current year | ||||||||||
| - Ignores copyright header updates made to a file as it is not source code change. | ||||||||||
|
|
||||||||||
| **LICENSE Files:** | ||||||||||
| - End year: Set to current year when any project file is modified | ||||||||||
| - Git history determines if update is needed (compares repo's last commit year to copyright end year) | ||||||||||
| - When triggered, end year updates to current year | ||||||||||
| - Preserves historical accuracy for archived projects (no forced updates) | ||||||||||
|
|
||||||||||
| **Key Distinction:** Git history is used as a trigger to determine *whether* an update is needed, but the actual end year value is always set to the current year when an update occurs. | ||||||||||
|
|
||||||||||
| ## Config Structure | ||||||||||
|
|
||||||||||
| > :bulb: You can automatically generate a new `.copywrite.hcl` config with the | ||||||||||
|
|
@@ -99,8 +130,8 @@ project { | |||||||||
|
|
||||||||||
| # (OPTIONAL) Represents the year that the project initially began | ||||||||||
| # This is used as the starting year in copyright statements | ||||||||||
| # If set and different from current year, headers will show: "copyright_year, current_year" | ||||||||||
| # If set and same as current year, headers will show: "current_year" | ||||||||||
| # If set and different from current year, headers will show: "copyright_year, year-2" | ||||||||||
| # If set and same as year-2, headers will show: "copyright_year" | ||||||||||
| # If not set (0), the tool will auto-detect from git history (first commit year) | ||||||||||
| # If auto-detection fails, it will fallback to current year only | ||||||||||
|
Comment on lines
-102
to
136
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this is now (still) incorrect. I don't know if we want/need either of the two years to be configurable during updating (I think it would be nice for consistence). Currently they are always inferred during updating - i.e. there is no way of overriding them via configuration. For new additions, the end year is not configurable either. Lines 147 to 150 in 421a509
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As discussed privately, year-1 can still be overridden via config file. |
||||||||||
| # Default: 0 (auto-detect) | ||||||||||
|
|
||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,8 +6,14 @@ package cmd | |
| import ( | ||
| "fmt" | ||
| "os" | ||
| "path/filepath" | ||
| "runtime" | ||
| "strings" | ||
| "sync" | ||
| "sync/atomic" | ||
|
|
||
| "github.com/hashicorp/copywrite/addlicense" | ||
| "github.com/hashicorp/copywrite/licensecheck" | ||
| "github.com/hashicorp/go-hclog" | ||
| "github.com/jedib0t/go-pretty/v6/text" | ||
| "github.com/samber/lo" | ||
|
|
@@ -21,9 +27,13 @@ var ( | |
|
|
||
| var headersCmd = &cobra.Command{ | ||
| Use: "headers", | ||
| Short: "Adds missing copyright headers to all source code files", | ||
| Short: "Adds missing copyright headers and updates existing headers' year information in all source code files", | ||
| Long: `Recursively checks for all files in the given directory and subdirectories, | ||
| adding copyright statements and license headers to any that are missing them. | ||
| adding copyright statements and license headers to any that are missing them and | ||
| updating the year information in existing headers based on git history. | ||
|
|
||
| By default, the command will modify files in place. To perform a dry-run without | ||
| modifying any files, use the --plan flag. | ||
|
|
||
| Autogenerated files and common file types that don't support headers (e.g., prose) | ||
| will automatically be exempted. Any other files or folders should be added to the | ||
|
|
@@ -87,10 +97,23 @@ config, see the "copywrite init" command.`, | |
| ".github/workflows/**", | ||
| ".github/dependabot.yml", | ||
| "**/node_modules/**", | ||
| ".copywrite.hcl", | ||
| } | ||
| ignoredPatterns := lo.Union(conf.Project.HeaderIgnore, autoSkippedPatterns) | ||
|
|
||
| // Construct the configuration addLicense needs to properly format headers | ||
| // STEP 1: Update existing copyright headers | ||
| gha.StartGroup("Updating existing copyright headers:") | ||
| updatedCount, anyFileUpdated, licensePath := updateExistingHeaders(cmd, ignoredPatterns, plan) | ||
| gha.EndGroup() | ||
| if updatedCount > 0 { | ||
| if plan { | ||
| cmd.Printf("\n%s\n\n", text.FgYellow.Sprintf("[DRY RUN] Would update %d file(s) with new copyright years", updatedCount)) | ||
| } else { | ||
| cmd.Printf("\n%s\n\n", text.FgGreen.Sprintf("Successfully updated %d file(s) with new copyright years", updatedCount)) | ||
| } | ||
| } | ||
|
|
||
| // STEP 2: Construct the configuration addLicense needs to properly format headers | ||
| licenseData := addlicense.LicenseData{ | ||
| Year: conf.FormatCopyrightYears(), // Format year(s) for copyright statements | ||
| Holder: conf.Project.CopyrightHolder, | ||
|
|
@@ -112,10 +135,33 @@ config, see the "copywrite init" command.`, | |
| // cobra.CheckErr on the return, which will indeed output to stderr and | ||
| // return a non-zero error code. | ||
|
|
||
| gha.StartGroup("The following files are missing headers:") | ||
| err := addlicense.Run(ignoredPatterns, "only", licenseData, "", verbose, plan, []string{"."}, stdcliLogger) | ||
| // STEP 3: Add missing headers | ||
| gha.StartGroup("Adding missing copyright headers:") | ||
| var err error | ||
| // In dry-run mode, if updateExistingHeaders found files that would be | ||
| // updated (year bumps), treat that as an error so the command exits | ||
| // non-zero to indicate work would be performed. | ||
| if plan && updatedCount > 0 { | ||
| err = fmt.Errorf("[DRY RUN] %d file(s) would be updated with new copyright years", updatedCount) | ||
| } | ||
| runErr := addlicense.Run(ignoredPatterns, "only", licenseData, "", verbose, plan, []string{"."}, stdcliLogger) | ||
| if err != nil && runErr != nil { | ||
| err = fmt.Errorf("%v; %v", err, runErr) | ||
| } else if err == nil { | ||
| err = runErr | ||
| } | ||
| gha.EndGroup() | ||
|
|
||
| // STEP 4: Update LICENSE file if any files were modified (either updated or added headers) | ||
| // In plan mode: if addlicense found missing headers (returns error), assume files would be modified | ||
| // In normal mode: if addlicense succeeded, assume files were modified | ||
| if runErr != nil || (!plan && runErr == nil) { | ||
| anyFileUpdated = true | ||
| } | ||
|
|
||
| updateLicenseFile(cmd, licensePath, anyFileUpdated, plan) | ||
|
|
||
| // Check for errors after LICENSE file update so we still show what would happen | ||
| cobra.CheckErr(err) | ||
| }, | ||
| } | ||
|
|
@@ -131,3 +177,125 @@ func init() { | |
| headersCmd.Flags().StringP("spdx", "s", "", "SPDX-compliant license identifier (e.g., 'MPL-2.0')") | ||
| headersCmd.Flags().StringP("copyright-holder", "c", "", "Copyright holder (default \"IBM Corp.\")") | ||
| } | ||
|
|
||
| // updateExistingHeaders walks through files and updates copyright headers based on config and git history | ||
| // Returns the count of updated files, a boolean indicating if any file was updated, and the LICENSE file path (if found) | ||
| func updateExistingHeaders(cmd *cobra.Command, ignoredPatterns []string, dryRun bool) (int, bool, string) { | ||
| targetHolder := conf.Project.CopyrightHolder | ||
| if targetHolder == "" { | ||
| targetHolder = "IBM Corp." | ||
| } | ||
|
|
||
| configYear := conf.Project.CopyrightYear | ||
| updatedCount := 0 | ||
| anyFileUpdated := false | ||
| var licensePath string | ||
|
|
||
| // Producer/consumer: walk files (producer) and process them with a bounded | ||
| // worker pool (consumers). This preserves existing semantics while | ||
| // bounding concurrency and allowing the walk to run ahead of processors. | ||
| ch := make(chan string, 1000) | ||
|
|
||
| var wg sync.WaitGroup | ||
| var updatedCount64 int64 | ||
| var anyFileUpdatedFlag int32 | ||
| var mu sync.Mutex | ||
|
|
||
| workers := runtime.NumCPU() * 4 | ||
| if workers < 2 { | ||
| workers = 2 | ||
| } | ||
|
|
||
| // Start worker pool | ||
| wg.Add(workers) | ||
| for i := 0; i < workers; i++ { | ||
| go func() { | ||
| defer wg.Done() | ||
| for path := range ch { | ||
| // capture base and skip LICENSE files here as well | ||
| base := filepath.Base(path) | ||
| if strings.EqualFold(base, "LICENSE") || strings.EqualFold(base, "LICENSE.TXT") || strings.EqualFold(base, "LICENSE.MD") { | ||
| mu.Lock() | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why was a lock needed here ? Wouldn't all the workers update |
||
| if licensePath == "" { | ||
| licensePath = path | ||
| } | ||
| mu.Unlock() | ||
| continue | ||
| } | ||
|
|
||
| if !dryRun { | ||
| updated, err := licensecheck.UpdateCopyrightHeader(path, targetHolder, configYear, false) | ||
| if err == nil && updated { | ||
| cmd.Printf(" %s\n", path) | ||
| atomic.AddInt64(&updatedCount64, 1) | ||
| atomic.StoreInt32(&anyFileUpdatedFlag, 1) | ||
| } | ||
| } else { | ||
| needsUpdate, err := licensecheck.NeedsUpdate(path, targetHolder, configYear, false) | ||
| if err == nil && needsUpdate { | ||
| cmd.Printf(" %s\n", path) | ||
| atomic.AddInt64(&updatedCount64, 1) | ||
| atomic.StoreInt32(&anyFileUpdatedFlag, 1) | ||
| } | ||
| } | ||
| } | ||
| }() | ||
| } | ||
|
|
||
| // Producer: walk the tree and push files onto the channel | ||
| go func() { | ||
| _ = filepath.Walk(".", func(path string, info os.FileInfo, err error) error { | ||
| if err != nil || info.IsDir() { | ||
| return nil | ||
| } | ||
|
|
||
| // Check if file should be ignored | ||
| if addlicense.FileMatches(path, ignoredPatterns) { | ||
| return nil | ||
| } | ||
|
|
||
| // Non-ignored file -> enqueue for processing. If channel is full, | ||
| // this will block until a worker consumes entries, which is fine. | ||
| ch <- path | ||
| return nil | ||
| }) | ||
| close(ch) | ||
| }() | ||
|
|
||
| // wait for workers to finish | ||
| wg.Wait() | ||
|
|
||
| // finalize counts | ||
| updatedCount = int(atomic.LoadInt64(&updatedCount64)) | ||
| anyFileUpdated = atomic.LoadInt32(&anyFileUpdatedFlag) != 0 | ||
|
|
||
| return updatedCount, anyFileUpdated, licensePath | ||
| } | ||
|
|
||
| // updateLicenseFile updates the LICENSE file with current year if any files were modified | ||
| func updateLicenseFile(cmd *cobra.Command, licensePath string, anyFileUpdated bool, dryRun bool) { | ||
| // If no LICENSE file was found during the walk, nothing to do | ||
| if licensePath == "" { | ||
| return | ||
| } | ||
|
|
||
| targetHolder := conf.Project.CopyrightHolder | ||
| if targetHolder == "" { | ||
| targetHolder = "IBM Corp." | ||
| } | ||
|
|
||
| configYear := conf.Project.CopyrightYear | ||
|
|
||
| // Update LICENSE file, forcing current year if any file was updated | ||
| if !dryRun { | ||
| updated, err := licensecheck.UpdateCopyrightHeader(licensePath, targetHolder, configYear, anyFileUpdated) | ||
| if err == nil && updated { | ||
| cmd.Printf("\nUpdated LICENSE file: %s\n", licensePath) | ||
| } | ||
| } else { | ||
| needsUpdate, err := licensecheck.NeedsUpdate(licensePath, targetHolder, configYear, anyFileUpdated) | ||
| if err == nil && needsUpdate { | ||
| cmd.Printf("\n[DRY RUN] Would update LICENSE file: %s\n", licensePath) | ||
| } | ||
| } | ||
|
Comment on lines
+289
to
+300
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We discussed this privately already but just to remember, this will likely need updating to recognise our BUSL LICENSE files, e.g. https://github.com/hashicorp/terraform/blob/main/LICENSE
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This has been added to our backlog. We will initiate the conversation with legal and make necessary modifications to accommodate BUSL License updates for copyrights. C.C. - @CreatorHead @mallikabandaru |
||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change should probably be applied to the builtin help page too:
copywrite/cmd/headers.go
Line 24 in 421a509