Skip to content

fix(zip): make includeSources use allowlist behavior#2114

Open
kaigritun wants to merge 1 commit intowxt-dev:majorfrom
kaigritun:fix-zip-includeSources
Open

fix(zip): make includeSources use allowlist behavior#2114
kaigritun wants to merge 1 commit intowxt-dev:majorfrom
kaigritun:fix-zip-includeSources

Conversation

@kaigritun
Copy link

When includeSources patterns are provided, only files matching those patterns should be included in the sources ZIP. Previously, includeSources patterns were added to the default **/* glob, which could leak sensitive files that weren't explicitly excluded.

Changes per maintainer review (replaces #2111):

  • Move default ['**/*'] into resolveZipConfig
  • Pass excludeSources directly to glob's ignore option
  • Remove hardcoded node_modules glob (already in excludeSources default)
  • Simplify zipDir logic

BREAKING CHANGE: If you were using includeSources to add specific files to the default set (e.g., hidden files), you now need to include all files you want in the ZIP.

Fixes #2059
Supersedes #2111

When includeSources patterns are provided, only files matching those
patterns should be included in the sources ZIP. Previously, includeSources
patterns were added to the default **/* glob, which could leak sensitive
files that weren't explicitly excluded.

Changes per maintainer review:
- Move default ['**/*'] into resolveZipConfig
- Pass excludeSources directly to glob's ignore option
- Remove hardcoded node_modules glob (already in excludeSources default)
- Simplify zipDir logic

BREAKING CHANGE: If you were using includeSources to add specific files
to the default set (e.g., hidden files), you now need to include all
files you want in the ZIP.

Fixes wxt-dev#2059
Copy link
Member

@aklinker1 aklinker1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While reviewing, I remembered some past issues related to zipping sources that we need to support:

TLDR: We need to support including hidden files and files in hidden directories. I think this requirement was why the logic was so complex.

Right now, I don't think this PR supports adding hidden files at all.

I ran some tests to refresh my knowledge of fast-glob, verifying the three cases we care about (regular files, hidden files, regular files in hidden directories):

Details
import { glob, Options } from "fast-glob";
import fs from "node:fs/promises";
import { inspect } from "node:util";

const test = async (sources: string[], options: Options) => {
  const res = await glob(sources, options);
  console.log(
    `
───

glob(
  ${inspect(sources, { colors: true })},
  ${inspect(options, { colors: true })}
)
= ${inspect(res, { colors: true })}`,
  );
};

const allFiles = await fs.readdir(".", {
  recursive: true,
});
console.log("All files (minus node_modules):");
console.log(
  allFiles.filter((file: string) => !file.startsWith("node_modules")),
);

await test(["**/*"], { ignore: ["**/node_modules/**"] });
await test(["**/.*"], { ignore: ["**/node_modules/**"] });
await test(["**/.*/**/*"], { ignore: ["**/node_modules/**"] });

and the output:

All files (minus node_modules):
[ ".wxt", "bun.lock", "package.json", ".env", "index.ts", ".wxt/test.txt" ]

───

glob(
  [ '**/*' ],
  { ignore: [ '**/node_modules/**' ] }
)
= [ 'bun.lock', 'package.json', 'index.ts' ]

───

glob(
  [ '**/.*' ],
  { ignore: [ '**/node_modules/**' ] }
)
= [ '.env' ]

───

glob(
  [ '**/.*/**/*' ],
  { ignore: [ '**/node_modules/**' ] }
)
= []

But it seems like fast-glob doesn't support including files from inside hidden directories (I did try the dot: true option as well, but it didn't make a difference)...

So I tried glob instead:

Details
import { glob, GlobOptions } from "glob";
import fs from "node:fs/promises";
import { inspect } from "node:util";

const test = async (sources: string[], options: GlobOptions) => {
  const res = await glob(sources, options);
  console.log(
    `
───

glob(
  ${inspect(sources, { colors: true })},
  ${inspect(options, { colors: true })}
)
= ${inspect(res, { colors: true })}`,
  );
};

const allFiles = await fs.readdir(".", {
  recursive: true,
});
console.log("All files (minus node_modules):");
console.log(
  allFiles.filter((file: string) => !file.startsWith("node_modules")),
);

await test(["**/*"], { ignore: ["**/node_modules/**"] });
await test(["**/.*"], { ignore: ["**/node_modules/**"] });
await test(["**/.*/**/*"], { ignore: ["**/node_modules/**"] });
await test(["**/.*/**/*"], { ignore: ["**/node_modules/**"], nodir: true });
All files (minus node_modules):
[ ".wxt", "bun.lock", "package.json", ".env", "index.ts", ".wxt/test.txt" ]

───

glob(
  [ '**/*' ],
  { ignore: [ '**/node_modules/**' ] }
)
= [ 'index.ts', 'package.json', 'bun.lock' ]

───

glob(
  [ '**/.*' ],
  { ignore: [ '**/node_modules/**' ] }
)
= [ '.env', '.wxt' ]

───

glob(
  [ '**/.*/**/*' ],
  { ignore: [ '**/node_modules/**' ] }
)
= [ '.wxt/test.txt' ]

───

glob(
  [ '**/.*/**/*' ],
  { ignore: [ '**/node_modules/**' ], nodir: true }
)
= []

It works, but I can't get it to return just files. It does support returning with file-types, so we could filter out directories that way.

Unforunate since glob is 10x as large...

Maybe there's another glob library that's lighter-weight and would allow devs to include hidden files and files in hidden directories?

Otherwise I think we have to switch from fast-glob to glob.

import JSZip from 'jszip';
import glob from 'fast-glob';
import { normalizePath } from './utils/paths';
import { minimatchMultiple } from './utils/minimatch-multiple';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can delete this file and it's test completely, this was the only place they were used.

@aklinker1
Copy link
Member

Can you also update the docs, adding a section about this change to the docs here:

Maybe wait to do that until all the changes are finalized through.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants