-
Notifications
You must be signed in to change notification settings - Fork 38
feat(actor): adds actor types generation #1000
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
l2ysho
wants to merge
3
commits into
master
Choose a base branch
from
993-implement-a-command-for-generating-type-definitions-from-schemas
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+410
−29
Draft
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,6 +12,7 @@ _mytests | |
| .idea | ||
| .zed | ||
| docs/changelog.md | ||
| .generated | ||
|
|
||
| # Yarn files | ||
| .yarn/install-state.gz | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| import { mkdir, writeFile } from 'node:fs/promises'; | ||
| import path from 'node:path'; | ||
| import process from 'node:process'; | ||
|
|
||
| import type { JSONSchema4 } from 'json-schema'; | ||
| import { compile } from 'json-schema-to-typescript'; | ||
|
|
||
| import { ApifyCommand } from '../../lib/command-framework/apify-command.js'; | ||
| import { Args } from '../../lib/command-framework/args.js'; | ||
| import { Flags } from '../../lib/command-framework/flags.js'; | ||
| import { LOCAL_CONFIG_PATH } from '../../lib/consts.js'; | ||
| import { readAndValidateInputSchema } from '../../lib/input_schema.js'; | ||
| import { success } from '../../lib/outputs.js'; | ||
|
|
||
| export const BANNER_COMMENT = ` | ||
| /* eslint-disable */ | ||
| /* biome-ignore-all lint */ | ||
| /* biome-ignore-all format */ | ||
| /* prettier-ignore-start */ | ||
| /** | ||
| * This file was automatically generated by json-schema-to-typescript. | ||
| * DO NOT MODIFY IT BY HAND. Instead, modify the source JSONSchema file, | ||
| * and run apify actor generate-types to regenerate this file. | ||
| */ | ||
| `; | ||
|
|
||
| export class ActorGenerateTypesCommand extends ApifyCommand<typeof ActorGenerateTypesCommand> { | ||
| static override name = 'generate-types' as const; | ||
|
|
||
| static override description = `Generate TypeScript types from an Actor input schema. | ||
|
|
||
| Reads the input schema from one of these locations (in priority order): | ||
| 1. Object in '${LOCAL_CONFIG_PATH}' under "input" key | ||
| 2. JSON file path in '${LOCAL_CONFIG_PATH}' "input" key | ||
| 3. .actor/INPUT_SCHEMA.json | ||
| 4. INPUT_SCHEMA.json | ||
|
|
||
| Optionally specify custom schema path to use.`; | ||
|
|
||
| static override flags = { | ||
| output: Flags.string({ | ||
| char: 'o', | ||
| description: 'Directory where the generated files should be outputted.', | ||
| required: false, | ||
| default: './.generated/actor/', | ||
| }), | ||
| strict: Flags.boolean({ | ||
| char: 's', | ||
| description: 'Whether generated interfaces should be strict (no index signature [key: string]: unknown).', | ||
| required: false, | ||
| default: true, | ||
| }), | ||
| }; | ||
|
|
||
| static override args = { | ||
| path: Args.string({ | ||
| required: false, | ||
| description: 'Optional path to the input schema file. If not provided, searches default locations.', | ||
| }), | ||
| }; | ||
|
|
||
| async run() { | ||
| const { inputSchema, inputSchemaPath } = await readAndValidateInputSchema({ | ||
| forcePath: this.args.path, | ||
| cwd: process.cwd(), | ||
| action: 'Generating types from', | ||
| }); | ||
|
Comment on lines
+63
to
+67
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. generate-type share a validation logic with validate-schema now |
||
|
|
||
| const name = inputSchemaPath ? path.basename(inputSchemaPath, path.extname(inputSchemaPath)) : 'input'; | ||
|
|
||
| const result = await compile(inputSchema as JSONSchema4, name, { | ||
l2ysho marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| bannerComment: BANNER_COMMENT, | ||
| maxItems: -1, | ||
| unknownAny: true, | ||
| format: true, | ||
| additionalProperties: !this.flags.strict, | ||
| $refOptions: { resolve: { external: false, file: false, http: false } }, | ||
| }); | ||
|
|
||
| const outputDir = path.resolve(process.cwd(), this.flags.output); | ||
| await mkdir(outputDir, { recursive: true }); | ||
|
|
||
| const outputFile = path.join(outputDir, `${name}.ts`); | ||
| await writeFile(outputFile, result, 'utf-8'); | ||
|
|
||
| success({ message: `Generated types written to ${outputFile}` }); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| { | ||
| "$schema": "https://apify-projects.github.io/actor-json-schemas/input.ide.json?v=0.6", | ||
| "title": "Example Actor Input Schema", | ||
| "description": "A sample input schema demonstrating different input field types.", | ||
| "type": "object", | ||
| "schemaVersion": 1, | ||
| "properties": { | ||
| "startUrls": { | ||
| "title": "Start URLs", | ||
| "description": "List of URLs to start crawling", | ||
| "type": "array", | ||
| "editor": "requestListSources", | ||
| "prefill": [ | ||
| { "url": "https://example.com" }, | ||
| { "url": "https://example.org" } | ||
| ] | ||
| }, | ||
| "searchQuery": { | ||
| "title": "Search query", | ||
| "description": "The keyword or phrase to search for", | ||
| "type": "string", | ||
| "editor": "textfield", | ||
| "minLength": 3, | ||
| "default": "apify" | ||
| }, | ||
| "maxItems": { | ||
| "title": "Maximum items to fetch", | ||
| "description": "Limit the number of items the Actor will process", | ||
| "type": "integer", | ||
| "editor": "number", | ||
| "minimum": 1, | ||
| "default": 100 | ||
| }, | ||
| "includeImages": { | ||
| "title": "Include images", | ||
| "description": "Whether to include image data in the results", | ||
| "type": "boolean", | ||
| "editor": "checkbox", | ||
| "default": false | ||
| }, | ||
| "crawlerType": { | ||
| "title": "Crawler type", | ||
| "description": "Select the crawling engine to use", | ||
| "type": "string", | ||
| "editor": "select", | ||
| "enum": ["cheerio", "puppeteer", "playwright"], | ||
| "enumTitles": [ | ||
| "Cheerio crawler", | ||
| "Puppeteer browser", | ||
| "Playwright browser" | ||
| ], | ||
| "default": "cheerio" | ||
| }, | ||
| "proxyConfig": { | ||
| "title": "Proxy configuration", | ||
| "description": "Optional proxy settings to use while crawling", | ||
| "type": "object", | ||
| "editor": "json", | ||
| "properties": { | ||
| "useApifyProxy": { | ||
| "title": "Use Apify Proxy", | ||
| "description": "Enable Apify Proxy", | ||
| "type": "boolean" | ||
| }, | ||
| "customProxyUrls": { | ||
| "title": "Custom proxy URLs", | ||
| "description": "List of custom proxy URLs", | ||
| "type": "array", | ||
| "editor": "json", | ||
| "items": { | ||
| "type": "string" | ||
| } | ||
| } | ||
| }, | ||
| "required": ["useApifyProxy"], | ||
| "additionalProperties": false | ||
| } | ||
| }, | ||
| "required": ["startUrls", "searchQuery"] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actor templates have typescript setup with only
srcas include, so you cannot import it right away and you have to adjust it. Maybe better idea is to generate it directly tosrc/.generated/actor/? cc @vladfranguThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this might be best to discuss with @B4nan and @patrikbraborec too. Or maybe we wanna make it like how prisma handles it? so you'd have
@apify/actorin node-modules which gets filled out by the cli?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think changed this in v7, because it produced a lot of side effects, e.g. with caching. But we could explore this as well, maybe for our use case it would be fine.
If we won't go that way, I am slightly in favor of putting this into the
srcfolder, the alternative would be dynamically appending the.generatedfolder toincludesin tsconfig. But that would mean you end up withdist/srcanddist/.generatedif I'm not mistaken, which is a bit ugly (but also rather irrelevant).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fwiw modifying tsconfig can be risky, people may have god knows what setups in their tsconfig files and we shouldn't randomly append our folder.
Can you help me understand what you mean by this? Did prisma change this or what are you referring to with v7?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, prisma changed that.
https://www.prisma.io/docs/orm/more/upgrade-guides/upgrading-versions/upgrading-to-prisma-7#prisma-schema-changes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just came back cuz I read that, holy heck I did not know.... But they also use the top level generated dir by def, interesting