Skip to content

feat: add database output support for storing scan results #2363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dogancanbakir wants to merge 4 commits into
base: dev
Choose a base branch
from
Open

feat: add database output support for storing scan results #2363

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2ea576f
feat: add database output support for storing scan results
dogancanbakir Jan 6, 2026
f2690f5
fix: address CodeRabbitAI review comments for database output
dogancanbakir Jan 6, 2026
c1c1fd0
refactor: use batcher utility and StringVarEnv
dogancanbakir Jan 12, 2026
96ac054
Merge branch 'dev' into feature/database-output
dogancanbakir Jan 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 52 additions & 46 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,42 +98,40 @@ INPUT:
-im, -input-mode string mode of input file (burp)

PROBES:
-sc, -status-code display response status-code
-cl, -content-length display response content-length
-ct, -content-type display response content-type
-location display response redirect location
-favicon display mmh3 hash for '/favicon.ico' file
-hash string display response body hash (supported: md5,mmh3,simhash,sha1,sha256,sha512)
-jarm display jarm fingerprint hash
-rt, -response-time display response time
-lc, -line-count display response body line count
-wc, -word-count display response body word count
-title display page title
-bp, -body-preview display first N characters of response body (default 100)
-server, -web-server display server name
-sc, -status-code display response status-code
-cl, -content-length display response content-length
-ct, -content-type display response content-type
-location display response redirect location
-favicon display mmh3 hash for '/favicon.ico' file
-hash string display response body hash (supported: md5,mmh3,simhash,sha1,sha256,sha512)
-jarm display jarm fingerprint hash
-rt, -response-time display response time
-lc, -line-count display response body line count
-wc, -word-count display response body word count
-title display page title
-bp, -body-preview display first N characters of response body (default 100)
-server, -web-server display server name
-td, -tech-detect display technology in use based on wappalyzer dataset
-cff, -custom-fingerprint-file string path to a custom fingerprint file for technology detection
-cpe display CPE (Common Platform Enumeration) based on awesome-search-queries
-wp, -wordpress display WordPress plugins and themes
-method display http request method
-ws, -websocket display server using websocket
-ip display host ip
-cname display host cname
-extract-fqdn, -efqdn get domain and subdomains from response body and header in jsonl/csv output
-asn display host asn information
-cdn display cdn/waf in use (default true)
-probe display probe status
-ws, -websocket display server using websocket
-ip display host ip
-cname display host cname
-extract-fqdn, -efqdn get domain and subdomains from response body and header in jsonl/csv output
-asn display host asn information
-cdn display cdn/waf in use (default true)
-probe display probe status

HEADLESS:
-ss, -screenshot enable saving screenshot of the page using headless browser
-system-chrome enable using local installed chrome for screenshot
-ho, -headless-options string[] start headless chrome with additional options
-esb, -exclude-screenshot-bytes enable excluding screenshot bytes from json output
-no-screenshot-full-page disable saving full page screenshot
-ehb, -exclude-headless-body enable excluding headless header from json output
-no-screenshot-full-page disable saving full page screenshot
-st, -screenshot-timeout value set timeout for screenshot in seconds (default 10s)
-sid, -screenshot-idle value set idle time before taking screenshot in seconds (default 1s)
-jsc, -javascript-code string[] execute JavaScript code after navigation
-jsc, -javascript-code string[] execute JavaScript code after navigation

MATCHERS:
-mc, -match-code string match response with specified status code (-mc 200,302)
Expand All @@ -143,7 +141,7 @@ MATCHERS:
-mfc, -match-favicon string[] match response with specified favicon hash (-mfc 1494302000)
-ms, -match-string string[] match response with specified string (-ms admin)
-mr, -match-regex string[] match response with specified regex (-mr admin)
-mcdn, -match-cdn string[] match host with specified cdn provider (cloudfront, fastly, google)
-mcdn, -match-cdn string[] match host with specified cdn provider (cloudfront, fastly, google, etc.)
-mrt, -match-response-time string match response with specified response time in seconds (-mrt '< 1')
-mdc, -match-condition string match response with dsl expression condition

Expand All @@ -152,19 +150,21 @@ EXTRACTOR:
-ep, -extract-preset string[] display response content matched by a pre-defined regex (url,ipv4,mail)

FILTERS:
-fc, -filter-code string filter response with specified status code (-fc 403,401)
-fep, -filter-error-page filter response with ML based error page detection
-fd, -filter-duplicates filter out near-duplicate responses (only first response is retained)
-fl, -filter-length string filter response with specified content length (-fl 23,33)
-flc, -filter-line-count string filter response body with specified line count (-flc 423,532)
-fwc, -filter-word-count string filter response body with specified word count (-fwc 423,532)
-ffc, -filter-favicon string[] filter response with specified favicon hash (-ffc 1494302000)
-fs, -filter-string string[] filter response with specified string (-fs admin)
-fe, -filter-regex string[] filter response with specified regex (-fe admin)
-fcdn, -filter-cdn string[] filter host with specified cdn provider (cloudfront, fastly, google)
-frt, -filter-response-time string filter response with specified response time in seconds (-frt '> 1')
-fdc, -filter-condition string filter response with dsl expression condition
-strip strips all tags in response. supported formats: html,xml (default html)
-fc, -filter-code string filter response with specified status code (-fc 403,401)
-fep, -filter-error-page filter response with ML based error page detection
-fd, -filter-duplicates filter out near-duplicate responses (only first response is retained)
-fl, -filter-length string filter response with specified content length (-fl 23,33)
-flc, -filter-line-count string filter response body with specified line count (-flc 423,532)
-fwc, -filter-word-count string filter response body with specified word count (-fwc 423,532)
-ffc, -filter-favicon string[] filter response with specified favicon hash (-ffc 1494302000)
-fs, -filter-string string[] filter response with specified string (-fs admin)
-fe, -filter-regex string[] filter response with specified regex (-fe admin)
-fcdn, -filter-cdn string[] filter host with specified cdn provider (cloudfront, fastly, google, etc.)
-frt, -filter-response-time string filter response with specified response time in seconds (-frt '> 1')
-fdc, -filter-condition string filter response with dsl expression condition
-strip strips all tags in response. supported formats: html,xml (default html)
-lof, -list-output-fields list of fields to output (comma separated)
-eof, -exclude-output-fields string[] exclude output fields output based on a condition

RATE-LIMIT:
-t, -threads int number of threads to use (default 50)
Expand Down Expand Up @@ -202,10 +202,16 @@ OUTPUT:
-include-chain include redirect http chain in JSON output (-json only)
-store-chain include http redirect chain in responses (-sr only)
-svrc, -store-vision-recon-cluster include visual recon clusters (-ss and -sr only)
-pr, -protocol string protocol to use (unknown, http11)
-pr, -protocol string protocol to use (unknown, http11, http2, http3)
-fepp, -filter-error-page-path string path to store filtered error pages (default "filtered_error_page.json")
-lof, -list-output-fields list available output field names for filtering
-eof, -exclude-output-fields string[] exclude specified output fields from results
-rdb, -result-db store results in database
-rdbc, -result-db-config string path to database config file
-rdbt, -result-db-type string database type (mongodb, postgres, mysql)
-rdbcs, -result-db-conn string database connection string (env: HTTPX_DB_CONNECTION_STRING)
-rdbn, -result-db-name string database name (default "httpx")
-rdbtb, -result-db-table string table/collection name (default "results")
-rdbbs, -result-db-batch-size int batch size for database inserts (default 100)
-rdbor, -result-db-omit-raw omit raw request/response data from database
Comment on lines +205 to +214
Copy link

@coderabbitai coderabbitai bot Jan 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

# Search for protocol-related validation and constants
rg -n "ValidateOptions|protocol" -S -g'*.go' | head -50

Repository: projectdiscovery/httpx

Length of output: 4320

🏁 Script executed:

# Search for specific protocol values
rg -n "http3|http2|HTTP2|HTTP3|UNKNOWN|HTTP11|http11" -S -g'*.go' | head -100

Repository: projectdiscovery/httpx

Length of output: 2626

🏁 Script executed:

# Look for protocol type/enum definitions
rg -n "type.*Protocol|const.*Protocol" -S -g'*.go'

Repository: projectdiscovery/httpx

Length of output: 48

🏁 Script executed:

# Check README lines 205-214
sed -n '205,214p' README.md

Repository: projectdiscovery/httpx

Length of output: 927

🏁 Script executed:

# Get full ValidateOptions function to see complete validation logic
rg -A 50 "func (options \*Options) ValidateOptions" runner/options.go | head -70

Repository: projectdiscovery/httpx

Length of output: 48

🏁 Script executed:

# Get the ValidateOptions function more explicitly
sed -n '694,850p' runner/options.go

Repository: projectdiscovery/httpx

Length of output: 6007

Protocol validation does not support http2 and http3 despite documentation claiming they are available. ValidateOptions at runner/options.go:816 only accepts unknown and http11, but README.md lists all four protocols. The flag help text also incorrectly lists http2 and http3 as [experimental] options. Either enable these protocols in the validation logic or remove them from documentation.

🤖 Prompt for AI Agents 
In `@README.md` around lines 205 - 214, The README claims support for http2 and
http3 but ValidateOptions in runner/options.go (function ValidateOptions)
currently only allows "unknown" and "http11"; update the validation to accept
"http2" and "http3" (and mark experimental if needed) by adding those values to
the allowed protocol set in ValidateOptions, and ensure any flag help text/usage
strings (the protocol flag handling) reflect the new valid values;
alternatively, if you prefer to not enable them yet, remove http2 and http3 from
README/protocol flag help text so docs match the current ValidateOptions
behavior.


CONFIGURATIONS:
-config string path to the httpx configuration file (default $HOME/.config/httpx/config.yaml)
Expand All @@ -214,9 +220,9 @@ CONFIGURATIONS:
-deny string[] denied list of IP/CIDR's to process (file or comma separated)
-sni, -sni-name string custom TLS SNI name
-random-agent enable Random User-Agent to use (default true)
-auto-referer set the Referer header to the current URL (default false)
-auto-referer set the Referer header to the current URL
-H, -header string[] custom http headers to send with request
-http-proxy, -proxy string http proxy to use (eg http://127.0.0.1:8080)
-http-proxy, -proxy string proxy (http|socks) to use (eg http://127.0.0.1:8080)
-unsafe send raw requests skipping golang normalization
-resume resume scan using resume.cfg
-fr, -follow-redirects follow http redirects
Expand Down Expand Up @@ -252,14 +258,14 @@ DEBUG:

OPTIMIZATIONS:
-nf, -no-fallback display both probed protocol (HTTPS and HTTP)
-nfs, -no-fallback-scheme probe with protocol scheme specified in input
-nfs, -no-fallback-scheme probe with protocol scheme specified in input
-maxhr, -max-host-error int max error count per host before skipping remaining path/s (default 30)
-e, -exclude string[] exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex)
-retries int number of retries
-timeout int timeout in seconds (default 10)
-delay value duration between each http request (eg: 200ms, 1s) (default -1ns)
-rsts, -response-size-to-save int max response size to save in bytes (default 2147483647)
-rstr, -response-size-to-read int max response size to read in bytes (default 2147483647)
-rsts, -response-size-to-save int max response size to save in bytes (default 512000000)
-rstr, -response-size-to-read int max response size to read in bytes (default 512000000)

CLOUD:
-auth configure projectdiscovery cloud (pdcp) api key (default true)
Expand Down
63 changes: 63 additions & 0 deletions cmd/httpx/httpx.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (

"github.com/logrusorgru/aurora"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/httpx/internal/db"
"github.com/projectdiscovery/httpx/internal/pdcp"
"github.com/projectdiscovery/httpx/runner"
pdcpauth "github.com/projectdiscovery/utils/auth/pdcp"
Expand Down Expand Up @@ -64,6 +65,9 @@ func main() {
// setup optional asset upload
_ = setupOptionalAssetUpload(options)

// setup optional database output
_ = setupDatabaseOutput(options)

httpxRunner, err := runner.New(options)
if err != nil {
gologger.Fatal().Msgf("Could not create runner: %s\n", err)
Expand Down Expand Up @@ -143,3 +147,62 @@ func setupOptionalAssetUpload(opts *runner.Options) *pdcp.UploadWriter {
}
return writer
}

// setupDatabaseOutput sets up database output for storing results
// This is optional and only initialized when explicitly enabled via -rdb flag
func setupDatabaseOutput(opts *runner.Options) *db.Writer {
if !opts.ResultDatabase {
return nil
}

var cfg *db.Config
var err error

if opts.ResultDatabaseConfig != "" {
// Load configuration from file
cfg, err = db.LoadConfigFromFile(opts.ResultDatabaseConfig)
if err != nil {
gologger.Fatal().Msgf("Could not load database config: %s\n", err)
}
} else {
// Build configuration from CLI options
dbOpts := &db.Options{
Enabled: opts.ResultDatabase,
Type: opts.ResultDatabaseType,
ConnectionString: opts.ResultDatabaseConnStr,
DatabaseName: opts.ResultDatabaseName,
TableName: opts.ResultDatabaseTable,
BatchSize: opts.ResultDatabaseBatchSize,
OmitRaw: opts.ResultDatabaseOmitRaw,
}
cfg, err = dbOpts.ToConfig()
if err != nil {
gologger.Fatal().Msgf("Invalid database configuration: %s\n", err)
}
}

writer, err := db.NewWriter(context.Background(), cfg)
if err != nil {
gologger.Fatal().Msgf("Could not setup database output: %s\n", err)
}

// Chain with existing OnResult callback if present
existingCallback := opts.OnResult
opts.OnResult = func(r runner.Result) {
if existingCallback != nil {
existingCallback(r)
}
writer.GetWriterCallback()(r)
}

// Chain with existing OnClose callback if present
existingClose := opts.OnClose
opts.OnClose = func() {
writer.Close()
if existingClose != nil {
existingClose()
}
}

return writer
}
8 changes: 8 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ require (
require (
github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.0
github.com/dustin/go-humanize v1.0.1
github.com/go-sql-driver/mysql v1.9.3
github.com/lib/pq v1.10.9
go.mongodb.org/mongo-driver v1.17.6
github.com/go-viper/mapstructure/v2 v2.5.0
github.com/gocarina/gocsv v0.0.0-20240520201108-78e41c74b4b1
github.com/projectdiscovery/awesome-search-queries v0.0.0-20260104120501-961ef30f7193
Expand All @@ -62,6 +65,7 @@ require (

require (
aead.dev/minisign v0.2.0 // indirect
filippo.io/edwards25519 v1.1.0 // indirect
github.com/JohannesKaufmann/dom v0.2.0 // indirect
github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible // indirect
github.com/Masterminds/semver/v3 v3.2.1 // indirect
Expand Down Expand Up @@ -124,6 +128,7 @@ require (
github.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/montanaflynn/stats v0.7.1 // indirect
github.com/muesli/reflow v0.3.0 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646 // indirect
Expand Down Expand Up @@ -159,6 +164,9 @@ require (
github.com/ulikunitz/xz v0.5.15 // indirect
github.com/vulncheck-oss/go-exploit v1.51.0 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/xdg-go/scram v1.1.2 // indirect
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
github.com/ysmood/fetchup v0.2.3 // indirect
github.com/ysmood/goob v0.4.0 // indirect
github.com/ysmood/got v0.40.0 // indirect
Expand Down
Loading
Loading