Add support for tiled CT logs

[mimi89999]

No description provided.

[mimi89999]

I omitted some features to keep the size of the diff manageable. I plan to add them once this PR will be merged.

The missing features are:

• Recovery
• Specifying tiled logs in the config
• Fetching partial tiles

There are also some aspects that need discussion:

• The cert_link field in the data. Should it be removed completely? Should it contain a link to the tile?

[Knight1]

Is there anything which would prevent this to merge @d-Rickyy-b ?

[d-Rickyy-b]

Hi @Knight1, shame on me, I haven't found the time to check the PR yet. Same with the other open PRs and issues. Over the holidays I should have some time to check and test each of them thoroughly and merge them :)

PS: Main priority is stability, so I want to test what I am about to merge.

[mimi89999]

If you want to merge this, please let me know and I will rebase and update the PR

[d-Rickyy-b]

I am currently reviewing the PR. At first, thanks to @mimi89999 for the effort, and sorry for the long wait. I couldn't find the time sooner.

1.) One issue I spotted is related to generating the cert link that is contained in the cert's JSON we generate. Currently, this is only done for regular logs, not for tiled logs:

certstream-server-go/internal/certificatetransparency/ct-parser.go

Line 26 in e40aeff

certLink := fmt.Sprintf("%s/ct/v1/get-entries?start=%d&end=%d", ctURL, entry.Index, entry.Index)

From my understanding of the static CT monitoring API, there is no such thing as a direct link to a certificate anymore. Certs are grouped together in tiles, so we could add a link to the tile the cert is a part of. I am not yet sure what's the best solution for this.

2.) Apart from that, I think we should give clients the ability to identify if a cert came from a tiled log or from a regular RFC 6962 one. Therefor, I think adding a field "Type" to the Source struct should help. That can be used to identify if we're dealing with a tiled or RFC 6962 based log.

certstream-server-go/internal/certificatetransparency/ct-parser.go

Lines 33 to 38 in e40aeff

	Source: models.Source{
	Name: logName,
	URL: ctURL,
	Operator: operatorName,
	NormalizedURL: normalizeCtlogURL(ctURL),
	},

3.) While testing the PR locally, I added a new option to the config to manually specify custom TiledLogs. This also needed a modification of the getAllLogs method to process them accordingly. Those options are definitely needed in order to merge the PR.

4.) Since the RFC 6962 logs got their own buffer, we could also implement a buffer that buffers logs before adding them to the broadcast channel.

certstream-server-go/internal/certificatetransparency/ct-watcher.go

Line 400 in 9c59d27

BufferSize: config.AppConfig.General.BufferSizes.CTLog,

5.) Due to the log "Processed tile...", we're spamming a lot to stdout. I do appreciate the log and I think we should definitely add it back as debug output, after we implemented leveled logging.

6.) Also, there is currently no check for the validity of signatures of the content of tiled logs. Given the purpose of the tool, this doesn't sound too critical, but we might add something like that later.

7.) We need to add an entry regarding these changes to the changelog.md file. I can add a commit for that to the PR.

8.) Do you think this feature should be published as v2.0.0 or, as it doesn't constitute a breaking change, as v1.9.0?

[d-Rickyy-b]

I merged this PR into its own branch so that I can publish a beta release for the feature in order to collect feedback on the changes.