sqlite: Add all core extensions

[Ecco]

Sqlite's CLI comes packaged with a set of extensions that are definitely useful in node as well. Those extensions are base64, base85, decimal, ieee754, regexp, series, sha1, shatree, stmtrand and uint.

[nodejs-github-bot]

Review requested:

• @nodejs/gyp
• @nodejs/security-wg
• @nodejs/sqlite

[geeksilva97]

I'm not sure if we need this. They are extensions. When needed, one can load using loadExtension.

[Ecco]

Hi @geeksilva! You're right that one could load an extension. But for end-users and developers, those are a pain in the butt to build and distribute.

Here's why I think it's a great idea to add those core extensions to node:

• Those features are built into Sqlite's CLI. Which means many users will be surprised not to be able to use them when it just works in their terminal.
• Those functions are useful: we're talking about base64 encoding/decoding, hash computation, regular expression matching, fixed-point math, etc… We can reasonably expect developers using sqlite on node to need those features at some point.
• The binary increase is minimal: this adds 5k lines of C code, compared to sqlite's 270k. That's not even a 2% increase in size of sqlite-related code.

The reason those functions are not in Sqlite's amalgamation by default is because the amalgamation is meant to be statically built into specific end-user applications, with known-ahead requirements. For example, if I create a phone book app, I can tell at compile-time that I will, for instance, not need base64 decoding.

In node's case it's different: we're providing a toolset to end-users, and I think it's a mistake to remove some pretty important tools from the set. I think we should follow the decision Sqlite's developers made when they built their CLI, and include those core extensions in node as well.

Note: This PR is far from adding every Sqlite extensions to the build. This is a cherry-picked set that matches what Sqlite included in their CLI. For the record, that's 9 extensions out of close of a 100 extensions available in Sqlite's source tree.

[tpoisseau]

SGTM, missing extension embed by sqlite cli can be annoying.

[TheOneTheOnlyJJ]

If this gets approved, I believe this should be documented as well. The compile-time-enabled extension should, too. A new docs section or page covering all the extensions shipped by default would definitely be helpful.

Edit: Would it be possible to add tests to ensure there is always parity between these extensions and those included by the SQLite CLI?
Regression tests can be hand-written, but how could it be easy to detect if the CLI ships a new extension besides the ones already mentioned here?

[Ecco]

Would it be possible to add tests to ensure there is always parity between these extensions and those included by the SQLite CLI?

You're making a very good point. I don't have any good idea to easily automate this. The list of extensions that gets built into Sqlite's CLI comes from shell.c.in in Sqlite's source code. And it's pretty much some C source code that happens to call the init functions of some extensions, so that'd be quite hard to parse. Is there any architecture already in place in node's source code regarding updating third-party dependencies?

As a matter of fact, a new core extension has just been added to Sqlite's CLI!

On a different note, for the record, this PR doesn't currently come with any test. I have a draft ready though, but it needs some cleanup. If the idea of this PR gets approved, I'll finish my draft and add it to the PR.

Regarding the doc, same thing: I'd be happy to contribute some docs to mention this. I don't think the documentation needs to be comprehensive though: I believe simply mentionning that node comes with the same set of extensions as Sqlite's CLI would do the job. Indeed, if you're a developer using Sqlite, you'll eventually read Sqlite's own doc, and they are already very explicit about whether an extension is in the CLI or not.