Which database should be used as the underlying data store?

[valerie-autumn-skye]

Discussed during morning session on June 11.

Currently using MongoDB, but other, more robust databases are available:

• PostgreSQL has geodatabase extensions, JSONB column support (with indexing and query capability into the data structure), and extensions available for JSON validation against a schema. However, it is an open question of whether these three features will work together.
• LinkML supports JSON-Schema draft-07. There are multiple extensions available that enable JSON validation. There are likely more, these are just the first I found and are likely some of the oldest. There may be newer extensions that are more suitable.
  • is_json_valid supports draft-4 and draft-7
  • postgres-json-schema supports draft-4 except for remote (http) references, implemented as PL/pgSQL function

During the June 11 hackathon session, it was agreed that for the short term, MongoDB is sufficient for our needs and will facilitate rapid development, but this issue should be re-examined in a later iteration of development. @aclum and @jeff-cohere added support in favor of PostgreSQL over MongoDB in June 12 hackathon session. Our strict schema dependence is at odds with the flexible schema model of MongoDB.

[eecavanna]

Also consider the Lakehouse.

Another topic that arose during discussion is "LinkML Store".

[shreddd]

I think we should shift this discussion to how we would port this into a Lakehouse ready format. (which would also be SQL based)