Refactor: Change the DeclarativeBase sqlalchemy style

[HeloiseJoffe]

closed: #612

Changes:

• Replaced declarative_base with DeclarativeBase style
• Cast 'table' to Table type to satisfy mypy

Questions:
There is other new styles in SQLAlchemy, for example the declarative use of Column is replaced by mapped_column.
Should this be applied as well?
In the Bookkeeping schema, this has been done, but the other changes haven’t been applied. Is there a reason for that?

[ryuwd]

All the steps under https://docs.sqlalchemy.org/en/20/changelog/whatsnew_20.html#migrating-an-existing-mapping should be followed before closing #612

substituting mapped_column is enough to start fixing the typing errors but we lose the usefulness of the type checks without the additional annotations (which I still need to do for the bookkeeping schema)

[ryuwd]

Thanks for working on this! Next steps:

1. Column to mapped_column (see diff comment example in job_logging/schema.py)
2. Add Mapped[...] annotations for all schema files
3. Remove cast(__table__, Table) everywhere

[ryuwd]

The cast shouldn't be necessary here once the schamas have been fully migrated

[ryuwd]

Same here and everywhere else

[ryuwd]

To migrate this class:

1. first change the factories in diracx-db/src/diracx/db/sql/utils.py

-from sqlalchemy import Column as RawColumn
+from sqlalchemy.orm import mapped_column

-Column: partial[RawColumn] = partial(RawColumn, nullable=False)
-NullColumn: partial[RawColumn] = partial(RawColumn, nullable=True)
-DateNowColumn = partial(Column, type_=DateTime(timezone=True), server_default=utcnow())
+Column: partial[mapped_column] = partial(mapped_column, nullable=False)
+NullColumn: partial[mapped_column] = partial(mapped_column, nullable=True)
+DateNowColumn = partial(mapped_column, type_=DateTime(timezone=True), server_default=utcnow())

N.B. for any use of NullColumn you can use Optional inside the type hint

id_token: Mapped[Optional[dict]] = NullColumn("IDToken", JSON())

2. add Mapped type annotations to this class

+from sqlalchemy.orm import Mapped
+
 class LoggingInfo(JobLoggingDBBase):
     __tablename__ = "LoggingInfo"
     
-    job_id = Column("JobID", Integer)
+    job_id: Mapped[int] = Column("JobID", Integer)
-    seq_num = Column("SeqNum", Integer)
+    seq_num: Mapped[int] = Column("SeqNum", Integer)
-    status = Column("Status", String(32), default="")
+    status: Mapped[str] = Column("Status", String(32), default="")
-    minor_status = Column("MinorStatus", String(128), default="")
+    minor_status: Mapped[str] = Column("MinorStatus", String(128), default="")
-    application_status = Column("ApplicationStatus", String(255), default="")
+    application_status: Mapped[str] = Column("ApplicationStatus", String(255), default="")
-    status_time = DateNowColumn("StatusTime")
+    status_time: Mapped[datetime] = DateNowColumn("StatusTime")
-    status_time_order = Column("StatusTimeOrder", MagicEpochDateTime, default=0)
+    status_time_order: Mapped[int] = Column("StatusTimeOrder", MagicEpochDateTime, default=0)
-    source = Column("StatusSource", String(32), default="Unknown")
+    source: Mapped[str] = Column("StatusSource", String(32), default="Unknown")
     
     __table_args__ = (PrimaryKeyConstraint("JobID", "SeqNum"),)

[HeloiseJoffe]

Yes, I started like this, but with step 5 "make use of pep-593 Annotated to package common directives into types", doesn't it change the approach?
For example, shouldn't we do something like:

# In utils.py
datetime_now = Annotated[datetime, mapped_column(DateTime(timezone=True), server_default=utcnow())]

Instead of keeping DateNowColumn as partial[mapped_column] ?
Then use it like this :

status_time: Mapped[datetime_now] = mapped_column("StatusTime")

What do you think about it ?

[ryuwd]

Yes! For the type hints on the schema attributes I'd encourage to make aliases for the more complicated annotated types likely to be used more than once

As for the mapped_column objects we can reuse the (refactored) partial factories in utils.py, if that is not ideal we can think of a cleaner approach

Therefore we end up with

datetime_now = Annotated[datetime, DateNowColumn("StatusTime")]
...
class JobLoggingDB(JobLoggingDBBase):
    ...
    status_time: Mapped[datetime_now]
    ...