Improve migration framework and add migration 101

[hjanott]

This draft PR aims at improving our migration framework so that it's usable for small migrations which entail single or few collections. The migration is for testing and should be added in a separate PR.
Important for #3476
Related to #3323

I'm introducing the terms adjacent views and tables. These are the non intermediate tables and their views which have a relation to a table that needs to be migrated. Adjacent tables need a migration view also. In the code they are referred to as additional_views.

This already does:

• more helper methods
• Improve the regular expressions
• add additional_views information to replace tables during setup
• iterate all views to create mig copies of only relevant
• create only relevant triggers
• create only relevant fkeys and indexes
• try to drop exactly relevant views (migration views and adjacent views)
• move ORIGIN_COLLECTIONS to class
• add rudimentary mig 0101 and test
• cleanup in migratable_models calculation
• only output first exception on migration route and print error class
• recursive execute command to use different connections between migrate and finalize
• mock listdir for migration tests to only apply current migration

What it is still lacking:

• including all adjacent collections into the replace tables
  • Migration handlier does probably not collect all additional views during migration setup.
• check that view and table names in trigger definitions are properly replaced for adjacent tables
• I didn't check the behaviour for foreign key constraints during migration
• In general tables and intermediate tables cannot be deleted before their views. Things like these shouldn't happen vaguely with 'IF EXISTS' or CASCADE to keep it explicit and specific to understand exactly what is being done.
• split logical segments of long functions into new functions
• Before migration does not setup the fkey constraints according to which migration tables were created and which should still point to origin tables. See TODO at line 128
  • For this write a check to assure that after migrate there exists a foreign key from an adjacent origin tables to a migration table.
  • In that area also drop the old constraint. See commented out code.
• Write check to assure that also for adjacent tables a migration view is created. Best case also compare to the correct view definition to assure that its pointing to existing migration tables and in all other cases to origin tables.
• does not nicely make use of additional_views information (for instance creation of real_replace_tables). There is a possibility to store that information analogously in its own column.
• BaseMigrationTestCase doesn't hold the cursor so it's always required to pass that to assert_content_not_none.
• Needs to properly check the recreation of views and triggers for collections like committee adjacent to origin tables like meeting
• check TODO at line 619 of migration handler
• TODO in test 101 line 61 may help to generalize some parts of the test setup.
• Does not assert the general functionality of the framework if the test for 101 is gone. Maybe we can find a solution by mocking or integrating it into the mig 100 test.
• There is the general problem of information getting lost in models yml between migrations. As get_replace_tables in migration_helper relies on that information, we may need specific alterations defined by each migration. One approach could be to set the information in the data definition function or create a function like this analogously to write the information lost into the version table either directly into the replace_tables column or its own. Another one would be to write such information into a data structure of the migration class. There was also the discussion about creating diffs from yml files.
• Needs a general code cleanup like variable names hopefully solved todos and code structure
• implement all changes including helper functions necessary for a migration like this: https://github.com/OpenSlides/openslides-meta/pull/372/changes and https://github.com/OpenSlides/openslides-vote-service/blob/feature/vote/Migration.md

[luisa-beerboom]

• BaseMigrationTestCase doesn't hold the cursor so it's always required to pass that with assert_content_not_none

?

[luisa-beerboom]

• tables and intermediate tables cannot be deleted before their views. Things like these shouldn't happen vaguely with 'IF EXISTS' or CASCADE to keep it explicit and specific to understand exactly what is being done.

What part of this is a description of current state, what part is a "should happen" statement?

[luisa-beerboom]

• check TODO at line 619 of migration helper

migration helper has no ln 619. Did you mean "migration handler"?

[luisa-beerboom]

• There is the general problem of information getting lost in models yml between migrations. As get_replace_tables in migration_helper relies on that information, we may need specific alterations defined by each migration.

Can you give an example of how this could happen?