Feature/step1 #2

e-buerger · 2022-05-17T12:58:46Z

setup database with sqlalchemy and sqlite
implement enums for workflow language, exec profile type and job status
implement method for creating jobs in job_factory
implement Execprofile class which only contains attributes and no methods
implement Job class which has abstract methods (prepare, exec, eval, finalize, cancel)
implement PythonJob class which inherits from Job and has the additional attribute inputs (this will be the class from which the class that is implemented by the user user inherits when he wants to use the python exec profile)
implement PyExecSession class which runs the method implemented by the user
implement tests for the functions in job_dao.py

KerstenBreuer

Vielen Dank. Ein paar generelle Hinweise. Vielleicht können wir gleich drüber gehen.

KerstenBreuer · 2022-05-19T12:04:09Z

exec_manager/__init__.py

 # limitations under the License.

-"""Short description of package"""  # Please adapt to package
+"""backend"""  # Please adapt to package


Suggested change

"""backend""" # Please adapt to package

"""

A package managing execution of jobs in a way that is agnostic to

- the workflow execution environment

- the language used to describe the workflow

"""

KerstenBreuer · 2022-05-19T12:04:36Z

exec_manager/dao/__init__.py

+# Copyright 2021 - 2022 Universität Tübingen, DKFZ and EMBL
+# for the German Human Genome-Phenome Archive (GHGA)


Copyright muss noch angepasst werden

KerstenBreuer · 2022-05-19T12:15:44Z

exec_manager/wf_lang_type.py

+    """enumerate workflow language types"""
+
+    CWL = "cwl"
+    """cwl language"""


Doc strings werden normalerweise nur an den Anfang von einer Klasse, Funktion, oder einem Modul gestellt.

KerstenBreuer · 2022-05-19T12:21:06Z

exec_manager/job.py

+from uuid import UUID
+
+from exec_manager.exec_profile import ExecProfile
+from exec_manager.job_status_type import JobStatusType
+
+
+class Job:


Job sollte am Besten eine abstrakte Klasse sein:

Suggested change

from uuid import UUID

from exec_manager.exec_profile import ExecProfile

from exec_manager.job_status_type import JobStatusType

class Job:

from abc import ABC, abstractmethod

from uuid import UUID

from exec_manager.exec_profile import ExecProfile

from exec_manager.job_status_type import JobStatusType

class Job(ABC):

KerstenBreuer · 2022-05-19T12:23:08Z

exec_manager/job.py

+    def prepare(self) -> None:
+        """
+        Prepares the job.
+
+        Parameters
+        ----------
+
+        Returns
+        -------
+        NONE
+        """


Das hier sollten abstracte methoden sein:

Suggested change

def prepare(self) -> None:

"""

Prepares the job.

Parameters

----------

Returns

-------

NONE

"""

@abstractmethod

def prepare(self) -> None:

"""

Prepares the job.

Parameters

----------

Returns

-------

NONE

"""

...

KerstenBreuer · 2022-05-19T12:41:48Z

exec_manager/dao/db_models.py

+
+from sqlalchemy import JSON, Column, String
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm.decl_api import DeclarativeMeta


Du kannst den UUID Typ von SQLalchemy benutzen:

Suggested change

from sqlalchemy.orm.decl_api import DeclarativeMeta

import uuid

from sqlalchemy.orm.decl_api import DeclarativeMeta

from sqlalchemy.dialects.postgresql import UUID

KerstenBreuer · 2022-05-19T12:43:54Z

exec_manager/dao/db_models.py

-    id = Column(Integer, primary_key=True)
-    name = Column(String, nullable=False)
-    active = Column(Boolean, nullable=False)
+    job_id = Column(String, primary_key=True)


Und hier dann:

Suggested change

job_id = Column(String, primary_key=True)

job_id = Column(UUID(as_uuid=True), default=uuid.uuid4, primary_key=True)

So wird die UUID automatisch vergeben.

KerstenBreuer · 2022-05-19T12:45:06Z

exec_manager/dao/job_dao.py

+    job_id = generate_job_id()
+    job_id_str = str(job_id)


Siehe die Vorschläge im db_model. Dort habe ich einen default generator für die id eingefügt. Dann musst du dich hier nicht mehr drum kümmern.

KerstenBreuer · 2022-05-19T12:47:18Z

exec_manager/exec_profile.py

+from exec_manager.wf_lang_type import WfLangType
+
+
+class ExecProfile:


Für das ExecProfile bietet es sich an entweder eine Dataclass oder ein pydantic model zu verwenden, da diese Klasse lediglich daten speichern soll.

Siehe z.B. hier: https://pydantic-docs.helpmanual.io/#example

KerstenBreuer · 2022-05-19T12:50:34Z

exec_manager/dao/job_dao.py

+    exec_profile_json = json.dumps(
+        {
+            "exec_profile_type": exec_profile.exec_profile_type.value,
+            "wf_lang": exec_profile.wf_lang.value,
+        }
+    )


Wenn das ExecProfile auf einem pydantic model beruht kann man einfach schreiben:

Suggested change

exec_profile_json = json.dumps(

{

"exec_profile_type": exec_profile.exec_profile_type.value,

"wf_lang": exec_profile.wf_lang.value,

}

)

exec_profile_dict = exec_profile.to_dict()

Das hier returned ein dict. Also kein JSON. Aber die Conversion zu JSON kann man auch sqlalchemy machen lassen.

create classfiles adapting code for passing precommit adapt code for passing precommit adapt code for passing precommit implement a few functions change type of db columns satisfy pre-commit implement exec method of PythonJob suppres bandit warnigns and add comments remove __init__.py from dao directory remove context manager fix session error in job_dao.py change db commands change db commands change db commands fix import errors by using absolute path and an __init__ file remove content from exec method add license headers add license headers implement PyExecSession implement unit tests for job_dao satisfy pipeline satisfy pipeline add license header update README update doc strings; change job_dao functions for better unit tests change engine type

e-buerger added 27 commits May 4, 2022 09:07

create classfiles

fd7fce9

adapting code for passing precommit

dad4372

adapt code for passing precommit

f59a516

adapt code for passing precommit

a19747c

implement a few functions

7588dde

change type of db columns

56af3aa

satisfy pre-commit

566dbe1

implement exec method of PythonJob

37ae4a4

suppres bandit warnigns and add comments

5529827

remove __init__.py from dao directory

5f17ab3

remove context manager

cd6f426

fix session error in job_dao.py

792ce11

change db commands

4eeafcf

change db commands

af7d7a9

change db commands

a1d9e61

fix import errors by using absolute path and an __init__ file

1993d54

remove content from exec method

ae53782

add license headers

211f348

add license headers

46f313e

implement PyExecSession

c5253c3

implement unit tests for job_dao

97c902a

satisfy pipeline

6491949

satisfy pipeline

c344999

add license header

d93859c

update README

c862e18

update doc strings; change job_dao functions for better unit tests

37b711c

change engine type

bf7255d

e-buerger requested a review from KerstenBreuer May 17, 2022 12:58

KerstenBreuer requested changes May 19, 2022

View reviewed changes

KerstenBreuer force-pushed the feature/step1 branch from bf7255d to f671e43 Compare May 19, 2022 13:23

KerstenBreuer and others added 6 commits May 19, 2022 14:15

refactor

c286f37

resolve merge conflicts

5ab918d

restructure modules

9370930

remove unneccassery class

e5e559b

satisfy pipeline

86b72dd

update doc strings

9856610

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/step1 #2

Feature/step1 #2

Uh oh!

e-buerger commented May 17, 2022

Uh oh!

KerstenBreuer left a comment

Uh oh!

KerstenBreuer May 19, 2022

Uh oh!

KerstenBreuer May 19, 2022

Uh oh!

KerstenBreuer May 19, 2022

Uh oh!

KerstenBreuer May 19, 2022

Uh oh!

KerstenBreuer May 19, 2022

Uh oh!

KerstenBreuer May 19, 2022

Uh oh!

KerstenBreuer May 19, 2022

Uh oh!

KerstenBreuer May 19, 2022

Uh oh!

KerstenBreuer May 19, 2022

Uh oh!

KerstenBreuer May 19, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-"""backend"""  # Please adapt to package
+"""
+A package managing execution of jobs in a way that is agnostic to
+- the workflow execution environment
+- the language used to describe the workflow
+"""

		# Copyright 2021 - 2022 Universität Tübingen, DKFZ and EMBL
		# for the German Human Genome-Phenome Archive (GHGA)

	job_id = Column(String, primary_key=True)
	job_id = Column(UUID(as_uuid=True), default=uuid.uuid4, primary_key=True)

		from exec_manager.wf_lang_type import WfLangType


		class ExecProfile:

Feature/step1 #2

Are you sure you want to change the base?

Feature/step1 #2

Uh oh!

Conversation

e-buerger commented May 17, 2022

Uh oh!

KerstenBreuer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants