Upgrade from unsupported django version to version 1.11.11

[pchengi]

Hi,

COG now uses django 1.10.4 which is officially unsupported, and this also has two vulnerabilities that can lead to denial of service (CVE-2018-7536, CVE-2018-7537).
I have tried to manually bump up the django version to 1.11.11, but doing so requires psycopg2 to be upgraded from version 2.5.2 to 2.5.4. When the pyscopg2 is also upgraded to version 2.5.4, CoGInstall fails with the following error-trace. Please upgrade to django 1.11.11 at the earliest.

>>> 3) Executing CoGInstall...^M
INFO:root:Site manager: using CoG settings from file(s): ['/usr/local/cog/cog_config/cog_settings.cfg']^M
Initialized CoG settings from file: /usr/local/cog/cog_config/cog_settings.cfg^M
INFO:root:Postgres database: cogdb created^M
ESC[36;1mOperations to perform:ESC[0m^M
ESC[1m  Apply all migrations: ESC[0madmin, auth, captcha, cog, contenttypes, django_comments, django_openid_auth, sessions, sites^M
ESC[36;1mRunning migrations:ESC[0m^M
  Applying contenttypes.0001_initial...ESC[32;1m OKESC[0m^M
  Applying auth.0001_initial...ESC[32;1m OKESC[0m^M
  Applying admin.0001_initial...ESC[32;1m OKESC[0m^M
  Applying admin.0002_logentry_remove_auto_add...ESC[32;1m OKESC[0m^M
  Applying contenttypes.0002_remove_content_type_name...ESC[32;1m OKESC[0m^M
  Applying auth.0002_alter_permission_name_max_length...ESC[32;1m OKESC[0m^M
  Applying auth.0003_alter_user_email_max_length...ESC[32;1m OKESC[0m^M
  Applying auth.0004_alter_user_username_opts...ESC[32;1m OKESC[0m^M
  Applying auth.0005_alter_user_last_login_null...ESC[32;1m OKESC[0m^M
  Applying auth.0006_require_contenttypes_0002...ESC[32;1m OKESC[0m^M
  Applying auth.0007_alter_validators_add_error_messages...ESC[32;1m OKESC[0m^M
  Applying auth.0008_alter_user_username_max_length...Traceback (most recent call last):^M
  File "setup.py", line 29, in <module>^M
    "distutils.commands": [ "setup_cog = cog.installation.setup:CogSetupCommand"]^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/setuptools/__init__.py", line 129, in setup^M
    return distutils.core.setup(**attrs)^M
  File "/opt/esgf/python/lib/python2.7/distutils/core.py", line 151, in setup^M
    dist.run_commands()^M
  File "/opt/esgf/python/lib/python2.7/distutils/dist.py", line 953, in run_commands^M
    self.run_command(cmd)^M
  File "/opt/esgf/python/lib/python2.7/distutils/dist.py", line 972, in run_command^M
    cmd_obj.run()^M
  File "/usr/local/cog/cog_install/cog/installation/setup.py", line 45, in run^M
    cogInstall.install()^M
  File "/usr/local/cog/cog_install/cog/installation/install.py", line 46, in install^M
    self._upgradeCog()^M
  File "/usr/local/cog/cog_install/cog/installation/install.py", line 68, in _upgradeCog^M
    management.call_command("migrate", interactive=False)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command^M
    return command.execute(*args, **defaults)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute^M
    output = self.handle(*args, **options)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle^M
    fake_initial=fake_initial,^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate^M
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards^M
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration^M
    state = migration.apply(state, schema_editor)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply^M
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/migrations/operations/fields.py", line 221, in database_forwards^M
    schema_editor.alter_field(from_model, from_field, to_field)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 515, in alter_field^M
    old_db_params, new_db_params, strict)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/postgresql/schema.py", line 112, in _alter_field^M
    new_db_params, strict,^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 559, in _alter_field^M
    new_rel.related_model, [new_rel.field.column], foreign_key=True^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 995, in _constraint_names^M
    constraints = self.connection.introspection.get_constraints(cursor, model._meta.db_table)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/postgresql/introspection.py", line 233, in get_constraints^M
    """, [table_name])^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute^M
    return self.cursor.execute(sql, params)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__^M
    six.reraise(dj_exc_type, dj_exc_value, traceback)^M
  File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute^M
    return self.cursor.execute(sql, params)^M
django.db.utils.ProgrammingError: syntax error at or near "ORDER"^M
LINE 3:                 indexname, array_agg(attname ORDER BY rnum),...^M

[LucaCinquini]

Hi Prashanth, thanks for trying - in general, upgrading to a major Django release is not as simple as switching a version number, there are software changes that need to be performed, and extensive testing. I will try to take care of this as soon as I can. Thanks, Luca

…

On Mar 9, 2018, at 5:45 AM, pchengi ***@***.***> wrote: Hi, COG now uses django 1.10.4 which is officially unsupported, and this also has two vulnerabilities that can lead to denial of service (CVE-2018-7536, CVE-2018-7537). I have tried to manually bump up the django version to 1.11.11, but doing so requires psycopg2 to be upgraded from version 2.5.2 to 2.5.4. When the pyscopg2 is also upgraded to version 2.5.4, CoGInstall fails with the following error-trace. Please upgrade to django 1.11.11 at the earliest. >>> 3) Executing CoGInstall...^M INFO:root:Site manager: using CoG settings from file(s): ['/usr/local/cog/cog_config/cog_settings.cfg']^M Initialized CoG settings from file: /usr/local/cog/cog_config/cog_settings.cfg^M INFO:root:Postgres database: cogdb created^M ESC[36;1mOperations to perform:ESC[0m^M ESC[1m Apply all migrations: ESC[0madmin, auth, captcha, cog, contenttypes, django_comments, django_openid_auth, sessions, sites^M ESC[36;1mRunning migrations:ESC[0m^M Applying contenttypes.0001_initial...ESC[32;1m OKESC[0m^M Applying auth.0001_initial...ESC[32;1m OKESC[0m^M Applying admin.0001_initial...ESC[32;1m OKESC[0m^M Applying admin.0002_logentry_remove_auto_add...ESC[32;1m OKESC[0m^M Applying contenttypes.0002_remove_content_type_name...ESC[32;1m OKESC[0m^M Applying auth.0002_alter_permission_name_max_length...ESC[32;1m OKESC[0m^M Applying auth.0003_alter_user_email_max_length...ESC[32;1m OKESC[0m^M Applying auth.0004_alter_user_username_opts...ESC[32;1m OKESC[0m^M Applying auth.0005_alter_user_last_login_null...ESC[32;1m OKESC[0m^M Applying auth.0006_require_contenttypes_0002...ESC[32;1m OKESC[0m^M Applying auth.0007_alter_validators_add_error_messages...ESC[32;1m OKESC[0m^M Applying auth.0008_alter_user_username_max_length...Traceback (most recent call last):^M File "setup.py", line 29, in <module>^M "distutils.commands": [ "setup_cog = cog.installation.setup:CogSetupCommand"]^M File "/usr/local/cog/venv/lib/python2.7/site-packages/setuptools/__init__.py", line 129, in setup^M return distutils.core.setup(**attrs)^M File "/opt/esgf/python/lib/python2.7/distutils/core.py", line 151, in setup^M dist.run_commands()^M File "/opt/esgf/python/lib/python2.7/distutils/dist.py", line 953, in run_commands^M self.run_command(cmd)^M File "/opt/esgf/python/lib/python2.7/distutils/dist.py", line 972, in run_command^M cmd_obj.run()^M File "/usr/local/cog/cog_install/cog/installation/setup.py", line 45, in run^M cogInstall.install()^M File "/usr/local/cog/cog_install/cog/installation/install.py", line 46, in install^M self._upgradeCog()^M File "/usr/local/cog/cog_install/cog/installation/install.py", line 68, in _upgradeCog^M management.call_command("migrate", interactive=False)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command^M return command.execute(*args, **defaults)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute^M output = self.handle(*args, **options)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle^M fake_initial=fake_initial,^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate^M state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards^M state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration^M state = migration.apply(state, schema_editor)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply^M operation.database_forwards(self.app_label, schema_editor, old_state, project_state)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/migrations/operations/fields.py", line 221, in database_forwards^M schema_editor.alter_field(from_model, from_field, to_field)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 515, in alter_field^M old_db_params, new_db_params, strict)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/postgresql/schema.py", line 112, in _alter_field^M new_db_params, strict,^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 559, in _alter_field^M new_rel.related_model, [new_rel.field.column], foreign_key=True^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 995, in _constraint_names^M constraints = self.connection.introspection.get_constraints(cursor, model._meta.db_table)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/postgresql/introspection.py", line 233, in get_constraints^M """, [table_name])^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute^M return self.cursor.execute(sql, params)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__^M six.reraise(dj_exc_type, dj_exc_value, traceback)^M File "/usr/local/cog/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute^M return self.cursor.execute(sql, params)^M django.db.utils.ProgrammingError: syntax error at or near "ORDER"^M LINE 3: indexname, array_agg(attname ORDER BY rnum),...^M — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1387>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAm4g0CKRleIz2Subs3EvKYRLkEKV3mVks5tcl1YgaJpZM4SkFMG>.

[pchengi]

Thanks Luca!

[LucaCinquini]

This seems to be an internal Django error when executing the database migration. Unfortunately, Django 1.11.X requires Postgres 9.3 or later:
https://docs.djangoproject.com/en/1.11/ref/databases/#postgresql-notes
while ESGF is still stuck on Postgres 8.4. So we first need to upgrade the ESGF Postgres, then we can think about upgrading the CoG Django to 1.11.X.

[mauzey1]

CoG will soon be upgraded to Django 2 and Python 3.