zulip_fail

root@chat:/home/ubuntu# /home/zulip/deployments/current/scripts/upgrade-zulip-from-git main
2022-10-14 00:56:27,928 upgrade-zulip-from-git: Fetching the latest commits
2022-10-14 00:56:29,024 upgrade-zulip-from-git: Upgrading to faabc0f6ad1dc4e29dfa2cd96a1e259a9b7454ef, in /home/zulip/deployments/2022-10-14-00-56-27
2022-10-14 00:56:30,132 upgrade-zulip-stage-2: Upgrading system packages...
Hit:1 http://ppa.launchpad.net/groonga/ppa/ubuntu focal InRelease
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Hit:5 http://apt.postgresql.org/pub/repos/apt focal-pgdg InRelease
Hit:6 https://packages.redis.io/deb focal InRelease
Get:7 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [596 B]
Get:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B]
Hit:10 https://packages.groonga.org/ubuntu focal InRelease
Fetched 337 kB in 1s (247 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages were automatically installed and are no longer required:
  libhiredis0.14 libjemalloc2 libllvm9 liblua5.1-0 lua-bitop lua-cjson
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
+ apt-get -y install build-essential libffi-dev libfreetype6-dev zlib1g-dev libjpeg-dev libldap2-dev python3-dev python3-pip virtualenv libxml2-dev libxslt1-dev libpq-dev libssl-dev libmagic1 libyaml-dev libxmlsec1-dev pkg-config jq libsasl2-dev
Reading package lists...
Building dependency tree...
Reading state information...
libffi-dev is already the newest version (3.3-4).
libjpeg-dev is already the newest version (8c-2ubuntu8).
libmagic1 is already the newest version (1:5.38-4).
libxmlsec1-dev is already the newest version (1.2.28-2).
libyaml-dev is already the newest version (0.2.2-1).
pkg-config is already the newest version (0.29.1-0ubuntu4).
python3-dev is already the newest version (3.8.2-0ubuntu2).
build-essential is already the newest version (12.8ubuntu1.1).
libfreetype6-dev is already the newest version (2.10.1-2ubuntu0.2).
libldap2-dev is already the newest version (2.4.49+dfsg-2ubuntu1.9).
libsasl2-dev is already the newest version (2.1.27+dfsg-2ubuntu0.1).
libssl-dev is already the newest version (1.1.1f-1ubuntu2.16).
libxml2-dev is already the newest version (2.9.10+dfsg-5ubuntu0.20.04.4).
libxslt1-dev is already the newest version (1.1.34-4ubuntu0.20.04.1).
zlib1g-dev is already the newest version (1:1.2.11.dfsg-2ubuntu1.4).
jq is already the newest version (1.6-1ubuntu0.20.04.1).
python3-pip is already the newest version (20.0.2-5ubuntu1.6).
virtualenv is already the newest version (20.0.17-1ubuntu0.4).
libpq-dev is already the newest version (15~beta2-1.pgdg20.04+1).
The following packages were automatically installed and are no longer required:
  libhiredis0.14 libjemalloc2 libllvm9 liblua5.1-0 lua-bitop lua-cjson
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Using cached Python venv from /srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv
+ ln -nsf /srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv /home/zulip/deployments/2022-10-14-00-56-27/zulip-py3-venv
generate_secrets: No new secrets to generate.
2022-10-14 00:56:38,243 upgrade-zulip-stage-2: Building static assets...
Cached version not found! Installing node modules.
+ /srv/zulip-yarn/bin/yarn install --non-interactive --frozen-lockfile --prod
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > postcss-media-minmax@5.0.0" has incorrect peer dependency "postcss@8.4.5".
[4/4] Building fresh packages...
warning Ignored scripts due to flag.
Done in 16.35s.
Using cached node modules from /srv/zulip-npm-cache/65c17c4409432ee74b7c833de59cd3052995efab/node_modules
+ ./tools/setup/emoji/build_emoji
build_emoji: Using cached emojis from /srv/zulip-emoji-cache/e884c6de717189886ad470f3d540d5bee79fcaa9/emoji
+ ./scripts/setup/inline_email_css.py
+ ./tools/setup/generate_zulip_bots_static_files.py
+ ./tools/setup/build_pygments_data
+ ./tools/setup/build_timezone_values
+ ./tools/webpack --quiet
+ ./manage.py collectstatic --no-default-ignore -v0 --noinput --ignore=assets --ignore=emoji-styles --ignore=html --ignore=js --ignore=styles --ignore=templates
+ ./manage.py compilemessages -v0
2022-10-14 00:57:59,007 upgrade-zulip-stage-2: Caching Zulip Git version...
System check identified no issues (15 silenced).
2022-10-14 00:58:01,753 upgrade-zulip-stage-2: Checking for needed migrations
2022-10-14 00:58:03,188 upgrade-zulip-stage-2: Pre-checking for puppet changes...
2022-10-14 00:58:07,736 upgrade-zulip-stage-2: No puppet changes found, skipping!
2022-10-14 00:58:07,737 upgrade-zulip-stage-2: Stopping Zulip...

Zulip stopped successfully!
2022-10-14 00:58:08,011 upgrade-zulip-stage-2: Applying database migrations...
Operations to perform:
  Apply all migrations: analytics, auth, confirmation, contenttypes, otp_static, otp_totp, sessions, social_django, two_factor, zerver
Running migrations:
  Applying zerver.0420_alter_archivedmessage_realm_alter_message_realm...Traceback (most recent call last):
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/home/zulip/deployments/2022-10-14-00-56-27/zerver/lib/db.py", line 34, in execute
    wrapper_execute(self, super().execute, query, vars)
  File "/home/zulip/deployments/2022-10-14-00-56-27/zerver/lib/db.py", line 19, in wrapper_execute
    action(sql, params)
psycopg2.errors.NotNullViolation: column "realm_id" of relation "zerver_message" contains null values


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 151, in <module>
    execute_from_command_line(sys.argv)
  File "./manage.py", line 116, in execute_from_command_line
    utility.execute()
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 448, in execute
    output = self.handle(*args, **options)
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 96, in wrapped
    res = handle_func(*args, **kwargs)
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 349, in handle
    post_migrate_state = executor.migrate(
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 130, in apply
    operation.database_forwards(
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 235, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 788, in alter_field
    self._alter_field(
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/postgresql/schema.py", line 246, in _alter_field
    super()._alter_field(
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 1007, in _alter_field
    self.execute(
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 199, in execute
    cursor.execute(sql, params)
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 562, in execute
    return real_execute(self, sql, params)
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/srv/zulip-venv-cache/4cb7e7f76b9747af6e3d88a027e730e8d0dedf5d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/home/zulip/deployments/2022-10-14-00-56-27/zerver/lib/db.py", line 34, in execute
    wrapper_execute(self, super().execute, query, vars)
  File "/home/zulip/deployments/2022-10-14-00-56-27/zerver/lib/db.py", line 19, in wrapper_execute
    action(sql, params)
django.db.utils.IntegrityError: column "realm_id" of relation "zerver_message" contains null values

Sentry is attempting to send 2 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
2022-10-14 00:58:12.128 WARN [urllib3.connectionpool] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f49c4331ac0>: Failed to establish a new connection: [Errno 113] No route to host')': /api/2/store/
2022-10-14 00:58:15.200 WARN [urllib3.connectionpool] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f49c438f610>: Failed to establish a new connection: [Errno 113] No route to host')': /api/2/store/
2022-10-14 00:58:18.272 WARN [urllib3.connectionpool] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f49c4331550>: Failed to establish a new connection: [Errno 113] No route to host')': /api/2/store/
Traceback (most recent call last):
  File "/home/zulip/deployments/2022-10-14-00-56-27/scripts/lib/upgrade-zulip-stage-2", line 463, in <module>
    subprocess.check_call(
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./manage.py', 'migrate', '--noinput', '--skip-checks']' returned non-zero exit status 1.

Zulip upgrade failed (exit code 1)!

The upgrade process is designed to be idempotent, so you can retry after resolving whatever issue caused the failure (there should be a traceback above). A log of this installation is available in /var/log/zulip/upgrade.log
root@chat:/home/ubuntu#