otrebor opened a new issue, #30596: URL: https://github.com/apache/superset/issues/30596
### Bug description We are trying to upgrade our superset instance to 410rc2-py310 when we start superset, we get this error while the migration scripts are trying to update the catalog: ``` Loaded your LOCAL configuration at [/app/superset_config.py] WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments /usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( INFO:superset.utils.screenshots:No PIL installation found INFO:superset.utils.pdf:No PIL installation found INFO [alembic.env] Starting the migration scripts. INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade ec54aca4c8a2 -> 317970b4400c, Added always_filter_main_dttm to datasource INFO [alembic.runtime.migration] Running upgrade 317970b4400c -> 4b85906e5b91, add on delete cascade for dashboard_roles INFO [alembic.runtime.migration] Running upgrade 4b85906e5b91 -> b7851ee5522f, replay 317970b4400c INFO [alembic.runtime.migration] Running upgrade b7851ee5522f -> 06dd9ff00fe8, add_percent_calculation_type_funnel_chart INFO [alembic.runtime.migration] Running upgrade 06dd9ff00fe8 -> 65a167d4c62e, add indexes to report models INFO [alembic.runtime.migration] Running upgrade 65a167d4c62e -> 59a1450b3c10, drop_filter_sets_table INFO [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> a32e0c4d8646, migrate-sunburst-chart INFO [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> 96164e3017c6 INFO [alembic.runtime.migration] Running upgrade 96164e3017c6, a32e0c4d8646 -> 15a2c68a2e6b, merging two heads INFO [alembic.runtime.migration] Running upgrade a32e0c4d8646 -> 214f580d09c9, migrate_filter_boxes_to_native_filters INFO [alembic.runtime.migration] Running upgrade 214f580d09c9 -> e863403c0c50, drop_url_table INFO [alembic.runtime.migration] Running upgrade e863403c0c50, 15a2c68a2e6b -> 1cf8e4344e2b, merging INFO [alembic.runtime.migration] Running upgrade 1cf8e4344e2b -> 87d38ad83218, Migrate can_view_and_drill permission INFO [alembic.runtime.migration] Running upgrade 87d38ad83218 -> 17fcea065655, change_text_to_mediumtext INFO [alembic.runtime.migration] Running upgrade 17fcea065655 -> be1b217cd8cd, big_number_kpi_single_metric INFO [alembic.runtime.migration] Running upgrade be1b217cd8cd -> 678eefb4ab44, Add access token table INFO [alembic.runtime.migration] Running upgrade 678eefb4ab44 -> c22cb5c2e546, empty message INFO [alembic.runtime.migration] Running upgrade c22cb5c2e546 -> 5ad7321c2169, mig new csv upload perm INFO [alembic.runtime.migration] Running upgrade 5ad7321c2169 -> d60591c5515f, mig new excel upload perm INFO [alembic.runtime.migration] Running upgrade d60591c5515f -> 5f57af97bc3f, Add catalog column INFO [alembic.runtime.migration] Running upgrade 5f57af97bc3f -> 3dfd0e78650e, add_query_sql_editor_id_index INFO [alembic.runtime.migration] Running upgrade 5f57af97bc3f -> 4a33124c18ad, mig new columnar upload perm INFO [alembic.runtime.migration] Running upgrade 4a33124c18ad -> 58d051681a3b, Add catalog_perm to tables 26/26 7/7 Traceback (most recent call last): File "/usr/local/bin/superset", line 8, in <module> sys.exit(superset()) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 358, in decorator return __ctx.invoke(f, *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/flask_migrate/cli.py", line 149, in upgrade _upgrade(directory, revision, sql, tag, x_arg) File "/usr/local/lib/python3.10/site-packages/flask_migrate/__init__.py", line 98, in wrapped f(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/flask_migrate/__init__.py", line 185, in upgrade command.upgrade(config, revision, sql=sql, tag=tag) File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 403, in upgrade script.run_env() File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 583, in run_env util.load_python_file(self.dir, "env.py") File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file module = load_module_py(module_id, path) File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py spec.loader.exec_module(module) # type: ignore File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/app/superset/extensions/../migrations/env.py", line 142, in <module> run_migrations_online() File "/app/superset/extensions/../migrations/env.py", line 133, in run_migrations_online context.run_migrations() File "<string>", line 8, in run_migrations File "/usr/local/lib/python3.10/site-packages/alembic/runtime/environment.py", line 948, in run_migrations self.get_context().run_migrations(**kw) File "/usr/local/lib/python3.10/site-packages/alembic/runtime/migration.py", line 627, in run_migrations step.migration_fn(**kw) File "/app/superset/migrations/versions/2024-05-01_10-52_58d051681a3b_add_catalog_perm_to_tables.py", line 47, in upgrade upgrade_catalog_perms(engines={"postgresql"}) File "/app/superset/migrations/shared/catalogs.py", line 373, in upgrade_catalog_perms for database in session.query(Database).all(): File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2773, in all return self._iter().all() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1476, in all return self._allrows() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 401, in _allrows rows = self._fetchall_impl() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1389, in _fetchall_impl return self._real_result._fetchall_impl() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1813, in _fetchall_impl return list(self.iterator) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 147, in chunks fetch = cursor._raw_all_rows() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 393, in _raw_all_rows return [make_row(row) for row in rows] File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 393, in <listcomp> return [make_row(row) for row in rows] File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/type_api.py", line 1716, in process return process_value(impl_processor(value), dialect) File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 479, in process_result_value value = super().process_result_value(value=value, dialect=dialect) File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 423, in process_result_value self._update_key() File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 392, in _update_key self.engine._update_key(key) File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 52, in _update_key digest.update(key) TypeError: argument 'data': a bytes-like object is required, not 'NoneType' WARNING:root:Applying Monkey Patching for gevent ``` I can't really understand if it's a configuration problem or a bug. The connection string seems fine and previous tables are updated. we see other issues like https://github.com/apache/superset/issues/29492 but i don't think that's the problem for us. ### Screenshots/recordings _No response_ ### Superset version master / latest-dev ### Python version 3.10 ### Node version 16 ### Browser Chrome ### Additional context ``` Loaded your LOCAL configuration at [/app/superset_config.py] WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments /usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( INFO:superset.utils.screenshots:No PIL installation found INFO:superset.utils.pdf:No PIL installation found INFO [alembic.env] Starting the migration scripts. INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade ec54aca4c8a2 -> 317970b4400c, Added always_filter_main_dttm to datasource INFO [alembic.runtime.migration] Running upgrade 317970b4400c -> 4b85906e5b91, add on delete cascade for dashboard_roles INFO [alembic.runtime.migration] Running upgrade 4b85906e5b91 -> b7851ee5522f, replay 317970b4400c INFO [alembic.runtime.migration] Running upgrade b7851ee5522f -> 06dd9ff00fe8, add_percent_calculation_type_funnel_chart INFO [alembic.runtime.migration] Running upgrade 06dd9ff00fe8 -> 65a167d4c62e, add indexes to report models INFO [alembic.runtime.migration] Running upgrade 65a167d4c62e -> 59a1450b3c10, drop_filter_sets_table INFO [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> a32e0c4d8646, migrate-sunburst-chart INFO [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> 96164e3017c6 INFO [alembic.runtime.migration] Running upgrade 96164e3017c6, a32e0c4d8646 -> 15a2c68a2e6b, merging two heads INFO [alembic.runtime.migration] Running upgrade a32e0c4d8646 -> 214f580d09c9, migrate_filter_boxes_to_native_filters INFO [alembic.runtime.migration] Running upgrade 214f580d09c9 -> e863403c0c50, drop_url_table INFO [alembic.runtime.migration] Running upgrade e863403c0c50, 15a2c68a2e6b -> 1cf8e4344e2b, merging INFO [alembic.runtime.migration] Running upgrade 1cf8e4344e2b -> 87d38ad83218, Migrate can_view_and_drill permission INFO [alembic.runtime.migration] Running upgrade 87d38ad83218 -> 17fcea065655, change_text_to_mediumtext INFO [alembic.runtime.migration] Running upgrade 17fcea065655 -> be1b217cd8cd, big_number_kpi_single_metric INFO [alembic.runtime.migration] Running upgrade be1b217cd8cd -> 678eefb4ab44, Add access token table INFO [alembic.runtime.migration] Running upgrade 678eefb4ab44 -> c22cb5c2e546, empty message INFO [alembic.runtime.migration] Running upgrade c22cb5c2e546 -> 5ad7321c2169, mig new csv upload perm INFO [alembic.runtime.migration] Running upgrade 5ad7321c2169 -> d60591c5515f, mig new excel upload perm INFO [alembic.runtime.migration] Running upgrade d60591c5515f -> 5f57af97bc3f, Add catalog column INFO [alembic.runtime.migration] Running upgrade 5f57af97bc3f -> 3dfd0e78650e, add_query_sql_editor_id_index INFO [alembic.runtime.migration] Running upgrade 5f57af97bc3f -> 4a33124c18ad, mig new columnar upload perm INFO [alembic.runtime.migration] Running upgrade 4a33124c18ad -> 58d051681a3b, Add catalog_perm to tables 26/26 7/7 Traceback (most recent call last): File "/usr/local/bin/superset", line 8, in <module> sys.exit(superset()) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 358, in decorator return __ctx.invoke(f, *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/flask_migrate/cli.py", line 149, in upgrade _upgrade(directory, revision, sql, tag, x_arg) File "/usr/local/lib/python3.10/site-packages/flask_migrate/__init__.py", line 98, in wrapped f(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/flask_migrate/__init__.py", line 185, in upgrade command.upgrade(config, revision, sql=sql, tag=tag) File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 403, in upgrade script.run_env() File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 583, in run_env util.load_python_file(self.dir, "env.py") File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file module = load_module_py(module_id, path) File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py spec.loader.exec_module(module) # type: ignore File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/app/superset/extensions/../migrations/env.py", line 142, in <module> run_migrations_online() File "/app/superset/extensions/../migrations/env.py", line 133, in run_migrations_online context.run_migrations() File "<string>", line 8, in run_migrations File "/usr/local/lib/python3.10/site-packages/alembic/runtime/environment.py", line 948, in run_migrations self.get_context().run_migrations(**kw) File "/usr/local/lib/python3.10/site-packages/alembic/runtime/migration.py", line 627, in run_migrations step.migration_fn(**kw) File "/app/superset/migrations/versions/2024-05-01_10-52_58d051681a3b_add_catalog_perm_to_tables.py", line 47, in upgrade upgrade_catalog_perms(engines={"postgresql"}) File "/app/superset/migrations/shared/catalogs.py", line 373, in upgrade_catalog_perms for database in session.query(Database).all(): File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2773, in all return self._iter().all() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1476, in all return self._allrows() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 401, in _allrows rows = self._fetchall_impl() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1389, in _fetchall_impl return self._real_result._fetchall_impl() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1813, in _fetchall_impl return list(self.iterator) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 147, in chunks fetch = cursor._raw_all_rows() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 393, in _raw_all_rows return [make_row(row) for row in rows] File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 393, in <listcomp> return [make_row(row) for row in rows] File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/type_api.py", line 1716, in process return process_value(impl_processor(value), dialect) File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 479, in process_result_value value = super().process_result_value(value=value, dialect=dialect) File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 423, in process_result_value self._update_key() File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 392, in _update_key self.engine._update_key(key) File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 52, in _update_key digest.update(key) TypeError: argument 'data': a bytes-like object is required, not 'NoneType' WARNING:root:Applying Monkey Patching for gevent ``` ### Checklist - [X] I have searched Superset docs and Slack and didn't find a solution to my problem. - [X] I have searched the GitHub issue tracker and didn't find a similar bug report. - [X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
