Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Error on startup: AttributeError: ('max_upload_size', "not found in [...]") #10626

Closed
anoadragon453 opened this issue Aug 17, 2021 · 4 comments · Fixed by #10628
Closed

Error on startup: AttributeError: ('max_upload_size', "not found in [...]") #10626

anoadragon453 opened this issue Aug 17, 2021 · 4 comments · Fixed by #10628
Labels
A-Admin-API A-Media-Repository Uploading, downloading images and video, thumbnailing X-Release-Blocker Must be resolved before making a release

Comments

@anoadragon453
Copy link
Member

We received the following traceback on the main process when attempting to deploy v1.41.0rc1 (18572fd) to matrix.org:

2021-08-17 13:47:11,357 - synapse.federation.federation_server - 1204 - INFO - sentinel - Registering federation EDU handler for 'org.matrix.signing_key_update'
2021-08-17 13:47:11,357 - synapse.federation.federation_server - 1224 - INFO - sentinel - Registering federation query handler for 'client_keys'
2021-08-17 13:47:11,368 - synapse.app._base - 208 - CRITICAL - sentinel - Error during startup
Capture point (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/synapse/src/synapse/app/homeserver.py", line 462, in <module>
    main()
  File "/home/synapse/src/synapse/app/homeserver.py", line 458, in main
    run(hs)
  File "/home/synapse/src/synapse/app/homeserver.py", line 444, in run
    logger=logger,
  File "/home/synapse/src/synapse/app/_base.py", line 144, in start_reactor
    run()
  File "/home/synapse/src/synapse/app/_base.py", line 128, in run
    run_command()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 1318, in run
    self.mainLoop()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 1331, in mainLoop
    reactorBaseSelf.doIteration(t)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/epollreactor.py", line 244, in doPoll
    log.callWithLogger(selectable, _drdw, selectable, fd, event)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/log.py", line 101, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/log.py", line 85, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/posixbase.py", line 686, in _doReadOrWrite
    why = selectable.doRead()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/tcp.py", line 247, in doRead
    return self._dataReceived(data)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/tcp.py", line 252, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/endpoints.py", line 149, in dataReceived
    return self._wrappedProtocol.dataReceived(data)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/protocols/tls.py", line 324, in dataReceived
    self._flushReceiveBIO()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/protocols/tls.py", line 290, in _flushReceiveBIO
    ProtocolWrapper.dataReceived(self, bytes)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/protocols/policies.py", line 109, in dataReceived
    self.wrappedProtocol.dataReceived(data)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1641, in dataReceived
    self._parser.dataReceived(bytes)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/_newclient.py", line 374, in dataReceived
    HTTPParser.dataReceived(self, data)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/protocols/basic.py", line 554, in dataReceived
    why = self.rawDataReceived(data)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/_newclient.py", line 292, in rawDataReceived
    self.bodyDecoder.dataReceived(data)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/http.py", line 1767, in dataReceived
    finishCallback(data[contentLength:])
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/_newclient.py", line 435, in _finished
    self.finisher(rest)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1033, in dispatcher
    return func(*args, **kwargs)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1599, in _finishResponse_WAITING
    self._disconnectParser(reason)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1623, in _disconnectParser
    parser.connectionLost(reason)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/_newclient.py", line 545, in connectionLost
    self.response._bodyDataFinished()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1033, in dispatcher
    return func(*args, **kwargs)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1273, in _bodyDataFinished_CONNECTED
    self._bodyProtocol.connectionLost(reason)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/web/client.py", line 2279, in connectionLost
    self.deferred.callback(b"".join(self.dataBuffer))
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 662, in callback
    self._startRunCallbacks(result)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 859, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1751, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1664, in _inlineCallbacks
    status.deferred.callback(getattr(e, "value", None))
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 662, in callback
    self._startRunCallbacks(result)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 859, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1751, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    result = current_context.run(gen.send, result)
Traceback (most recent call last):
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    result = current_context.run(gen.send, result)
StopIteration: {'keys': [{'kty': 'RSA', 'kid': 'eXaunmL', 'use': 'sig', 'alg': 'RS256', 'n': '[redacted]', 'e': 'AQAB'}, {'kty': 'RSA', 'kid': '86D88Kf', 'use': 'sig', 'alg': 'RS256', 'n': '[redacted]', 'e': 'AQAB'}, {'kty': 'RSA', 'kid': 'YuyXoY', 'use': 'sig', 'alg': 'RS256', 'n': '1JiU4

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/src/synapse/app/_base.py", line 193, in wrapper
    await cb(*args, **kwargs)
  File "/home/synapse/src/synapse/app/homeserver.py", line 371, in start
    await _base.start(hs)
  File "/home/synapse/src/synapse/app/_base.py", line 378, in start
    hs.start_listening()
  File "/home/synapse/src/synapse/app/homeserver.py", line 290, in start_listening
    self._listener_http(self.config, listener)
  File "/home/synapse/src/synapse/app/homeserver.py", line 103, in _listener_http
    resources.update(self._configure_named_resource(name, res.compress))
  File "/home/synapse/src/synapse/app/homeserver.py", line 193, in _configure_named_resource
    "/_synapse/admin": AdminRestResource(self),
  File "/home/synapse/src/synapse/rest/admin/__init__.py", line 210, in __init__
  File "/home/synapse/src/synapse/rest/admin/__init__.py", line 228, in register_servlets
    UserMembershipRestServlet(hs).register(http_server)
  File "/home/synapse/src/synapse/rest/admin/users.py", line 876, in __init__
    allowed_values=(
  File "/home/synapse/src/synapse/server.py", line 183, in _get
    building[0] = True
  File "/home/synapse/src/synapse/server.py", line 609, in get_media_repository
    # build the media repo resource. This indirects through the HomeServer
  File "/home/synapse/src/synapse/rest/media/v1/media_repository.py", line 76, in __init__
    self.max_upload_size = hs.config.max_upload_size
  File "/home/synapse/src/synapse/config/_base.py", line 340, in __getattr__
    Args:
  File "/home/synapse/src/synapse/config/_base.py", line 363, in _get_unclassed_config
    Args:
AttributeError: ('max_upload_size', "not found in ['modules', 'server', 'tls', 'federation', 'caches', 'database', 'logging', 'ratelimiting', 'media', 'captcha', 'voip', 'registration', 'account_validity', 'metrics', 'api', 'appservice', 'key', 'saml2', 'oidc', 'cas', 'sso', 'jwt', 'auth', 'email', 'authproviders', 'push', 'spamchecker', 'room', 'groups', 'userdirectory', 'consent', 'stats', 'servernotices', 'roomdirectory', 'thirdpartyrules', 'tracing', 'worker', 'redis', 'experimental']")

Note that max_upload_size: 100M is present in the homeserver config file.
@anoadragon453 anoadragon453 added the X-Release-Blocker Must be resolved before making a release label Aug 17, 2021
@anoadragon453
Copy link
Member Author

From an initial glance it looks as if we're attempting to register admin servlets before the media config has been initialised.

@anoadragon453
Copy link
Member Author

I suspect this may be caused by https://github.com/matrix-org/synapse/pull/10558/files#diff-9188025b7b62bc7582fb9c2554bc8161f1fb0333141313b828cee979702aa276R833.

@anoadragon453
Copy link
Member Author

anoadragon453 commented Aug 17, 2021 •
edited
Loading

It looks like this is caused by the main process having the media repository code disabled (as we have separate media repository workers), yet is calling hs.get_media_repository() when initialising the admin API.

I suspect the solution is to disable DELETE /users/{userId}/media on workers that don't load the media repository (or find a way to allow the data to be modified without going through the media_repository object).

@anoadragon453 anoadragon453 mentioned this issue Aug 17, 2021
Merged
@anoadragon453
Copy link
Member Author

Fixed by #10628.

@MadLittleMods MadLittleMods added A-Media-Repository Uploading, downloading images and video, thumbnailing A-Admin-API labels Jun 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
A-Admin-API A-Media-Repository Uploading, downloading images and video, thumbnailing X-Release-Blocker Must be resolved before making a release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Mount /_synapse/admin/v1/users/{userId}/media admin API on media workers only matrix-org/synapse
2 participants
@MadLittleMods @anoadragon453