Skip to content

Commit

Permalink
Wrap the normalize_file mongo operation inside a try/catch. This ac…
Browse files Browse the repository at this point in the history
…complishes two things: (#2483)

1. It improves process log output, making any follow-on troubleshooting straightforward.

  We go from a generic OperationFailure log, which makes it look like the analysis doc was too big:
  ```
  poetry[1072529]: 2025-01-31 11:24:41,229 [Task 1] [modules.reporting.mongodb] ERROR: Got MongoDB OperationFailure, code 10334
  poetry[1072529]: 2025-01-31 11:24:41,229 [Task 1] [modules.reporting.mongodb] WARNING: Deleting behavior process tree children from results.
  ```

  To this, which actually indicates that the hook failed and why:

  ```
poetry[1108630]: 2025-01-31 15:25:55,981 [Task 2] [dev_utils.mongo_hooks] ERROR: Mongo hook 'normalize_files' failed with code 10334: BSONObj size: 18852184 (0x11FA958) is invalid. Size must be between 0 and 16793600(16MB) First element: q: { _id: "...461a8" }, full error: {'ok': 0.0, 'errmsg': 'BSONObj size: 18852184 (0x11FA958) is invalid. Size must be between 0 and 16793600(16MB) First element: q: { _id: "...461a8" }', 'code': 10334, 'codeName': 'BSONObjectTooLarge'}
  ```

2. If the `normalize_file` bulk write operation errors out, it no longer gets caught by the exception handler in `modules/reporting/mongodb.MongoDB.run` leading to the behaviour process tree children being removed from the analysis report (See logs above - not good!).

What this does not fix, however, is the problem with large `strings` data that prevented it from being written to the `files` collection. It does provide a lot more visibility though, in addition to preventing the analysis document from being arbitrarily cut-down if an error does occur in the hook.

On the topic of other hooks, there are probably other hooks that require the same treatment, but this is a good place to start.
  • Loading branch information
josh-feather authored Feb 3, 2025
1 parent e7d0bc0 commit 60ec312
Showing 1 changed file with 7 additions and 3 deletions.
10 changes: 7 additions & 3 deletions dev_utils/mongo_hooks.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import itertools
import logging

from pymongo import UpdateOne
from pymongo import UpdateOne, errors

from dev_utils.mongodb import (
mongo_bulk_write,
Expand Down Expand Up @@ -82,8 +82,12 @@ def normalize_files(report):
request = normalize_file(file_dict, report["info"]["id"])
if request:
requests.append(request)
if requests:
mongo_bulk_write(FILES_COLL, requests, ordered=False)

try:
if requests:
mongo_bulk_write(FILES_COLL, requests, ordered=False)
except errors.OperationFailure as exc:
log.error("Mongo hook 'normalize_files' failed with code %d: %s", exc.code, exc)

return report

Expand Down

0 comments on commit 60ec312

Please sign in to comment.