Event buffer for structlog #4358
Conversation
core/dbt/events/functions.py
Outdated
| # create the global event history buffer with a max size of 100k records | ||
| # TODO: make the maxlen something configurable from the command line via args(?) | ||
| global EVENT_HISTORY | ||
| EVENT_HISTORY: Deque = deque(maxlen=100000) |
There was a problem hiding this comment.
@kwigley is this going to fit the bill for what's needed in Client/Server? It should make a globally available deque of events so you can do whatever's needed.
There was a problem hiding this comment.
awesome! Ya, I believe so.
test/unit/test_events.py
Outdated
| # ensure events drop from the front of the buffer when buffer maxsize is reached | ||
| def test_buffer_FIFOs(self): | ||
| for n in range(0,100001): |
There was a problem hiding this comment.
FYI -- This is adding ~10 seconds to the unit test run time... can be fixed by correcting TODO on core/dbt/events/functions.py line 24.
There was a problem hiding this comment.
ahh. making it configurable so we can test with a smaller queue.
| global EVENT_HISTORY | ||
| if len(EVENT_HISTORY) == (100000 - 1): | ||
| EVENT_HISTORY.append(e) | ||
| fire_event(EventBufferFull()) |
There was a problem hiding this comment.
what if someone is running on a machine so small that memory runs out before this hard-coded limit? would we never log it?
There was a problem hiding this comment.
I'm ok looking into this after the RC3 since I feel like it will be a rare case
core/dbt/events/functions.py
Outdated
| if len(EVENT_HISTORY) == (100000 - 1): | ||
| EVENT_HISTORY.append(e) | ||
| fire_event(EventBufferFull()) | ||
| else: |
There was a problem hiding this comment.
I'd prefer this if/else to be in the form:
if we_need_to_log:
log()
append()|
@nathaniel-may we can merge w/o the 3.9 Mac test succeeding. It looks like a problem with Actions. The Macs are known to have issues... |
Add Internal event buffer Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com> automatic commit by git-black, original commits: 3a904a8
resolves #4320
Description
Adds a global event history buffer using a FIFO strategy to keep memory usage reasonable. Can be used when wrapping dbt or in general to access past events.
Checklist
CHANGELOG.mdand added information about my change