Memory leak in instrumentation-pg #2479

yandrushchak · 2024-10-11T14:40:05Z

What version of OpenTelemetry are you using?

@opentelemetry/instrumentation-pg: 0.45.1

What version of Node are you using?

v20.15.1

What did you do?

Used @opentelemetry/instrumentation-pg with TypeORM and pg-pool. Received warning about possible memory leak, which happens since listeners are constantly attached for each query. Partial stack trace:

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 connect listeners added to [BoundPool]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
    at genericNodeError (node:internal/errors:984:15)
    at wrappedFn (node:internal/errors:538:14)
    at _addListener (node:events:593:17)
    at BoundPool.addListener (node:events:611:10)
    at BoundPool.connect (c:/test/node_modules\@opentelemetry\instrumentation-pg\src\instrumentation.ts:461:12)
    at <anonymous> (c:/test/node_modules\typeorm\driver\src\driver\postgres\PostgresDriver.ts:1175:25)
    at new Promise (<anonymous>)
    at PostgresDriver.obtainMasterConnection (c:/test/node_modules\typeorm\driver\src\driver\postgres\PostgresDriver.ts:1174:16)
    at PostgresQueryRunner.connect (c:/test/node_modules\typeorm\driver\src\driver\postgres\PostgresQueryRunner.ts:111:18)
    at PostgresQueryRunner.query (c:/test/node_modules\typeorm\driver\src\driver\postgres\PostgresQueryRunner.ts:248:47)
....

What did you expect to see?

No memory leak.

Additional context

This issue was introduced in this commit - 12adb43#diff-1360de1aed58c38c5cbd3a1759a8af761049ea2c562a477964623fb2358b2730R379
Since pg.Pool.connect is called for each query to obtain connection, duplicated event listeners are attached each time, which would probably also lead to wrong metrics data.

It seems in order to fix it, additional check is needed to make sure event listeners are added only on a first connect call.

The text was updated successfully, but these errors were encountered:

astrot1988 · 2024-10-14T18:23:15Z

same issue

clintonb · 2024-10-15T02:04:21Z

We observed this issue after upgrading from @opentelemetry/auto-instrumentations-node 0.46.1 to 0.50.1.

clintonb · 2024-10-15T02:26:31Z

Here is a stack trace from our container startup. The line in instrumentation.js corresponds with:

opentelemetry-js-contrib/plugins/node/opentelemetry-instrumentation-pg/src/instrumentation.ts

Lines 454 to 461 in ad560df

    
           this.on('acquire', () => { 
        
             plugin._connectionsCounter = utils.updateCounter( 
        
               this, 
        
               plugin._connectionsCount, 
        
               plugin._connectionPendingRequests, 
        
               plugin._connectionsCounter 
        
             ); 
        
           });

(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 remove listeners added to [BoundPool]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
    at new Promise (<anonymous>)
    at /srv/node_modules/.pnpm/@nestjs+terminus@10.2.3_drcoupryhn6e72cgxfoljijmlu/node_modules/@nestjs/terminus/dist/health-indicator/database/typeorm.health.js:17:71
    at Generator.next (<anonymous>)
    at TypeOrmHealthIndicator.<anonymous> (/srv/node_modules/.pnpm/@nestjs+terminus@10.2.3_drcoupryhn6e72cgxfoljijmlu/node_modules/@nestjs/terminus/dist/health-indicator/database/typeorm.health.js:134:28)
    at TypeOrmHealthIndicator.pingDb (/srv/node_modules/.pnpm/@nestjs+terminus@10.2.3_drcoupryhn6e72cgxfoljijmlu/node_modules/@nestjs/terminus/dist/health-indicator/database/typeorm.health.js:94:16)
    at __awaiter (/srv/node_modules/.pnpm/@nestjs+terminus@10.2.3_drcoupryhn6e72cgxfoljijmlu/node_modules/@nestjs/terminus/dist/health-indicator/database/typeorm.health.js:13:12)
    at new Promise (<anonymous>)
    at /srv/node_modules/.pnpm/@nestjs+terminus@10.2.3_drcoupryhn6e72cgxfoljijmlu/node_modules/@nestjs/terminus/dist/health-indicator/database/typeorm.health.js:17:71
    at Generator.next (<anonymous>)
    at TypeOrmHealthIndicator.<anonymous> (/srv/node_modules/.pnpm/@nestjs+terminus@10.2.3_drcoupryhn6e72cgxfoljijmlu/node_modules/@nestjs/terminus/dist/health-indicator/database/typeorm.health.js:107:40)
    at DataSource.<anonymous> (/srv/node_modules/.pnpm/opentelemetry-instrumentation-typeorm@0.41.0_@opentelemetry+api@1.9.0/node_modules/opentelemetry-instrumentation-typeorm/dist/src/typeorm.js:267:42)
    at ContextAPI.with (/srv/node_modules/.pnpm/@opentelemetry+api@1.9.0/node_modules/@opentelemetry/api/build/src/api/context.js:60:46)
    at AsyncLocalStorageContextManager.with (/srv/node_modules/.pnpm/@opentelemetry+context-async-hooks@1.26.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
    at AsyncLocalStorage.run (node:async_hooks:346:14)
    at /srv/node_modules/.pnpm/opentelemetry-instrumentation-typeorm@0.41.0_@opentelemetry+api@1.9.0/node_modules/opentelemetry-instrumentation-typeorm/dist/src/typeorm.js:267:82
    at TypeormInstrumentation._endSpan (/srv/node_modules/.pnpm/opentelemetry-instrumentation-typeorm@0.41.0_@opentelemetry+api@1.9.0/node_modules/opentelemetry-instrumentation-typeorm/dist/src/typeorm.js:283:30)
    at /srv/node_modules/.pnpm/opentelemetry-instrumentation-typeorm@0.41.0_@opentelemetry+api@1.9.0/node_modules/opentelemetry-instrumentation-typeorm/dist/src/typeorm.js:267:106
    at DataSource.query (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/data-source/DataSource.js:345:42)
    at PostgresQueryRunner.query (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:174:47)
    at PostgresQueryRunner.connect (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:68:18)
    at PostgresDriver.obtainMasterConnection (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/driver/postgres/PostgresDriver.js:870:16)
    at new Promise (<anonymous>)
    at /srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/driver/postgres/PostgresDriver.js:871:25
    at BoundPool.connect (/srv/node_modules/.pnpm/@opentelemetry+instrumentation-pg@0.45.1_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation-pg/build/src/instrumentation.js:306:22)
    at BoundPool.addListener (node:events:611:10)
    at _addListener (node:events:593:17)
    at wrappedFn (node:internal/errors:538:14)
    at genericNodeError (node:internal/errors:984:15)

thecopy · 2024-10-15T08:19:18Z

Same issue, our database calls are now very very slow as well. Downgrading to auto-instrumentations-node: v0.48.0 solved this issue for us

yandrushchak · 2024-10-15T09:03:47Z

As a workaround, downgrade of auto-instrumentation-node to 0.50.0 solves it, since it references "@opentelemetry/instrumentation-pg": "^0.44.0", which doesn't have this issue.

pichlermarc · 2024-10-15T14:04:33Z

FYI @maryliag (component owner)

maryliag · 2024-10-15T16:15:59Z

Hi everyone, thank you for raising this issue! Indeed I made a mistake and was adding several event listeners to the same pool. Although this is obviously not ideal, I want to clarify the comment

duplicated event listeners are attached each time, which would probably also lead to wrong metrics data.

This (wrong metric data) is not happening, because I did noticed the updater was getting called a few times, so I made sure it always sends the correct value, event if called multiple times. I just didn't realize the reason why it was getting called multiple times 😓

Thanks to the input from this issue, I was able to create a fix for it. Also some minor performance improvements on #2484

If you continue seeing this issue on a version with this fix, please let me know!

zuk · 2024-10-15T17:51:09Z

Thanks @maryliag !

We've been seeing some serious performance issues that coincidentally began right after we updated our opentelemetry deps.

Specifically, over the course of several hours, db-pool.connect durations would get slower and slower, eventually growing from a few ms to 2+ seconds until eventually the whole process would come to a halt, waiting for db connections to open. Rolling back @opentelemetry/auto-instrumentations-node ^0.50.1 -> 0.48.0 fixed the problem for now.

julioantonini · 2024-10-15T18:44:19Z

Hi guys!
I was facing the same issue, so for now, I downgraded the @opentelemetry/auto-instrumentations-node to version 0.49.0.
This temporarily fixed the problem until a hotfix is released

"@opentelemetry/auto-instrumentations-node": "0.49.0"

maryliag · 2024-10-15T23:06:02Z

durations would get slower and slower, eventually growing from a few ms to 2+ seconds until eventually the whole process would come to a halt

this seems to align with a new listeners being added on every new connection, which is what the PR I created is fixing, so hopefully this will solve your problem.
I appreciate everybody's patience until this gets merged and added to a new release 🙏 And I apologize for any problems this might have caused.

Also, hope the new metrics related to pool will be helpful 😄

peterabbott · 2024-10-16T02:48:28Z

Expereienced this too, had to rollback to 0.44.0 of instrumentation-pg. Caused a significant performance impact on one of our systems.

peterabbott · 2024-10-16T02:49:48Z

To add, when using the event listeners, they should be a matching remove to avoid this.

https://github.com/open-telemetry/opentelemetry-js-contrib/commit/12adb4354f09ade438cd96340bdfd1f715b5fed3#diff-1360de1aed58c3[…]4623fb2358b2730R379

It would be nice of the metrics could be an opt-in

maryliag · 2024-10-16T04:45:44Z

they should be a matching remove to avoid this

when do you suggest the removal would occur?

maryliag · 2024-10-16T18:33:37Z

@opentelemetry/instrumentation-pg: 0.46.0 has been released with the fix

clintonb · 2024-10-17T23:35:07Z

I still see an error and immense DB slowness with the latest version of the package.

(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 release listeners added to [BoundPool]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

pichlermarc · 2024-10-18T09:05:34Z

@clintonb thanks for the feedback - would you mind sharing the full stacktrace of that error again (for the new version)? I'm trying to track down possible ways this flag could fail.

(the memory leak is currently the highest priority, once we've sorted this out we can tackle performance improvements in a separate issue)

clintonb · 2024-10-18T15:30:17Z

Hmm...trace shows @opentelemetry+instrumentation-pg@0.45.0 in the PNPM path. I think there may be a funky interaction with @opentelemetry/auto-instrumentations-node going on here.

(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 release listeners added to [BoundPool]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
    at fromFirestoreWithSqlAnnotations (/srv/dist/shared/dataAccess/User.js:152:27)
    at getUserByFirestoreID (/srv/dist/shared/dataAccess/User.js:247:46)
    at Repository.findOneOrFail (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/repository/Repository.js:266:29)
    at EntityManager.findOneOrFail (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/entity-manager/EntityManager.js:637:21)
    at EntityManager.<anonymous> (/srv/node_modules/.pnpm/opentelemetry-instrumentation-typeorm@0.41.0_@opentelemetry+api@1.9.0/node_modules/opentelemetry-instrumentation-typeorm/dist/src/typeorm.js:175:42)
    at ContextAPI.with (/srv/node_modules/.pnpm/@opentelemetry+api@1.9.0/node_modules/@opentelemetry/api/build/src/api/context.js:60:46)
    at AsyncLocalStorageContextManager.with (/srv/node_modules/.pnpm/@opentelemetry+context-async-hooks@1.26.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
    at AsyncLocalStorage.run (node:async_hooks:346:14)
    at /srv/node_modules/.pnpm/opentelemetry-instrumentation-typeorm@0.41.0_@opentelemetry+api@1.9.0/node_modules/opentelemetry-instrumentation-typeorm/dist/src/typeorm.js:175:82
    at TypeormInstrumentation._endSpan (/srv/node_modules/.pnpm/opentelemetry-instrumentation-typeorm@0.41.0_@opentelemetry+api@1.9.0/node_modules/opentelemetry-instrumentation-typeorm/dist/src/typeorm.js:283:30)
    at /srv/node_modules/.pnpm/opentelemetry-instrumentation-typeorm@0.41.0_@opentelemetry+api@1.9.0/node_modules/opentelemetry-instrumentation-typeorm/dist/src/typeorm.js:175:106
    at EntityManager.findOne (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/entity-manager/EntityManager.js:596:14)
    at SelectQueryBuilder.<anonymous> (/srv/node_modules/.pnpm/opentelemetry-instrumentation-typeorm@0.41.0_@opentelemetry+api@1.9.0/node_modules/opentelemetry-instrumentation-typeorm/dist/src/typeorm.js:185:37)
    at SelectQueryBuilder.getOne (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/query-builder/SelectQueryBuilder.js:711:36)
    at SelectQueryBuilder.<anonymous> (/srv/node_modules/.pnpm/opentelemetry-instrumentation-typeorm@0.41.0_@opentelemetry+api@1.9.0/node_modules/opentelemetry-instrumentation-typeorm/dist/src/typeorm.js:185:37)
    at SelectQueryBuilder.getRawAndEntities (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/query-builder/SelectQueryBuilder.js:684:40)
    at SelectQueryBuilder.executeEntitiesAndRawResults (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/query-builder/SelectQueryBuilder.js:2034:37)
    at SelectQueryBuilder.loadRawResults (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/query-builder/SelectQueryBuilder.js:2183:43)
    at PostgresQueryRunner.query (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:174:47)
    at PostgresQueryRunner.connect (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:68:18)
    at PostgresDriver.obtainMasterConnection (/srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/driver/postgres/PostgresDriver.js:870:16)
        at new Promise (<anonymous>)
    at /srv/node_modules/.pnpm/typeorm@0.3.17_better-sqlite3@9.4.0_ioredis@5.3.2_pg-query-stream@4.5.3_pg@8.11.3__pg@8.11.3__nbwcw7yzxgjrpbcbytvl7a4bfy/node_modules/typeorm/driver/postgres/PostgresDriver.js:871:25
    at BoundPool.connect (/srv/node_modules/.pnpm/@opentelemetry+instrumentation-pg@0.45.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation-pg/build/src/instrumentation.js:309:22)
    at BoundPool.addListener (node:events:611:10)
    at _addListener (node:events:593:17)
    at wrappedFn (node:internal/errors:538:14)
    at genericNodeError (node:internal/errors:984:15)

nanotower · 2024-10-28T10:24:34Z

It's happening for me as well.
@opentelemetry/auto-instrumentations-node": "^0.50.2", importing
"@opentelemetry/instrumentation-pg": "^0.47.0", which in turn is one minor version above the fix

(node:66) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 acquire listeners added to [BoundPool]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
at genericNodeError (node: internal/errors:984:15)
at wrappedFn (node: internal/errors:538:14)
at _addListener (node:events:593:17)
at BoundPool.addListener (node:events:611:10)
at BoundPooL. connect (/usr/src/app/node modules/@opentelemetry/instrumentation-pg/build/src/instrumentation.js:306:22)
at PostgresDriver.acquireConnection (/usr/src/app/node modules/kysely/dist/cjs/dialect/postgres/postgres-driver.js:21:41)
at RuntimeDriver.acquireConnection (/usr/src/app/node_modules/kysely/dist/cjs/driver/runtime-driver.js:46:47)
at DefaultConnectionProvider.provideConnection (/usr/src/app/node_modules/kysely/dist/cjs/driver/default-connection-provider.js:10:47) at DefaultQueryExecutor.provideConnection (/usr/src/app/node_modules/kysely/dist/cjs/query-executor/default-query-executor.js:22:41) at DefaultQueryExecutor.executeQuery (/us/src/app/node_modules/kysely/dist/cjs/query-executor/query-executor-base.js:36:27

maryliag · 2024-10-28T13:23:46Z

Hey @nanotower , anyway to get more info on your stacktrace? I want to confirm that the version being used is indeed the one you imported.
For example, on the trace above shared by Clinton you can see

at BoundPool.connect (/srv/node_modules/.pnpm/@opentelemetry+instrumentation-pg@0.45.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation-pg/build/src/instrumentation.js:309:22)
    at BoundPool.addListener (node:events:611:10)

that has a clear @opentelemetry+instrumentation-pg@0.45.0, but I don't see the version on your trace itself, just

at BoundPooL. connect (/usr/src/app/node modules/@opentelemetry/instrumentation-pg/build/src/instrumentation.js:306:22)

which points to a line (306) on current version that is a comment

nanotower · 2024-10-28T15:04:46Z

You're right @maryliag . My node_modules are running

"name": "@opentelemetry/instrumentation-pg",
"version": "0.45.1",

Minor version below the one with the fix. Thanks for the heads up.

maryliag · 2024-10-29T23:06:52Z

Since there hasn't been more reports of this issue with the new version, I will close it

yandrushchak added the bug Something isn't working label Oct 11, 2024

pichlermarc added priority:p1 Bugs which cause problems in end-user applications such as crashes, data inconsistencies pkg:instrumentation-pg labels Oct 15, 2024

maryliag mentioned this issue Oct 15, 2024

fix(instrumentation-pg): not add duplicate listeners to pg pool #2484

Merged

maryliag self-assigned this Oct 15, 2024

pichlermarc linked a pull request Oct 16, 2024 that will close this issue

fix(instrumentation-pg): not add duplicate listeners to pg pool #2484

Merged

pichlermarc closed this as completed in #2484 Oct 16, 2024

pichlermarc reopened this Oct 18, 2024

maryliag closed this as completed Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory leak in instrumentation-pg #2479

Memory leak in instrumentation-pg #2479

yandrushchak commented Oct 11, 2024

astrot1988 commented Oct 14, 2024

clintonb commented Oct 15, 2024

clintonb commented Oct 15, 2024

thecopy commented Oct 15, 2024 •

edited

Loading

yandrushchak commented Oct 15, 2024

pichlermarc commented Oct 15, 2024

maryliag commented Oct 15, 2024

zuk commented Oct 15, 2024

julioantonini commented Oct 15, 2024

maryliag commented Oct 15, 2024

peterabbott commented Oct 16, 2024

peterabbott commented Oct 16, 2024 •

edited

Loading

maryliag commented Oct 16, 2024 •

edited

Loading

maryliag commented Oct 16, 2024

clintonb commented Oct 17, 2024

pichlermarc commented Oct 18, 2024 •

edited

Loading

clintonb commented Oct 18, 2024

nanotower commented Oct 28, 2024 •

edited

Loading

maryliag commented Oct 28, 2024

nanotower commented Oct 28, 2024

maryliag commented Oct 29, 2024

Memory leak in instrumentation-pg #2479

Memory leak in instrumentation-pg #2479

Comments

yandrushchak commented Oct 11, 2024

What version of OpenTelemetry are you using?

What version of Node are you using?

What did you do?

What did you expect to see?

Additional context

astrot1988 commented Oct 14, 2024

clintonb commented Oct 15, 2024

clintonb commented Oct 15, 2024

thecopy commented Oct 15, 2024 • edited Loading

yandrushchak commented Oct 15, 2024

pichlermarc commented Oct 15, 2024

maryliag commented Oct 15, 2024

zuk commented Oct 15, 2024

julioantonini commented Oct 15, 2024

maryliag commented Oct 15, 2024

peterabbott commented Oct 16, 2024

peterabbott commented Oct 16, 2024 • edited Loading

maryliag commented Oct 16, 2024 • edited Loading

maryliag commented Oct 16, 2024

clintonb commented Oct 17, 2024

pichlermarc commented Oct 18, 2024 • edited Loading

clintonb commented Oct 18, 2024

nanotower commented Oct 28, 2024 • edited Loading

maryliag commented Oct 28, 2024

nanotower commented Oct 28, 2024

maryliag commented Oct 29, 2024

thecopy commented Oct 15, 2024 •

edited

Loading

peterabbott commented Oct 16, 2024 •

edited

Loading

maryliag commented Oct 16, 2024 •

edited

Loading

pichlermarc commented Oct 18, 2024 •

edited

Loading

nanotower commented Oct 28, 2024 •

edited

Loading