Stability

[bonomani]

Memory Leak / Memory Increase

Given the complexity of this Perl program, a slight memory leak/increase is anticipated (and observed). Yet, minimizing this is among our primary objectives:

• On Linux (WIP)
• On BSD?
• An issue previously identified in xymon-rrd should now be resolved. (Which version?)

Plan

Step #1: Detect/Identify Sources of Problems (WIP)

• Valgrind: Testing on Xymon rrd suggests it does not leak. It seems that Xymon never frees memory, although it is expected to do so upon stopping.
• Explore different tools:
  • Devel::Cycle: Testing for circular references (No output -> No Leak?)
  • Devel::MAT (https://metacpan.org/dist/Devel-MAT/view/lib/Devel/MAT/UserGuide.pod) (WIP)
  • Any others?
• Divide the program into smaller segments to pinpoint the issue.
• Considering external libraries? Try the Pure Perl (PP) version first.
• Compilation of Xymon rrd emits numerous warnings, some of which are fixed in version 4.4.
  • Therabithia utilizes rrd-cache, set to be included in the standard Xymon version 4.4. (Will this influence the outcome?)

Step #2: Fix/Apply Corrections: Work in Progress

• Address all Devmon errors: Execute Devmon in the foreground and monitor logs (largely DONE).
• Adhere to Perl Best Practices (WIP).

Step #3: Control Results

Current Status

Step #1: Some Problem Sources Identified

1. If stale forks are terminated, their memory isn't properly released, resulting in a memory leak.
2. Forks replicate the main process, thus they're not memory-optimized (10 Children + 1 Master = Memory × 11, which contradicts the expectation due to Copy-On-Write efficiency)
   • A pool of 10 processes is currently employed (why this limitation? Increasing the number could potentially accommodate more requests).
   • Implemented following a Boss-Worker model.
   • Worker processes are re-forked upon unexpected termination.

Step #2: Implement Corrections

1. Prevent fork termination / Rewrote the SNMPWalk engine to manage retries (COMPLETED v0.23.07, for SNMP_Session only)

   • Partial retries
   • Retry only when there is sufficient time left to complete polling.
   • Implemented dynamic timeout and retry optimization.

2. Memory Duplication with Forks: A comprehensive review of the forking process is underway (WIP).

Step #3: Control (WIP)

1. Avoid terminating forks / Rewrite of the SNMPWalk engine to manage retries
   • Elimination of unexpected process termination: A significant and stable improvement observed over several months.
   • Memory increment persists: ?????, albeit seemingly reduced compared to before.
     • It's uncertain if this pertains to a devmon issue: Perl processes do not appear to consume excessive memory.
     • The issue seems to sporadically arise (challenging to pinpoint precisely when), typically suspected during network outages. (This implies when SNMP polling is incomplete, as it no longer fails: no terminated processes).
     • Interpreting memory information (from xymon) or identifying which process utilized the memory remains unclear.
     • RRD is under suspicion. (The next step is to remove all graphs. Furthermore, the introduction of RRD-cache in Xymon version 4.4, as seen in Therabithia, postpones RRD troubleshooting, but this problem source is expected to be confirmed later.)

Number of Forks

• Description: The fork number indicates the count of SNMP polling processes operating concurrently.
• The default is set to 10.
• An increased number of forks (>10) is experimental and appears to cause process blockages (the exact reason is yet to be determined).