Add used() API in AddressAllocator

[ylzh10]

Summary of the PR

bit dumb here, but I thought it should be a common use case to get the available memories in AddressAllocator, while i don't see a way on how to get this info today.
Or there is already a way to get this info from allocator and somehow i missed it? (please lmk)

So add a new private var used

• init it w/ 0
• inc/dec it when allocate/free()
• expose it in a new public API used() so we can infer how many available memories are left (We don't want to expose available() directly since it can be confusing/inaccurate due to fragmentations most times)

Requirements

Before submitting your PR, please make sure you addressed the following
requirements:

• All commits in this PR have Signed-Off-By trailers (with
  git commit -s), and the commit message has max 60 characters for the
  summary and max 75 characters for each description line.
• All added/changed functionality has a corresponding unit/integration
  test.
• All added/changed public-facing functionality has entries in the "Upcoming
  Release" section of CHANGELOG.md (if no such section exists, please create one).
• Any newly added unsafe code is properly documented.

[ylzh10]

dup of #94

@roypat thanks for reviewing my PR.
My usecase is simply to allocate/free some memory and then check the rest available memories in the addr space to account for the memory usage and potential mem leaks.
yes i understand that due to fragmentation, available() may return more than the actual "available" memory spaces, but in our use case, when we do allocate(size, alignment), size is always aligned with alignment as well, not just the start addr, so say alignment == 4k, we always allocate a memory block of multiple of 4k size and since its start addr is also aligned w/ 4k, this available info should be "more accurate" (tho still has fragmentations), and we'd like to use this info to as a indicator to tell API users the available memory spaces (best effort tho : )).

[AlexandruCihodaru]

dup of #94

@roypat thanks for reviewing my PR. My usecase is simply to allocate some memory and then check the rest available memories in the addr space to account for the memory usage and potential mem leaks. yes i understand that due to fragmentation, available() may return more than the actual "available" memory spaces, but in our use case, when we do allocate(size, alignment), size is always aligned with alignment as well, not just the start addr, so say alignment == 4k, we always allocate a memory block of multiple of 4k size and since its start addr is also aligned w/ 4k, this available info should be accurate, right?

You can have a memory pool from 0 to 12K. You can allocate a block with ExactMatch from 4k to 8k. Now you have 8k of available memory but you can not allocate a 8k blob. I think that in this scenario "available" as it is implemented is misleading.

My usecase is simply to allocate some memory and then check the rest available memories in the addr space to account for the memory usage and potential mem leaks

Maybe you could check that the memory that has been freed is the same as memory that has been allocated.

[ylzh10]

dup of #94
@roypat thanks for reviewing my PR. My usecase is simply to allocate some memory and then check the rest available memories in the addr space to account for the memory usage and potential mem leaks. yes i understand that due to fragmentation, available() may return more than the actual "available" memory spaces, but in our use case, when we do allocate(size, alignment), size is always aligned with alignment as well, not just the start addr, so say alignment == 4k, we always allocate a memory block of multiple of 4k size and since its start addr is also aligned w/ 4k, this available info should be accurate, right?

You can have a memory pool from 0 to 12K. You can allocate a block with ExactMatch from 4k to 8k. Now you have 8k of available memory but you can not allocate a 8k blob. I think that in this scenario "available" as it is implemented is misleading.

My usecase is simply to allocate some memory and then check the rest available memories in the addr space to account for the memory usage and potential mem leaks

Maybe you could check that the memory that has been freed is the same as memory that has been allocated.

You can have a memory pool from 0 to 12K. You can allocate a block with ExactMatch from 4k to 8k. Now you have 8k of available memory but you can not allocate a 8k blob. I think that in this scenario "available" as it is implemented is misleading.

yeah, i understand this fragmentation and available can never be accurate due to this. It's expected in our use case and would give some signal on the max available free memory and i can add more documentation in this api for this behavior.
The issue we have today when using the AddressAllocator is it doesn't have any public API to allow callers to peek into the any free memory size info. Alternatively, exposing the interval tree and let users count the free memory blocks would let users ctrl how they want to count total "available" memory (or the max avail memory chunk size), but i feel it might be too much.

Maybe you could check that the memory that has been freed is the same as memory that has been allocated.

the alloc/free/get_avail calls happen asynchronously and some allocations may persist a longer time than others before they're freed, so we typically check the memory usage when at the beginning/end of the program/func to make sure they match, no mem leaks rather than accumulate each alloc/free calls.

[roypat]

maybe we can have the inverse of this API instead? Something that returns how many bytes are used from the allocator range? That should cover what you want to do, and be less misleading, API-wise.

[ylzh10]

maybe we can have the inverse of this API instead? Something that returns how many bytes are used from the allocator range? That should cover what you want to do, and be less misleading, API-wise.

sg, that works too, any API that can expose the available/used memory in the address space/interval tree would be helpful. let me update the code.

[ylzh10]

@roypat @AlexandruCihodaru can you help review the changes again and see if you have any further comments?

[AlexandruCihodaru]

Overall LGTM. Please squash your commits and clean you commit messages.

[AlexandruCihodaru]

I am not sure that expect is the best solution here. I think it would be better to use ok_or(Error::Overflow) and propagate the error with ?

[AlexandruCihodaru]

Same here but with ok_or(Error:Underflow)

[roypat]

This error conditions is impossible to hit unless the logic in the library is fundamentally broken. In that case, panicking is the correct thing to do I'd say - if we just propagate with ? then the allocate will be left in an inconsistent state anyway, which I don't think would be recoverable

[ylzh10]

Same here but with ok_or(Error:Underflow)

same thought as @roypat, it should be ok to panic here since the checked_add/sub() should never fail unless allocate/free() before these two calls went very wrong like it's capable of allocating some total >=usize:MAX memory or free some non-existing memory calling used < 0.

[AlexandruCihodaru]

In that case, panicking is the correct thing to do I'd say - if we just propagate with ? then the allocate will be left in an inconsistent state anyway, which I don't think would be recoverable

You are right that the interval tree will be likely messed up. But the reason for propagating the error is to allow the caller to clean up after themselves before exiting.

[ylzh10]

In that case, panicking is the correct thing to do I'd say - if we just propagate with ? then the allocate will be left in an inconsistent state anyway, which I don't think would be recoverable

You are right that the interval tree will be likely messed up. But the reason for propagating the error is to allow the caller to clean up after themselves before exiting.

i don't really expect caller to handle this error externally, since it's non-recoverable and the program should crash if it hits here.
How would you expect caller to handle/clean up this, like undo the allocate/free()? but the previous allocate/free on the interval tree has succeeded before the self.used.add/sub() and callers have no way to manipulate self.used w/o changing the interval tree again, in which case the interval tree and used() can never be "consistent" w/ each other. IMHO, panic inside allocator would give callers a signal that this is non-recoverable, they should seek help to fix the allocator rather than remediate on their side