[Fixed] DataCache Find Method

[cschuchardt88]

Description

As you may or may not know, Find does not return all records within range. Also Find wouldn't allow and build the last key correctly. Now the ranges and last search key are built and returned correctly.

Change Log

• Added Find now builds the correct key ending.
• Added Find now return all the records within the range (Including the start and end key).
• Fixed unit tests TestFindRange in UT_DataCache class. (to check final return of records).

Type of change

• Optimization (the change is only an optimization)
• Style (the change is only a code style for better maintenance or standard purpose)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

• Unit Testing

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[cschuchardt88]

@roman-khimov I reverted back that change. Also @superboyiii tested data on mainnet.

[superboyiii]

@roman-khimov I reverted back that change. Also @superboyiii tested data on mainnet.

Mainnet data show it's compatible till now, but to ensure everything is the same, if behaviour is different, hardfork flag is necessary.

[cschuchardt88]

Reverted FindRange and TestFindRange.

Reused FindInternal method

[roman-khimov]

Regarding this. I understand what you're talking about, but I've failed to reproduce the problem, see master...roman-khimov:neo:level-fun

Please tell me how to make it fail with the current code. It was built (d6620cb/#2819) around this snippet in fact: https://github.com/syndtr/goleveldb/blob/v1.0.0/leveldb/util/range.go#L20 (https://pkg.go.dev/github.com/syndtr/goleveldb@v1.0.0/leveldb#DB.NewIterator is also relevant here).

And the intention is exactly to get to the next item and make a step back which completely solves this 0xff problem, you don't need to guess how many 0xff you need.

[cschuchardt88]

Hope this helps shows the issue that I'm fixing
In this PR I already reverted MyFindRange method's functionality, but i can add it back if you would like.

// myDataCache.Find() and myDataCache.FindRange() uses old way
// MyFind() and MyFindRange() uses new way
var dateTime = new DateTime(2025, 01, 01);
var timeStamp1 = (ulong)new DateTimeOffset(dateTime).ToUnixTimeMilliseconds();
var timeStamp2 = (ulong)new DateTimeOffset(dateTime.AddDays(1)).ToUnixTimeMilliseconds();
var recordId = 8888u;
StorageKey key1 = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp1)       // Timestamp
    .AddBigEndian(recordId);        // Record Id
StorageKey key2 = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp1)       // Timestamp
    .AddBigEndian(recordId + 1);    // Record Id
StorageKey key3 = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp2)       // Timestamp
    .AddBigEndian(recordId + 2);    // Record Id
StorageKey key4 = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp2)       // Timestamp
    .AddBigEndian(recordId + 3);    // Record Id

myDataCache.Add(key1, value1);
myDataCache.Add(key2, value1);
myDataCache.Add(key3, value1);
myDataCache.Add(key4, value1);

// Used to get all records for start date 01/01/2025
var startSearchKey = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp1)
    .ToArray();

// Used to get all records for end date 01/01/2026
var endSearchKey = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp2)
    .AddBigEndian(uint.MaxValue)    // Biggest Record Id
    .ToArray();

byte[] keyId = [0xff, 0xff,]; // Contract Id
var action = () => _ = myDataCache.Find(keyId, SeekDirection.Backward); // Get All records of Contract with Id int.MaxValue
Assert.ThrowsException<ArgumentException>(action); // Breaks your Find() method and could leave records leftover or missed; Example ContractManagement.Destory

// Get all records for contract with id of int.MaxValue
var results = MyFind(keyId, SeekDirection.Backward); // Get All records of Contract with Id int.MaxValue
Assert.AreEqual(4, results.Count());

// Doesn't work in descending order
results = myDataCache.FindRange(startSearchKey, endSearchKey, SeekDirection.Backward); // Get records in descending order
Assert.AreEqual(0, results.Count());

// Workaround to get in descending order
results = myDataCache.FindRange(endSearchKey, startSearchKey, SeekDirection.Backward); // Get records in descending order
Assert.AreEqual(4, results.Count());

// New Method fixes start and end with backwards searching based off description of method
results = MyFindRange(startSearchKey, endSearchKey, SeekDirection.Backward); // Get records in descending order
Assert.AreEqual(4, results.Count());

[roman-khimov]

var action = () => _ = myDataCache.Find(keyId, SeekDirection.Backward); // Get All records of Contract with Id int.MaxValue
Assert.ThrowsException<ArgumentException>(action); // Breaks your Find() method and could leave records leftover or missed; Example ContractManagement.Destory

Not a problem practically, these are evaluated in natural order.

// Workaround to get in descending order
results = myDataCache.FindRange(endSearchKey, startSearchKey, SeekDirection.Backward); // Get records in descending order
Assert.AreEqual(4, results.Count());

That's just the way FindRange() works. And there is code relying on this behavior in NEO and Role Management native contracts at least.

So, what exactly are we trying to fix here?

Also, regarding 0xff --- it's not needed and it doesn't improve anything, current code is correct since your initial seek for backwards iteration will routinely overshoot past the desired prefix:

  // Position at the first key in the source that is at or past target.
  // The iterator is Valid() after this call iff the source contains
  // an entry that comes at or past target.
  virtual void Seek(const Slice& target) = 0;