Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fixed] DataCache Find Method #3776

Closed
wants to merge 7 commits into from
+33 −44
Closed

[Fixed] DataCache Find Method #3776

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cd2334d
Fixed DataCache Find Methods
cschuchardt88 Feb 19, 2025
ab40b92
Update tests
cschuchardt88 Feb 19, 2025
8242c0f
Revert for @roman-khimov. So contracts don't change.
cschuchardt88 Feb 19, 2025
962e096
Update src/Neo/Persistence/DataCache.cs
cschuchardt88 Feb 19, 2025
9968c78
Merge branch 'master' into fix/datacache-find-methods
cschuchardt88 Feb 20, 2025
24d4e3e
Revert `FindRange` method and create reuse `FindInternal`
cschuchardt88 Feb 20, 2025
17a3ffe
Merge branch 'master' into fix/datacache-find-methods
cschuchardt88 Feb 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 33 additions & 44 deletions src/Neo/Persistence/DataCache.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,66 +197,55 @@ public void Delete(StorageKey key)
/// <summary>
/// Finds the entries starting with the specified prefix.
/// </summary>
/// <param name="key_prefix">The prefix of the key.</param>
/// <param name="direction">The search direction.</param>
/// <param name="keyOrPrefix">The prefix of the key.</param>
/// <param name="seekDirection">The search direction.</param>
/// <returns>The entries found with the desired prefix.</returns>
public IEnumerable<(StorageKey Key, StorageItem Value)> Find(byte[]? key_prefix = null, SeekDirection direction = SeekDirection.Forward)
public IEnumerable<(StorageKey Key, StorageItem Value)> Find(byte[]? keyOrPrefix = null, SeekDirection seekDirection = SeekDirection.Forward)
{
var seek_prefix = key_prefix;
if (direction == SeekDirection.Backward)
{
if (key_prefix == null)
{
// Backwards seek for null prefix is not supported for now.
throw new ArgumentNullException(nameof(key_prefix));
}
if (key_prefix.Length == 0)
{
// Backwards seek for zero prefix is not supported for now.
throw new ArgumentOutOfRangeException(nameof(key_prefix));
}
seek_prefix = null;
for (var i = key_prefix.Length - 1; i >= 0; i--)
{
if (key_prefix[i] < 0xff)
{
seek_prefix = key_prefix.Take(i + 1).ToArray();
// The next key after the key_prefix.
seek_prefix[i]++;
break;
}
}
if (seek_prefix == null)
{
throw new ArgumentException($"{nameof(key_prefix)} with all bytes being 0xff is not supported now");
}
}
return FindInternal(key_prefix, seek_prefix, direction);
if (seekDirection == SeekDirection.Backward && keyOrPrefix == null)
throw new ArgumentNullException(nameof(keyOrPrefix));
if (seekDirection == SeekDirection.Backward && keyOrPrefix?.Length == 0)
throw new ArgumentOutOfRangeException(nameof(keyOrPrefix));

var lastKey = new byte[ApplicationEngine.MaxStorageKeySize];
Array.Fill<byte>(lastKey, 0xff);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding this. I understand what you're talking about, but I've failed to reproduce the problem, see master...roman-khimov:neo:level-fun

Please tell me how to make it fail with the current code. It was built (d6620cb/#2819) around this snippet in fact: https://github.com/syndtr/goleveldb/blob/v1.0.0/leveldb/util/range.go#L20 (https://pkg.go.dev/github.com/syndtr/goleveldb@v1.0.0/leveldb#DB.NewIterator is also relevant here).

And the intention is exactly to get to the next item and make a step back which completely solves this 0xff problem, you don't need to guess how many 0xff you need.

Copy link
Member Author

@cschuchardt88 cschuchardt88 Feb 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hope this helps shows the issue that I'm fixing
In this PR I already reverted MyFindRange method's functionality, but i can add it back if you would like.

// myDataCache.Find() and myDataCache.FindRange() uses old way
// MyFind() and MyFindRange() uses new way
var dateTime = new DateTime(2025, 01, 01);
var timeStamp1 = (ulong)new DateTimeOffset(dateTime).ToUnixTimeMilliseconds();
var timeStamp2 = (ulong)new DateTimeOffset(dateTime.AddDays(1)).ToUnixTimeMilliseconds();
var recordId = 8888u;
StorageKey key1 = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp1)       // Timestamp
    .AddBigEndian(recordId);        // Record Id
StorageKey key2 = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp1)       // Timestamp
    .AddBigEndian(recordId + 1);    // Record Id
StorageKey key3 = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp2)       // Timestamp
    .AddBigEndian(recordId + 2);    // Record Id
StorageKey key4 = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp2)       // Timestamp
    .AddBigEndian(recordId + 3);    // Record Id

myDataCache.Add(key1, value1);
myDataCache.Add(key2, value1);
myDataCache.Add(key3, value1);
myDataCache.Add(key4, value1);

// Used to get all records for start date 01/01/2025
var startSearchKey = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp1)
    .ToArray();

// Used to get all records for end date 01/01/2026
var endSearchKey = new KeyBuilder(int.MaxValue, 0xff)
    .AddBigEndian(timeStamp2)
    .AddBigEndian(uint.MaxValue)    // Biggest Record Id
    .ToArray();

byte[] keyId = [0xff, 0xff,]; // Contract Id
var action = () => _ = myDataCache.Find(keyId, SeekDirection.Backward); // Get All records of Contract with Id int.MaxValue
Assert.ThrowsException<ArgumentException>(action); // Breaks your Find() method and could leave records leftover or missed; Example ContractManagement.Destory

// Get all records for contract with id of int.MaxValue
var results = MyFind(keyId, SeekDirection.Backward); // Get All records of Contract with Id int.MaxValue
Assert.AreEqual(4, results.Count());

// Doesn't work in descending order
results = myDataCache.FindRange(startSearchKey, endSearchKey, SeekDirection.Backward); // Get records in descending order
Assert.AreEqual(0, results.Count());

// Workaround to get in descending order
results = myDataCache.FindRange(endSearchKey, startSearchKey, SeekDirection.Backward); // Get records in descending order
Assert.AreEqual(4, results.Count());

// New Method fixes start and end with backwards searching based off description of method
results = MyFindRange(startSearchKey, endSearchKey, SeekDirection.Backward); // Get records in descending order
Assert.AreEqual(4, results.Count());

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

var action = () => _ = myDataCache.Find(keyId, SeekDirection.Backward); // Get All records of Contract with Id int.MaxValue
Assert.ThrowsException<ArgumentException>(action); // Breaks your Find() method and could leave records leftover or missed; Example ContractManagement.Destory

Not a problem practically, these are evaluated in natural order.

// Workaround to get in descending order
results = myDataCache.FindRange(endSearchKey, startSearchKey, SeekDirection.Backward); // Get records in descending order
Assert.AreEqual(4, results.Count());

That's just the way FindRange() works. And there is code relying on this behavior in NEO and Role Management native contracts at least.

So, what exactly are we trying to fix here?

Also, regarding 0xff --- it's not needed and it doesn't improve anything, current code is correct since your initial seek for backwards iteration will routinely overshoot past the desired prefix:

  // Position at the first key in the source that is at or past target.
  // The iterator is Valid() after this call iff the source contains
  // an entry that comes at or past target.
  virtual void Seek(const Slice& target) = 0;


keyOrPrefix ??= [];

if (keyOrPrefix.Length > 0)
keyOrPrefix.CopyTo(lastKey, 0);

return seekDirection == SeekDirection.Backward ?
FindInternal(lastKey, keyOrPrefix, seekDirection) :
FindInternal(keyOrPrefix, lastKey, seekDirection);
}

private IEnumerable<(StorageKey Key, StorageItem Value)> FindInternal(byte[]? key_prefix, byte[]? seek_prefix, SeekDirection direction)
private IEnumerable<(StorageKey Key, StorageItem Value)> FindInternal(byte[] startKeyOrPrefix, byte[] lastKeyOrPrefix, SeekDirection seekDirection)
{
foreach (var (key, value) in Seek(seek_prefix, direction))
if (key_prefix == null || key.ToArray().AsSpan().StartsWith(key_prefix))
var comparer = seekDirection == SeekDirection.Forward
? ByteArrayComparer.Default
: ByteArrayComparer.Reverse;
foreach (var (key, value) in Seek(startKeyOrPrefix, seekDirection))
if (comparer.Compare(key.ToArray(), lastKeyOrPrefix) <= 0)
yield return (key, value);
else if (direction == SeekDirection.Forward || (seek_prefix == null || !key.ToArray().SequenceEqual(seek_prefix)))
else
yield break;
}

/// <summary>
/// Finds the entries that between [start, end).
/// </summary>
/// <param name="start">The start key (inclusive).</param>
/// <param name="end">The end key (exclusive).</param>
/// <param name="direction">The search direction.</param>
/// <param name="startKeyOrPrefix">The start key (inclusive).</param>
/// <param name="lastKeyOrPrefix">The end key (exclusive).</param>
/// <param name="seekDirection">The search direction.</param>
/// <returns>The entries found with the desired range.</returns>
public IEnumerable<(StorageKey Key, StorageItem Value)> FindRange(byte[] start, byte[] end, SeekDirection direction = SeekDirection.Forward)
public IEnumerable<(StorageKey Key, StorageItem Value)> FindRange(byte[] startKeyOrPrefix, byte[] lastKeyOrPrefix, SeekDirection seekDirection = SeekDirection.Forward)
{
var comparer = direction == SeekDirection.Forward
var comparer = seekDirection == SeekDirection.Forward
? ByteArrayComparer.Default
: ByteArrayComparer.Reverse;
foreach (var (key, value) in Seek(start, direction))
if (comparer.Compare(key.ToArray(), end) < 0)
foreach (var (key, value) in Seek(startKeyOrPrefix, seekDirection))
if (comparer.Compare(key.ToArray(), lastKeyOrPrefix) < 0)
yield return (key, value);
else
yield break;
Expand Down