Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert ndarray scalars to Span<T> using the buffer protocol #187

Merged
merged 55 commits into from
Sep 18, 2024
Merged
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
b778aa8
Add numpy as a dependency for integration tests
tonybaloney Sep 5, 2024
73a9e3d
Expose ndarray scalars using the Python Buffer protocol
tonybaloney Sep 5, 2024
082c9b4
Add float scalars and test
tonybaloney Sep 5, 2024
7ffeb1a
Update src/CSnakes.Runtime/Python/IPyBuffer.cs
tonybaloney Sep 5, 2024
5cae82f
Update src/CSnakes.Runtime/Python/PyBuffer.cs
tonybaloney Sep 5, 2024
d339e8d
Update src/CSnakes.Runtime/Python/PyBuffer.cs
tonybaloney Sep 5, 2024
9dc0ddc
Update src/CSnakes.Runtime/Python/PyBuffer.cs
tonybaloney Sep 5, 2024
cb54930
Update src/CSnakes.Runtime/CPython/Buffer.cs
tonybaloney Sep 5, 2024
9e1f0c1
Update src/CSnakes.Runtime/CPython/Buffer.cs
tonybaloney Sep 5, 2024
0693a1b
Marshall format into a string
tonybaloney Sep 5, 2024
7f52a09
Add more conversions
tonybaloney Sep 5, 2024
6130c13
Using single-point precision for the float32 test
tonybaloney Sep 5, 2024
be317a8
Add byte-order and alignment accessor
tonybaloney Sep 5, 2024
9225cb7
Add the common format string types to the class
tonybaloney Sep 5, 2024
f44a778
Fix the struct sequence and add a 2D test
tonybaloney Sep 5, 2024
ff706ef
Add support for 2D vectors using the performance community extensions
tonybaloney Sep 6, 2024
31bab09
Add a test to verify that is writable
tonybaloney Sep 6, 2024
1840a78
Add some length and size checks
tonybaloney Sep 6, 2024
ec1f92f
Platform independent marshalling
tonybaloney Sep 6, 2024
d4077ab
Remove the Python 3.12 checks
tonybaloney Sep 6, 2024
8ee47fa
Add int8 int16 and readonly span
tonybaloney Sep 6, 2024
514454e
Use concrete readonly span implementations
tonybaloney Sep 6, 2024
8e4db0b
Test bytes and bytearray
tonybaloney Sep 6, 2024
deb3fde
Add a gIL to the initial buffer fetch and releases
tonybaloney Sep 6, 2024
e201c90
Explicitly request C Contiguous arrays incase someone tries to give t…
tonybaloney Sep 6, 2024
3f11ee0
Update Buffer.cs
tonybaloney Sep 6, 2024
80d9e7e
Update Buffer.cs
tonybaloney Sep 6, 2024
5d2333e
Update PyBuffer.cs
tonybaloney Sep 6, 2024
5d3d2c2
Update PyBuffer.cs
tonybaloney Sep 6, 2024
3cc7f97
Test ndarray transposition (non-contiguous array) errors, slicings an…
tonybaloney Sep 6, 2024
547db87
Merge branch 'numpy_tests' of https://github.com/tonybaloney/CSnakes …
tonybaloney Sep 6, 2024
75c0abb
Simplify types in API. Add bool, nint and unint conversions
tonybaloney Sep 16, 2024
d93295d
Apply suggestions from code review
tonybaloney Sep 16, 2024
49e06ac
Fix byte order enum parsing
tonybaloney Sep 16, 2024
c4f7948
Platform agnostic implementations
tonybaloney Sep 16, 2024
dabe33a
Add documentation
tonybaloney Sep 16, 2024
aacea48
Merge branch 'main' into numpy_tests
tonybaloney Sep 16, 2024
71d91f0
Make buffer span methods generic and move the convenience methods to …
tonybaloney Sep 17, 2024
3718b78
Merge branch 'numpy_tests' of https://github.com/tonybaloney/CSnakes …
tonybaloney Sep 17, 2024
ecf8100
Simplify names
tonybaloney Sep 17, 2024
f9f4bfa
Rename extension class
tonybaloney Sep 17, 2024
d6369b3
Tightening up the code a little
aaronpowell Sep 17, 2024
ec1d3c9
Update docs/buffers.md
tonybaloney Sep 17, 2024
d889bbb
Make a big red box with the warning about editing values
tonybaloney Sep 17, 2024
2c18f85
Merge branch 'numpy_tests' of https://github.com/tonybaloney/CSnakes …
tonybaloney Sep 17, 2024
aa110f1
Add remark about contiguous arrays
tonybaloney Sep 17, 2024
befea5c
Rebase from main
tonybaloney Sep 18, 2024
144e30c
Reference Tensor<T>
RussKie Sep 18, 2024
6bb3666
Merge pull request #213 from RussKie/numpy_tests
tonybaloney Sep 18, 2024
8b4dc70
Implement tensor spans
tonybaloney Sep 18, 2024
cdea255
Add extension methods
tonybaloney Sep 18, 2024
835ae1d
Add API type
tonybaloney Sep 18, 2024
3b652ea
Merge remote-tracking branch 'origin/main' into numpy_tests
tonybaloney Sep 18, 2024
f9ea39a
try LD_PRELOAD On tests
tonybaloney Sep 18, 2024
8471a7a
Use generic lib name
tonybaloney Sep 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add float scalars and test
  • Loading branch information
tonybaloney committed Sep 5, 2024
commit 082c9b41382d70bec5d322d6318ea477f4111826
1 change: 1 addition & 0 deletions src/CSnakes.Runtime/Python/IPyBuffer.cs
Original file line number Diff line number Diff line change
@@ -9,4 +9,5 @@ public interface IPyBuffer
Span<Int64> AsInt64Scalar();
Span<UInt32> AsUInt32Scalar();
Span<UInt64> AsUInt64Scalar();
public Span<float> AsFloatScalar();
}
6 changes: 6 additions & 0 deletions src/CSnakes.Runtime/Python/PyBuffer.cs
Original file line number Diff line number Diff line change
@@ -62,4 +62,10 @@ public unsafe Span<UInt64> AsUInt64Scalar()
EnsureFormat('Q');
return new Span<UInt64>((void*)_buffer.buf, (int)(Length / 8));
}

public unsafe Span<float> AsFloatScalar() {
EnsureScalar();
EnsureFormat('f');
return new Span<float>((void*)_buffer.buf, (int)(Length / 4));
}
}
18 changes: 18 additions & 0 deletions src/Integration.Tests/BufferTests.cs
Original file line number Diff line number Diff line change
@@ -21,4 +21,22 @@ public void TestSimpleBuffer()
Assert.Equal((int)5, result[4]);

}

[Fact]
public void TestBufferLargeFloatScalar()
{
// SKip if < Python 3.12
if (new Version(Env.Version.Split(' ')[0]) < new Version(3, 11, 0))
{
return;
}
var testModule = Env.TestBuffer();
var bufferObject = testModule.TestFloatScalar();
Assert.Equal(1532 * sizeof(float), bufferObject.Length);
Assert.True(bufferObject.Scalar);

// Check the buffer contents
Span<float> result = bufferObject.AsFloatScalar();
Assert.Equal(1532, result.Length);
}
}
5 changes: 5 additions & 0 deletions src/Integration.Tests/python/test_buffer.py
Original file line number Diff line number Diff line change
@@ -8,3 +8,8 @@

def test_simple_buffer() -> Buffer:
return np.array([1, 2, 3, 4, 5], dtype=np.int32)


def test_float_scalar() -> Buffer:
# create an array with 1532 floats between 0 and 1, similar to an embedding vector
return np.random.rand(1532).astype(np.float32)
Loading