Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cunumeric and numpy behave differently for some inplace operations and slicing #1110

Closed
CharlelieLrt opened this issue Jan 2, 2024 · 2 comments · Fixed by #1111
Closed
Assignees

Comments

@CharlelieLrt
Copy link

CharlelieLrt commented Jan 2, 2024

In the following function, cunumeric and numpy behave differently. The expected behavior is to obtain an array full of 1, which numpy does:

import cunumeric as num
import numpy as np


def interpolate(data):
    m = (slice(0, -1),)
    p = (slice(1, None),)
    data[p] += data[m] # The issue is caused by this line
    data[p] *= 0.5
    return data[p]


# Cunumeric
data_num = num.ones((20000,))
r_num = interpolate(data_num)
print((r_num == 1.).all()) # False: wrong behavior

# Numpy
data_np = np.ones((20000,))
r_np = interpolate(data_np)
print((r_np == 1.).all()) # True: expected behavior

I have observed 3 very different behavior of cunumeric depending on the size of the array:

  1. For small arrays (e.g. shape (8,)) cunumeric returns the expected result
  2. For large arrays (e.g. shape (20000,)) cunumeric does nor raise any error, but the result array is not full of 1, which is not the expected numpy behavior.
  3. For arrays of intermediate sizes (e.g. shape (10000,))) I have a Legion error because of interfering tasks:
[0 - 17246b000]    0.113712 {5}{runtime}: [error 356] LEGION ERROR: Aliased and interfering region requirements for individual tasks are not permitted. Region requirements 0 and 1 of task cunumeric::BinaryOpTask (UID 22) in parent task legion_python_top_level_task (UID 1) are interfering. (from file /Users/charlelie/Documents/RESEARCH/CODES/LEGATE/legate.core/_skbuild/macosx-12.0-arm64-3.10/cmake-build/_deps/legion-src/runtime/legion/legion_tasks.cc:6146)
For more information see:
http://legion.stanford.edu/messages/error_code.html#error_code_356

Signal 6 received by node 0, process 20170 (thread 17246b000) - obtaining backtrace
Signal 6 received by process 20170 (thread 17246b000) at: stack trace: 17 frames
  [0] = 0   libsystem_pthread.dylib             0x00000001a5bfbedf pthread_kill + 287
  [1] = 0   libsystem_c.dylib                   0x00000001a5b3633f abort + 167
  [2] = 0   liblegion.1.dylib                   0x0000000142450807 _ZN6Legion8Internal7Runtime20report_error_messageEiPKciS3_ + 51
  [3] = 0   liblegion.1.dylib                   0x00000001423177f7 _ZThn480_N6Legion8Internal14IndividualTask31report_interfering_requirementsEjj + 223
  [4] = 0   liblegion.1.dylib                   0x0000000142303f3b _ZN6Legion8Internal6TaskOp33perform_intra_task_alias_analysisEbPNS0_11LegionTraceERNSt3__16vectorINS0_14RegionTreePathENS4_9allocatorIS6_EEEE + 2083
  [5] = 0   liblegion.1.dylib                   0x000000014231708f _ZN6Legion8Internal14IndividualTask25trigger_prepipeline_stageEv + 603
  [6] = 0   liblegion.1.dylib                   0x00000001422236e3 _ZN6Legion8Internal9Operation25execute_prepipeline_stageEjb + 407
  [7] = 0   liblegion.1.dylib                   0x0000000142223a47 _ZN6Legion8Internal9Operation27execute_dependence_analysisEv + 51
  [8] = 0   liblegion.1.dylib                   0x000000014215c28f _ZN6Legion8Internal12InnerContext24process_dependence_stageEv + 771
  [9] = 0   liblegion.1.dylib                   0x00000001424f068b _ZN6Legion8Internal7Runtime19legion_runtime_taskEPKvmS3_mN5Realm9ProcessorE + 263
  [10] = 0   librealm.1.dylib                    0x000000014579b07b _ZN5Realm18LocalTaskProcessor12execute_taskEjRKNS_12ByteArrayRefE + 1119
  [11] = 0   librealm.1.dylib                    0x00000001457d802b _ZN5Realm4Task20execute_on_processorENS_9ProcessorE + 471
  [12] = 0   librealm.1.dylib                    0x00000001457deb23 _ZN5Realm25KernelThreadTaskScheduler12execute_taskEPNS_4TaskE + 23
  [13] = 0   librealm.1.dylib                    0x00000001457dd0ff _ZN5Realm21ThreadedTaskScheduler14scheduler_loopEv + 2095
  [14] = 0   librealm.1.dylib                    0x00000001457dd5a7 _ZN5Realm21ThreadedTaskScheduler20scheduler_loop_wlockEv + 39
  [15] = 0   librealm.1.dylib                    0x00000001457e4a0b _ZN5Realm12KernelThread13pthread_entryEPv + 499
  [16] = 0   libsystem_pthread.dylib             0x00000001a5bfc26b _pthread_start + 147

I think this was also reported in #57 .

@CharlelieLrt
Copy link
Author

On the other hand, I am using other in-place operators with overlap between the input and output region, without noticing any issue. For example:

num.einsum("ij...,ij...->i...", x, y, out=x[:, 0, ...])

So, I am wondering if such in-place operation only works for some operators and the bug only occurs for some others.

@manopapad
Copy link
Contributor

It looks like in-place arithmetic operations were missing an overlap check. Should be fixed with #1111

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants