Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RMACXX_Subarray_t not being obeyed #14

Open
MMukundi opened this issue Sep 29, 2022 · 2 comments
Open

RMACXX_Subarray_t not being obeyed #14

MMukundi opened this issue Sep 29, 2022 · 2 comments

Comments

@MMukundi
Copy link
Collaborator

MMukundi commented Sep 29, 2022

Reproduction:

int main(){
    // -- SNIP --
    win.fill(1);

    // create a vector, initially 0s
    std::vector<int> a(10); 
    int *abuf = a.data();
    
    // create subarray type
    // Note: Output is unchanged by changes to these values
    rmacxx::RMACXX_Subarray_t<int> foo({1},{2},{10}); 
    
    // put
    win(1,{0},{9}) << foo(abuf);
    win.flush(1);
    std::vector<int> results(10); 
    win(1,{0},{9}) >> results.data();
    win.flush(1);
    // -- SNIP --
}

Expected behavior:

a = { 0, 0, 1, 1, 1, 1, 1, 1, 1, 1};

Observed behavior:

// Note: Output is unchanged by values of subarray_t
a = { 0, 1, 1, 1, 1, 1, 1, 1, 1, 1};

Suspected cause:
include/rmacxx-mpi-rma.hpp

       #define RMACXX_BULK_PUT_NC(origin)\
194|        do{\
...|            /* -- SNIP */\
222|            MPI_Accumulate(origin.ptr_, 1, origin.dtype_, target_, \
223|              /*disp*/ 0, 1, sarr_type, MPI_REPLACE, win_); \
...|            /* -- SNIP */\
265|        }while(0)

No members of the RMACXX_Subarray_t origin, not even the MPI_Subarray_t that was already created, are included in the actual MPI_Accumulate call, likely meaning that all calls are relying on the exact same implementation, regardless of user inputs

@sg0
Copy link
Owner

sg0 commented Sep 29, 2022

Note, like my previous comment win(1,{0},{9}) << foo(abuf); is invoked by as many processes as you are running the program with. So, flushing only for process #1 (which translates to: "complete all outstanding RMA operations from my process to process #1") is erroneous before accessing the window, because the other process is also invoking the same statement and may not be done (in this case, flush is a safe option, which translates to flush_all, meaning: "complete all outstanding RMA operations from my process to all other processes").

@sg0
Copy link
Owner

sg0 commented Oct 9, 2022

Any comments? Is this a bug or a misunderstanding?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants