Error: 2 UNKNOWN: Transaction outcome unknown

[walshie4]

Environment details

• OS: CoreOS
• Node.js version: v8.9.4
• npm version: v6.0.1
• @google-cloud/spanner version: v1.4.1

Steps to reproduce

1. ???
2. ???

Not sure what causes this but these errors started trickling in this morning. There's been about ~75 throughout the course of the day. Mainly wondering what would cause this sort of error and how I can go about debugging it.

Error info:

 meta: {
   code: 2    
   details: "Transaction outcome unknown."    
   message: "2 UNKNOWN: Transaction outcome unknown."    
   metadata: {
      _internal_repr: {
    }
 }
 note: "Exception occurred in retry method that was not classified as transient"    
 stack: [
    0: "Error: 2 UNKNOWN: Transaction outcome unknown."     
    1: "at Object.exports.createStatusError (/opt/app/node_modules/google-gax/node_modules/grpc/src/common.js:87:15)"     
    2: "at Object.onReceiveStatus (/opt/app/node_modules/google-gax/node_modules/grpc/src/client_interceptors.js:1214:28)"     
    3: "at InterceptingListener._callNext (/opt/app/node_modules/google-gax/node_modules/grpc/src/client_interceptors.js:590:42)"     
    4: "at InterceptingListener.onReceiveStatus (/opt/app/node_modules/google-gax/node_modules/grpc/src/client_interceptors.js:640:8)"     
    5: "at callback (/opt/app/node_modules/google-gax/node_modules/grpc/src/client_interceptors.js:867:24)"     
    6: ""     
  ]
  statusCode: 500    
} 

[walshie4]

FWIW this error occurs in the same location as we currently get a bunch of errors like so:

Server: 6 ALREADY_EXISTS: Unique index violation on index <index name>

(although this is expected for the values reported later in the message, just thought it could be relevant)

[walshie4]

Another thing to note is that the requests which fail with this error all fail in just over 5 seconds. Could be a coincidence but they're all within 5020ms to 5275ms

[walshie4]

Here is the code which is causing this error (but succeeds usually):

// spanner is a Spanner database object ready to be used for operations
// tableName is a String of the desired table's name
// rowData is an object of data to write
export default (spanner, tableName, rowData) => {
  return new Promise((resolve, reject) => {
    // eslint-disable-next-line consistent-return
    spanner.runTransaction((errTrx, dbTrx) => {
      if (errTrx) {
        honeyLogger.error('V3 encountered error inserting', errTrx);
        return reject(errTrx);
      }

      const addedInfo = { storeShardId: getSpannerShardId(rowData.storeId) };
      const insertColumns = Object.assign({}, addedInfo, rowData);

      dbTrx.insert(tableName, insertColumns);
      dbTrx.commit((err) => {
        if (err) {
          dbTrx.end();
          return reject(err);
        }
        return resolve();
      });
    });
  })
  .then(() => rowData);
};

This code was written before we got further along with spanner so the approach is a bit odd but does work for nearly all inserts.

[crwilcox]

Just noting this is on the radar. We are currently handling some of the other issues which were prioritized over this.

[snehashah16]

This bug is being tracked in the backend, and in the interim the recommendation is to ignore the error "outcome unknown" and retry the request.

[JustinBeckwith]

@snehashah16 do you have a bug on another public issue tracker the customer can follow? If there is no action item for this library, I'd prefer to not track it here :)

[crwilcox]

@snehashah16, just following up on this again, is there an ETA for a fix on the backend of this? Should we harden retry on the client anyway?

[crwilcox]

@JustinBeckwith adding a link to the back end issue: https://issuetracker.google.com/issues/90028517

[crwilcox]

Just spoke with a member of the Spanner team and they advised modifying the client to handle this particular error by retrying. The backend will be updated as well but they still would like the client to understand how to retry these types of failures.

[crwilcox]

Fix released in https://www.npmjs.com/package/@google-cloud/spanner/v/1.4.3

[walshie4]

Still seeing this issue occasionally on v1.4.3, although they're now reported in a slightly different form with error code 4 and the message Deadline for Transaction exceeded..

[JustinBeckwith]

@walshie4 Apologies. Thanks for reporting. Would you mind opening a new issue with the call stack? Also priority wise, is this more important than the other issues?

[walshie4]

@JustinBeckwith Sure thing. If the other issue you're referring to is issue #183, that is more important.