Canonical format for stringifying JSON payload

[BLR-0118]

This was raised in the Community Call today by some participants about defining a canonical format for stringifying JSON payload for signing & verification. Is this really required? Irrespective of how the JSON payload is stringified, if the stringified JSON payload is signed and also used as the raw payload to verify the signature, there should not be any auth failures.
@1upkd @yaikhomba @mherle @NajeebMohammed @navdeep710 @kishorgandham @nitinmish

[mherle]

ensuring that all libraries across all frameworks produce the correct-reproducible way of serialization will be very hard and will end up with library/framework specific + version specific behaviour issues to debug.
Raw payload verification is exactly what needs to happen as part of signing and verification. Even if a sender has sent an incorrect syntax of JSON, verifying that the data has come non-tampered has to happen.

It might be a good idea for having a common tech resources where code can be shared by NPs for helping out others for popular language frameworks (such as: python-flask, java-springboot)

[kishorgandham]

I agree with @mherle

[kdqed]

@BLR-0118 I understand your concerns, and your approach will work if all the network participants send the exact stringified json used for signing with each request. Is it possible that someone could use different formats in the sent request body and calculating signature? That is the only possibility for issues I can see with your prescribed approach.

[BLR-0118]

Agree with @mherle. @1upkd - what's the issue you were mentioning today?

[kdqed]

@BLR-0118
The issue I was facing is this:

• When I was sending the payload to the gateway the signature mismatch was happening unless I included spaces in the JSON string.
• When verifying requests from seller apps, some were getting verified only when I remove spaces.
• In the call yesterday, I was suggesting we standardize the format for converting to JSON
• Based upon your suggestion above, I now used the raw request body sent by the seller apps to verify the signature. This is working perfectly fine with all the seller apps I am getting callbacks from.

Finally, my questions are these:

• Can I return 401 if this verification with the raw request body does not work?
• Can this be documented and all NPs be expected to sign auth header with the exact request body string they are sending? Because it is possible the signing is done with JSON Object by one library, and the request body was converted by another library which stringifies it differently. This can be a common source of confusion and error.

[mherle]

@1upkd
[1] You should send 401 if auth check on raw string/request body fails
[2] I agree this should be documented with special mention of "verify with raw payload". Have seen this confusion come up multiple times and this could potentially come with each new participant. Maybe, we should even have a special warning for doing verifying via raw -> json -> raw and then checking on that.
PS: Signing is always done on a byte stream in every library. Its just that the JSON object -> byte stream conversion can happen with different byte streams (but still be "correct") but signing output differs with single character/byte differing.

[kdqed]

Thanks @mherle. This clarifies most of my doubts now.

[BLR-0118]

@yaikhomba - what's the issue you're facing?

[yaikhomba]

@BLR-0118 - the issue I am facing is same as what @1upkd has described. Verification is failing for some retail sellers & logistic providers and the associated gateway signatures. Problem is due to the way different languages / frameworks stringify JSON. For some I am having to remove spaces and for some add spaces. There is still one where it is still failing.

[BLR-0118]

@yaikhomba - why does it matter to you how it's been stringified? If the stringified JSON has been signed (as explained in this thread) and the same stringified JSON is accessible through the raw payload to verify the signature, you'll be able to complete the verification.

[yaikhomba]

@BLR-0118 - yes you are right. I have been overlooking this. 🤦‍♂️ Have fixed it now.
We should highlight this in the documentation. Thanks

[nitinmish]

Standardizing a JSON format and serialization imperatives will be very difficult to implement as each language has its own way of formatting JSON data. Idea is to calculate hash of the request/response body before overlaying it on HTTP and then sign it. Whole HTTP request will be one monolithic combination of HTTP body and HTTP header that will ensure standardized implementation of Signing algorithm. This boils down to two fundamental principles of Encrypting the hash:

1. Creation of a HASH from UTF-8 encoded JSON byte array (Since some special characters may end up printing Unicode literals on consumption side)
2. Encapsulating the same byte array as HTTP body object which is consumed on consuming application and validated before deserializing it into JSON parsable object.

@mherle @BLR-0118 @yaikhomba @1upkd