ENH: Format decimal.Decimal as full precision strings in .to_json(...)

[Tolker-KU]

Encodes decimal.Decimal as full precision strings in .to_json(...). The old behaviour is to convert decimal.Decimal to floats potentially loosing precision.

• closes #xxxx (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[WillAyd]

Thanks for working on this

[Tolker-KU]

Thanks for working on this

Thanks for taking your time to review this. I'm planing to open an issue discussing if this should be hidden behind a flag, such that the default behaviour remains unchanged. This draft is just a proof of concept.

The decimal module has no C API, unfortunately. I found a recent issue in the cpython repo discussing an implementation but it hasn't happened yet. As far as I recall the maintainers of cpython thought this would bloat the C API without any real gain.

I do agree on your comments and will update the patch. It should be noted that most of the implementation is copy/pasted from here. Maybe this has some bugs as-well?

pandas/pandas/_libs/src/vendored/ujson/python/objToJSON.c

Lines 352 to 374 in 7415aca

	static char *PyTimeToJSON(JSOBJ _obj, JSONTypeContext *tc, size_t *outLen) {
	PyObject *obj = (PyObject *)_obj;
	PyObject *str = PyObject_CallMethod(obj, "isoformat", NULL);
	if (str == NULL) {
	*outLen = 0;
	if (!PyErr_Occurred()) {
	PyErr_SetString(PyExc_ValueError, "Failed to convert time");
	}
	((JSONObjectEncoder *)tc->encoder)->errorMsg = "";
	return NULL;
	}
	if (PyUnicode_Check(str)) {
	PyObject *tmp = str;
	str = PyUnicode_AsUTF8String(str);
	Py_DECREF(tmp);
	}
	GET_TC(tc)->newObj = str;
	*outLen = PyBytes_GET_SIZE(str);
	char *outValue = PyBytes_AS_STRING(str);
	return outValue;
	}

[WillAyd]

It should be noted that most of the implementation is copy/pasted from here. Maybe this has some bugs as-well?

That's certainly possible. Ironically I am probably also the one responsible for that. Without looking at the blame...I'll just say the less I know, the better.

Fixing that is out of scope for this PR, but I would prefer to not copy/paste those issues over. Thanks!

[WillAyd]

@Tolker-KU generally this looks pretty good. Can you add a whatsnew note to our 3.0 release for this?

[WillAyd]

Do the test cases cover this code? If not, can you add a NaN case?

[Tolker-KU]

This test should cover it:

https://github.com/Tolker-KU/pandas/blob/d60f71fe3d20bef8eca0861e742a392dc2c35d4a/pandas/tests/io/json/test_pandas.py#L2027-L2039

https://github.com/Tolker-KU/pandas/blob/d60f71fe3d20bef8eca0861e742a392dc2c35d4a/pandas/conftest.py#L476-L481

https://github.com/Tolker-KU/pandas/blob/d60f71fe3d20bef8eca0861e742a392dc2c35d4a/pandas/_testing/__init__.py#L165

[WillAyd]

Would be ideal if our functions retuned const char* so we didn't have to cast like this. A lot of this code is old and has changed hands many times, so not surprised there are some code smells...but if you are looking to continuing contributions on this, that cleanup would be great

[WillAyd]

@mroeschke mind taking a look?

[mroeschke]

Thanks @Tolker-KU