Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C API] Move undocumented private _PyArg C API to pycore_modsupport.h internal C API #110964

Closed
vstinner opened this issue Oct 17, 2023 · 4 comments

Comments

@vstinner
Copy link
Member

vstinner commented Oct 17, 2023

If a 3rd party C extension uses one of these functions, we should consider adding a clean, documented and tested public function to replace it.

Private functions and structures:

  • _PyArg_BadArgument()
  • _PyArg_CheckPositional()
  • _PyArg_NoKeywords()
  • _PyArg_NoPositional()
  • _PyArg_ParseStack()
  • _PyArg_ParseStackAndKeywords()
  • _PyArg_Parser structure
  • _PyArg_UnpackKeywords()
  • _PyArg_UnpackKeywordsWithVararg()
  • _PyArg_UnpackStack()
  • _Py_ANY_VARARGS()

Linked PRs

@vstinner vstinner changed the title [C API] Move undocumented private _PyArg C API to the pycore_modsupport.h internal C API [C API] Move undocumented private _PyArg C API to pycore_modsupport.h internal C API Oct 17, 2023
vstinner added a commit to vstinner/cpython that referenced this issue Oct 17, 2023
Move the following private functions and structures to
pycore_modsupport.h internal C API:

* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()

Changes:

* Python/getargs.h now includes pycore_modsupport.h to export
  functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
  is used.
* Add pycore_modsupport.h includes when a C extension uses one of
  these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
  directly or indirectly (via code generated by Argument Clinic)
  pycore_modsupport.h:

  * _csv
  * _curses_panel
  * _dbm
  * _gdbm
  * _multiprocessing.posixshmem
  * _sqlite.row
  * _statistics
  * grp
  * resource
  * syslog

* _testcapi: bad_get() no longer uses METH_FASTCALL calling
  convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
  PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
  by _testcapi sub-modules which need the internal C API
  (pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
  watchers.c.
* Remove Include/cpython/modsupport.h header file.
  Include/modsupport.h no longer includes the removed header file.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 17, 2023
Move the following private functions and structures to
pycore_modsupport.h internal C API:

* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()

Changes:

* Python/getargs.h now includes pycore_modsupport.h to export
  functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
  is used.
* Add pycore_modsupport.h includes when a C extension uses one of
  these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
  directly or indirectly (via code generated by Argument Clinic)
  pycore_modsupport.h:

  * _csv
  * _curses_panel
  * _dbm
  * _gdbm
  * _multiprocessing.posixshmem
  * _sqlite.row
  * _statistics
  * grp
  * resource
  * syslog

* _testcapi: bad_get() no longer uses METH_FASTCALL calling
  convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
  PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
  by _testcapi sub-modules which need the internal C API
  (pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
  watchers.c.
* Remove Include/cpython/modsupport.h header file.
  Include/modsupport.h no longer includes the removed header file.
* Fix mypy clinic.py
@vstinner
Copy link
Member Author

Basically, these functions are only used by Argument Clinic.

_PyArg_ParseTupleAndKeywordsFast(), _PyArg_ParseStackAndKeywords(), _PyArg_UnpackKeywords() and _PyArg_UnpackKeywordsWithVararg() have a _PyArg_Parser argument which is not really used without using the internal C API anyway. AC generates complex code using internal header files:

#if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE)
#  include "pycore_gc.h"          // PyGC_Head
#  include "pycore_runtime.h"     // _Py_ID()
#endif

(...)

    #if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE)

    #define NUM_KEYWORDS 1
    static struct {
        PyGC_Head _this_is_not_used;
        PyObject_VAR_HEAD
        PyObject *ob_item[NUM_KEYWORDS];
    } _kwtuple = {
        .ob_base = PyVarObject_HEAD_INIT(&PyTuple_Type, NUM_KEYWORDS)
        .ob_item = { &_Py_ID(loop), },
    };
    #undef NUM_KEYWORDS
    #define KWTUPLE (&_kwtuple.ob_base.ob_base)

    #else  // !Py_BUILD_CORE
    #  define KWTUPLE NULL
    #endif  // !Py_BUILD_CORE

    static const char * const _keywords[] = {"loop", NULL};
    static _PyArg_Parser _parser = {
        .keywords = _keywords,
        .fname = "Future",
        .kwtuple = KWTUPLE,
    };
    #undef KWTUPLE

PyGC_Head and _Py_ID() API are part of the internal C API, and _Py_ID() must not be used outside Python code base, since we don't guarantee which identifiers are available or not.

@vstinner
Copy link
Member Author

_PyArg_ParseStack(args, nargs, format, ...) is like PyArg_ParseTuple(args, format, ...) but for METH_FASTCALL and METH_VECTORCALL calling convention. We should provide an API like _PyArg_ParseStack() in the public API, but I don't know if this API is good or ,not.

These helper functions are used by AC. I don't know if it would make sense to make them public or not. They should not be complicated to rewrite if they are no longer usable in the public C API:

  • _PyArg_BadArgument()
  • _PyArg_CheckPositional()
  • _PyArg_NoKeywords()
  • _PyArg_NoPositional()
  • _PyArg_ParseStack()
  • _PyArg_UnpackStack()

vstinner added a commit that referenced this issue Oct 17, 2023
Move the following private functions and structures to
pycore_modsupport.h internal C API:

* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()

Changes:

* Python/getargs.h now includes pycore_modsupport.h to export
  functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
  is used.
* Add pycore_modsupport.h includes when a C extension uses one of
  these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
  directly or indirectly (via code generated by Argument Clinic)
  pycore_modsupport.h:

  * _csv
  * _curses_panel
  * _dbm
  * _gdbm
  * _multiprocessing.posixshmem
  * _sqlite.row
  * _statistics
  * grp
  * resource
  * syslog

* _testcapi: bad_get() no longer uses METH_FASTCALL calling
  convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
  PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
  by _testcapi sub-modules which need the internal C API
  (pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
  watchers.c.
* Remove Include/cpython/modsupport.h header file.
  Include/modsupport.h no longer includes the removed header file.
* Fix mypy clinic.py
vstinner added a commit to vstinner/cpython that referenced this issue Oct 17, 2023
vstinner added a commit to vstinner/cpython that referenced this issue Oct 17, 2023
vstinner added a commit to vstinner/cpython that referenced this issue Oct 17, 2023
Don't rely on the global 'clinic' argument: pass explicitly a
'clinic' argument.
@encukou
Copy link
Member

encukou commented Oct 17, 2023

I don't know if this API is good or ,not.

They're vararg functions that return borrowed references, which makes them C-specific and dependent on CPython implementation details. They're quite handy in C, but IMO they'll need a good deal of design work if we're to expose them properly.

@vstinner
Copy link
Member Author

IMO they'll need a good deal of design work if we're to expose them properly.

Yeah, I agree with you.

vstinner added a commit to vstinner/cpython that referenced this issue Oct 18, 2023
Don't rely on the global 'clinic' argument: pass explicitly a
'clinic' argument.
@vstinner vstinner closed this as completed Nov 8, 2023
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
Move the following private functions and structures to
pycore_modsupport.h internal C API:

* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()

Changes:

* Python/getargs.h now includes pycore_modsupport.h to export
  functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
  is used.
* Add pycore_modsupport.h includes when a C extension uses one of
  these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
  directly or indirectly (via code generated by Argument Clinic)
  pycore_modsupport.h:

  * _csv
  * _curses_panel
  * _dbm
  * _gdbm
  * _multiprocessing.posixshmem
  * _sqlite.row
  * _statistics
  * grp
  * resource
  * syslog

* _testcapi: bad_get() no longer uses METH_FASTCALL calling
  convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
  PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
  by _testcapi sub-modules which need the internal C API
  (pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
  watchers.c.
* Remove Include/cpython/modsupport.h header file.
  Include/modsupport.h no longer includes the removed header file.
* Fix mypy clinic.py
bdegreve added a commit to cocamware/lass that referenced this issue Aug 15, 2024
It's no longer available in the public headers [1]

On Linux with GCC 11:

  error: there are no arguments to ‘_PyArg_NoKeywords’ that depend on a template parameter, so a declaration of ‘_PyArg_NoKeywords’ must be available [-fpermissive]
    226 |                 if (!_PyArg_NoKeywords("function", iKw))

[1] python/cpython#110964
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
Move the following private functions and structures to
pycore_modsupport.h internal C API:

* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()

Changes:

* Python/getargs.h now includes pycore_modsupport.h to export
  functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
  is used.
* Add pycore_modsupport.h includes when a C extension uses one of
  these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
  directly or indirectly (via code generated by Argument Clinic)
  pycore_modsupport.h:

  * _csv
  * _curses_panel
  * _dbm
  * _gdbm
  * _multiprocessing.posixshmem
  * _sqlite.row
  * _statistics
  * grp
  * resource
  * syslog

* _testcapi: bad_get() no longer uses METH_FASTCALL calling
  convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
  PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
  by _testcapi sub-modules which need the internal C API
  (pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
  watchers.c.
* Remove Include/cpython/modsupport.h header file.
  Include/modsupport.h no longer includes the removed header file.
* Fix mypy clinic.py
ludwigschwardt added a commit to ludwigschwardt/python-gnureadline that referenced this issue Oct 17, 2024
Python 3.13 moved some private C API functions to internal header files.
See python/cpython#106320 for a general discussion on this (glad to see
that gnureadline was at least listed as an affected PyPI package :-) ).

These private functions affect us:
  - _Py_SetLocaleFromEnv -> pycore_pylifecycle.h
    (see python/cpython#106400)
  - _PyArg_CheckPositional, _PyArg_BadArgument -> pycore_modsupport.h
    (see python/cpython#110964)

Since we can't include these anymore, patch the relevant declarations into
the module code. The alternative (add the internal headers to this package)
is much messier, as we would have to pull in most of those headers.
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
Add declarations for private functions from pycore_modsupport

python/cpython#110964
Copied from pycore_modsupport.h
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
Add declarations for private functions from pycore_modsupport

python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
pmbarrett314 added a commit to pmbarrett314/windows-curses that referenced this issue Nov 28, 2024
python/cpython#110964
Copied from pycore_modsupport.h

remove PY_SSIZE_T_CLEAN definition
python/cpython#104922
stephanosio pushed a commit to pmbarrett314/windows-curses that referenced this issue Dec 30, 2024
stephanosio pushed a commit to zephyrproject-rtos/windows-curses that referenced this issue Dec 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants