C扩展Python

使用C扩展Python

1. PyErr_*()函数是将一个异常对象压入到Python解释器的异常栈中
2. PyErr_Clear()函数是将Python异常栈中栈顶的元素弹出, 调用这个函数通常就相当于在Python程序中的try: except: 语句中except的作用
3. Python中的API, 每一个模块对应的API是PyMODNAME_FUNCINMOD()
4. Python扩展模块的函数模板:

 // First static PyObject *exfunc(PyObject *self, PyObject *args) { ... // Get parameters /* Return 0 means an exception has been raised, you need to stop your program and rreturn NULL According to the official: It returns NULL (the error indicator for functions returning object pointers) if an error is detected in the argument list */ if (!PyArgs_ParseTuple(args, "format_string", &the_addr_of_c_variables)) { return NULL; } /* In this area, if you fail to call an user-defined function, what you are expected to do is call Python Exception API such as PyErr_SetString(PyExc_Type, "Exc_Info") to put an Exception Object in the Exception Stack in Python. */ ... } // Second // This is `method table` static PyMethodDef exfunc[] = { /* {NULL, NULL, 0, NULL} is the symbol of the end of PyMethodDef array, Python Interpretor will stop walking this PyMethodDef array when meeting it. */ /* According to the official: When using only METH_VARARGS, the function should expect the Python-level parameters to be passed in as a tuple acceptable for parsing via PyArg_ParseTuple() */ {"MethodName", your_func_pointer, METH_VARGS or METH_VARGS | METH_KEYWORDS, "Documents here"}, {NULL, NULL, 0, NULL} }; // Third static struct PyModuleDef exmodule = { PyMODULEDEF_HEAD_INIT "name", "doc", size, exfunc, }; // Fourth /* Actually, what Python Interpretor will call is the PyInit_*() function, so that will be entry to your extensional C module */ PyMODINIT_FUNC PyInit_exmodule { // PyModule_Init("ModuleName", &module_def); PyObject *m; /* When create an module object, an array of methods is need, but this rest is the history, there are classes, exceptions and so on existing in a module, so if you want to add something into a module, just use PyModule_AddObject() API like `PyErr_NewException("spam.error", NULL, NULL);` this will create a base exception implementing Exception */ m = PyModule_Create(&the_addr_of_PyModuleDef_array); if (!m) { return NULL; } } /*******************************************************************************************************/ // In Python 2.x, the third and fourth are different of Python3.x // Third PyMODINIT_FUNC initname { // Don't return, because PyMODINIT_FUNC is void which is PyObject * in Python 3.x // Use PyInit_Module("name", MethodDef); rather than PyModule_Create("name", ModuleDef) PyInit_Module("name", MethodDef); } 

1. 最好不直接使用malloc()函数, 如果该函数出现了错误, 应该调用PyErr_NoMemory()函数, 调用Python的API如PyLong_FromLong可以不考虑这个问题, 因为他底层已经封装了该异常
2. Python大多数函数, 以正数或者0表示成功, 以NULL或者-1表示失败, 官方手册中

PyArg_ParseTuple() and friends, functions that return an integer status usually return a positive value or zero for success and -1 for failure

1. 如果扩展模块中的函数不希望有返回值的话, 返回Py_None对象

Py_INCREF(Py_None); return Py_None;

1. 为NULL返回值的函数在Python中视为发生了异常的函数
2. 除了Python解释器要调用的PyMODINIT_FUNC PyInit_name()没有static修饰, 其他所有的函数和变量都是要statis修饰的
3. PyMODINIT_FUNC PyInit_name()函数的名字必须是PyInit_开头的, 并且返回模块对象, 这样在Python中sys.modules序列才能捕捉到该引用, 将其添加到sys.modules这个全局的模块池中
4. 当在Python中使用import extend_modules时, Python解释器就会调用C层面上的PyInit_extend_modules()函数, 在该函数中执行PyModule_Create创建模块, 并且加载module table中定义的函数集合
5. 可是这样说, Python中那么多的built in模块都是Guido按照自己规定的API扩展出来的, 所以我们编写的扩展模块也可以称之为built in模块
6. Python中的heapq, functools等内置模块的源码在Modules/xxmoudle.c
7. PyCallable_Check()判断是否一个PyObject *对象可以调用
8. PyObject_CallObject(my_callback, arglist); 第一个参数是PyObject *, 第二个参数就是调用该PyObject *对象需要的参数, 也为一个PyObject *(是一个Tuple对象), 这是没有关键词参数的情况; 使用PyObject_Call(my_callback, argslist, argkw); 可以有位置参数和关键词参数
9. 使用Py_BuildValue()函数创建对象是官方建议的
10. 每次调用一个Python API, 需要判断其返回值, 判断是否会发生异常, 所有C编写扩展模块会比较冗长
11. 参数就是一个PyTupleObject, 用户创建一个参数列表:

PyObject *arglist; arglist = P_BuildValue("(i, l)", 8, 8888888)

1. 慎用Py_INCREASE, Py_DECREASE, Py_XINCREASE, Py_XDREASE
2. 我们自己创建的参数列表, 在放入到一个函数中执行完之后, 使用Py_DREASE宏减少参数列表的引用计数, 说白了就是让这个Tuple消失, 因为已经没有用了

int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict, const char *format, char *kwlist[], ...); arg: tuple对象 kwdict: dict对象 format: 字符串格式 kwlist: 存放关键词参数的键, 最后为NULL表示结束 NOTE!! 此函数只能借此keyword而不能解析位置参数了

1. PyBuild_Value()函数

Py_BuildValue("") None Py_BuildValue("i", 123) 123 Py_BuildValue("iii", 123, 456, 789) (123, 456, 789) Py_BuildValue("s", "hello") 'hello' Py_BuildValue("y", "hello") b'hello' Py_BuildValue("ss", "hello", "world") ('hello', 'world') Py_BuildValue("s#", "hello", 4) 'hell' Py_BuildValue("y#", "hello", 4) b'hell' Py_BuildValue("()") () Py_BuildValue("(i)", 123) (123,) Py_BuildValue("(ii)", 123, 456) (123, 456) Py_BuildValue("(i,i)", 123, 456) (123, 456) Py_BuildValue("[i,i]", 123, 456) [123, 456] Py_BuildValue("{s:i,s:i}", "abc", 123, "def", 456) {'abc': 123, 'def': 456} Py_BuildValue("((ii)(ii)) (ii)", 1, 2, 3, 4, 5, 6) (((1, 2), (3, 4)), (5, 6))

1. 我们知道Python知道用特定的Py_Init
2. 如果是借来的引用的话, 是不会增加该对象的引用计数的, 但是在借来的基础上, 使用Py_INCREASE增加它的引用计数, 可以把借来的转为自己持有的
3. 几乎所有的Python创建PyObject对象的API都会返回一个拥有的引用, 同时大多数的从一个对象中取出另一个对象的函数API也都是拥有的引用, 例外的PyTuple_GetItem(), PyList_GetItem(), PyDict_GetItem(), PyImport_AddModule() and PyDict_GetItemString()等函数则是返回借来的对象
4. 传入参数到Python的API中, 大多数的参数对象都是不会增加引用的, 但是PyTuple_SetItem() and PyList_SetItem()是例外
5. 对24和25的补充, 使用借来的引用, 它的存活周期是一个PythonCFunction, 如果需要对象存起来的话, 就会调动Py_INCREASE增加引用计数, 但是如果要返回一个Python对象的话, 需要增加其引用计数
6. 一般PythonAPI不怎么检验Python对象为NULL的情况
7. 模块的一些初始化操作在PyInit_
8. 如果希望添加新的内置类型, 这个我最熟悉(那是多么痛的领悟), 因为我写过, 定义一个type类型结构体, 第一个是PyObject_HEAD, 接着在初始化type(一定要有.tp_flags = Py_TPFLAGS_DEFAULT,), 在PyInit_
9. 在扩展模块中设置的名称为modulename.subname

This initializes the Custom type, filling in a number of members to the appropriate default values, including ob_type that we initially set to NULL. if (PyType_Ready(&CustomType) < 0) return; This adds the type to the module dictionary. PyModule_AddObject(m, "Custom", (PyObject *) &CustomType); 

1. 一个完整的type赋值, 根据1, 2, 3得知, 为什么每次创建一个类型都会有new, init, del, 因为在其对应的PyTypeObject中需要指明, 并且不是通过PyMethodDef表控制的

static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom2.Custom", .tp_doc = "Custom objects", .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, .tp_new = Custom_new, // __new__ 1 .tp_init = (initproc) Custom_init, // __init__ 2 .tp_dealloc = (destructor) Custom_dealloc, // __del__ 3 .tp_traverse = (traverseproc) Custom_traverse, // GC will call it .tp_clear = (inquiry) Custom_clear, // GC will call it .tp_members = Custom_members, // Is the PyMemberDef table .tp_methods = Custom_methods, // Is the PyMethodDef table .tp_getset = Custom_getsetters, // Setter and getter methods are special, so part them with tp_methods }; // About the traverse and clear static int Custom_traverse(CustomObject *self, visitproc visit, void *arg) { Py_VISIT(self->first); Py_VISIT(self->last); return 0; } static int Custom_clear(CustomObject *self) { Py_CLEAR(self->first); Py_CLEAR(self->last); return 0; }

1. 如何要继承一个类型的话, 那么定义的类型的第一行就是那个类型对象而不是HEAD了, 那个类型对象中已经包含了HEAD了, 所有新建的对象就可以转换成新建的这个对象类型或者他的父类型, 同时在Init函数中需要指定该类型对象的tp_base是谁, 这样在Python层面上实现集成

Demo file, according to the official

#include <Python.h> typedef struct { PyListObject list; int state; } SubListObject; static PyObject * SubList_increment(SubListObject *self, PyObject *unused) { self->state++; return PyLong_FromLong(self->state); } static PyMethodDef SubList_methods[] = { {"increment", (PyCFunction) SubList_increment, METH_NOARGS, PyDoc_STR("increment state counter")}, {NULL}, }; static int SubList_init(SubListObject *self, PyObject *args, PyObject *kwds) { if (PyList_Type.tp_init((PyObject *) self, args, kwds) < 0) return -1; self->state = 0; return 0; } static PyTypeObject SubListType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "sublist.SubList", .tp_doc = "SubList objects", .tp_basicsize = sizeof(SubListObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, .tp_init = (initproc) SubList_init, .tp_methods = SubList_methods, }; static PyModuleDef sublistmodule = { PyModuleDef_HEAD_INIT, .m_name = "sublist", .m_doc = "Example module that creates an extension type.", .m_size = -1, }; PyMODINIT_FUNC PyInit_sublist(void) { PyObject *m; SubListType.tp_base = &PyList_Type; if (PyType_Ready(&SubListType) < 0) return NULL; m = PyModule_Create(&sublistmodule); if (m == NULL) return NULL; Py_INCREF(&SubListType); PyModule_AddObject(m, "SubList", (PyObject *) &SubListType); return m; } 

Another Demo

#include <Python.h> #include "structmember.h" typedef struct { PyObject_HEAD PyObject *first; /* first name */ PyObject *last; /* last name */ int number; } CustomObject; static void Custom_dealloc(CustomObject *self) { Py_XDECREF(self->first); Py_XDECREF(self->last); Py_TYPE(self)->tp_free((PyObject *) self); } static PyObject * Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { CustomObject *self; self = (CustomObject *) type->tp_alloc(type, 0); if (self != NULL) { self->first = PyUnicode_FromString(""); if (self->first == NULL) { Py_DECREF(self); return NULL; } self->last = PyUnicode_FromString(""); if (self->last == NULL) { Py_DECREF(self); return NULL; } self->number = 0; } return (PyObject *) self; } static int Custom_init(CustomObject *self, PyObject *args, PyObject *kwds) { static char *kwlist[] = {"first", "last", "number", NULL}; PyObject *first = NULL, *last = NULL, *tmp; if (!PyArg_ParseTupleAndKeywords(args, kwds, "|UUi", kwlist, &first, &last, &self->number)) return -1; if (first) { tmp = self->first; Py_INCREF(first); self->first = first; Py_DECREF(tmp); } if (last) { tmp = self->last; Py_INCREF(last); self->last = last; Py_DECREF(tmp); } return 0; } static PyMemberDef Custom_members[] = { {"number", T_INT, offsetof(CustomObject, number), 0, "custom number"}, {NULL} /* Sentinel */ }; static PyObject * Custom_getfirst(CustomObject *self, void *closure) { Py_INCREF(self->first); return self->first; } static int Custom_setfirst(CustomObject *self, PyObject *value, void *closure) { PyObject *tmp; if (value == NULL) { PyErr_SetString(PyExc_TypeError, "Cannot delete the first attribute"); return -1; } if (!PyUnicode_Check(value)) { PyErr_SetString(PyExc_TypeError, "The first attribute value must be a string"); return -1; } tmp = self->first; Py_INCREF(value); self->first = value; Py_DECREF(tmp); return 0; } static PyObject * Custom_getlast(CustomObject *self, void *closure) { Py_INCREF(self->last); return self->last; } static int Custom_setlast(CustomObject *self, PyObject *value, void *closure) { PyObject *tmp; if (value == NULL) { PyErr_SetString(PyExc_TypeError, "Cannot delete the last attribute"); return -1; } if (!PyUnicode_Check(value)) { PyErr_SetString(PyExc_TypeError, "The last attribute value must be a string"); return -1; } tmp = self->last; Py_INCREF(value); self->last = value; Py_DECREF(tmp); return 0; } static PyGetSetDef Custom_getsetters[] = { {"first", (getter) Custom_getfirst, (setter) Custom_setfirst, "first name", NULL}, {"last", (getter) Custom_getlast, (setter) Custom_setlast, "last name", NULL}, {NULL} /* Sentinel */ }; static PyObject * Custom_name(CustomObject *self, PyObject *Py_UNUSED(ignored)) { return PyUnicode_FromFormat("%S %S", self->first, self->last); } static PyMethodDef Custom_methods[] = { {"name", (PyCFunction) Custom_name, METH_NOARGS, "Return the name, combining the first and last name" }, {NULL} /* Sentinel */ }; static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom3.Custom", .tp_doc = "Custom objects", .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, .tp_new = Custom_new, .tp_init = (initproc) Custom_init, .tp_dealloc = (destructor) Custom_dealloc, .tp_members = Custom_members, .tp_methods = Custom_methods, .tp_getset = Custom_getsetters, }; static PyModuleDef custommodule = { PyModuleDef_HEAD_INIT, .m_name = "custom3", .m_doc = "Example module that creates an extension type.", .m_size = -1, }; PyMODINIT_FUNC PyInit_custom3(void) { PyObject *m; if (PyType_Ready(&CustomType) < 0) return NULL; m = PyModule_Create(&custommodule); if (m == NULL) return NULL; Py_INCREF(&CustomType); PyModule_AddObject(m, "Custom", (PyObject *) &CustomType); return m; }