Admin Control Panel v2.0

✎ Editing: cporting.html ← Back

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
 
 <title>Porting Extension Modules to Python 3 &mdash; Python 2.7.5 documentation</title>
 
 <link rel="stylesheet" href="../_static/default.css" type="text/css" />
 <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
 
 <script type="text/javascript">
 var DOCUMENTATION_OPTIONS = {
 URL_ROOT: '../',
 VERSION: '2.7.5',
 COLLAPSE_INDEX: false,
 FILE_SUFFIX: '.html',
 HAS_SOURCE: true
 };
 </script>
 <script type="text/javascript" src="../_static/jquery.js"></script>
 <script type="text/javascript" src="../_static/underscore.js"></script>
 <script type="text/javascript" src="../_static/doctools.js"></script>
 <script type="text/javascript" src="../_static/sidebar.js"></script>
 <link rel="search" type="application/opensearchdescription+xml"
 title="Search within Python 2.7.5 documentation"
 href="../_static/opensearch.xml"/>
 <link rel="author" title="About these documents" href="../about.html" />
 <link rel="copyright" title="Copyright" href="../copyright.html" />
 <link rel="top" title="Python 2.7.5 documentation" href="../index.html" />
 <link rel="up" title="Python HOWTOs" href="index.html" />
 <link rel="next" title="Curses Programming with Python" href="curses.html" />
 <link rel="prev" title="Porting Python 2 Code to Python 3" href="pyporting.html" />
 <link rel="shortcut icon" type="image/png" href="../_static/py.png" />
 <script type="text/javascript" src="../_static/copybutton.js"></script>

</head>
 <body>
 <div class="related">
 <h3>Navigation</h3>
 <ul>
 <li class="right" style="margin-right: 10px">
 <a href="../genindex.html" title="General Index"
 accesskey="I">index</a></li>
 <li class="right" >
 <a href="../py-modindex.html" title="Python Module Index"
 >modules</a> |</li>
 <li class="right" >
 <a href="curses.html" title="Curses Programming with Python"
 accesskey="N">next</a> |</li>
 <li class="right" >
 <a href="pyporting.html" title="Porting Python 2 Code to Python 3"
 accesskey="P">previous</a> |</li>
 <li><img src="../_static/py.png" alt=""
 style="vertical-align: middle; margin-top: -1px"/></li>
 <li><a href="http://www.python.org/">Python</a> &raquo;</li>
 <li>
 <a href="../index.html">Python 2.7.5 documentation</a> &raquo;
 </li>

<li><a href="index.html" accesskey="U">Python HOWTOs</a> &raquo;</li> 
 </ul>
 </div>

<div class="document">
 <div class="documentwrapper">
 <div class="bodywrapper">
 <div class="body">
 
 <div class="section" id="porting-extension-modules-to-python-3">
<h1>Porting Extension Modules to Python 3<a class="headerlink" href="#porting-extension-modules-to-python-3" title="Permalink to this headline">¶</a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">author:</th><td class="field-body">Benjamin Peterson</td>
</tr>
</tbody>
</table>
<div class="topic">
Abstract
Although changing the C-API was not one of Python 3&#8217;s objectives,
the many Python-level changes made leaving Python 2&#8217;s API intact
impossible. In fact, some changes such as <a class="reference internal" href="../library/functions.html#int" title="int"><tt class="xref py py-func docutils literal">int()</tt></a> and
<a class="reference internal" href="../library/functions.html#long" title="long"><tt class="xref py py-func docutils literal">long()</tt></a> unification are more obvious on the C level. This
document endeavors to document incompatibilities and how they can
be worked around.
</div>
<div class="section" id="conditional-compilation">
<h2>Conditional compilation<a class="headerlink" href="#conditional-compilation" title="Permalink to this headline">¶</a></h2>
The easiest way to compile only some code for Python 3 is to check
if <tt class="xref c c-macro docutils literal">PY_MAJOR_VERSION</tt> is greater than or equal to 3.
<div class="highlight-c"><div class="highlight"><pre>#if PY_MAJOR_VERSION &gt;= 3
#define IS_PY3K
#endif
</pre></div>
</div>
API functions that are not present can be aliased to their equivalents within
conditional blocks.
</div>
<div class="section" id="changes-to-object-apis">
<h2>Changes to Object APIs<a class="headerlink" href="#changes-to-object-apis" title="Permalink to this headline">¶</a></h2>
Python 3 merged together some types with similar functions while cleanly
separating others.
<div class="section" id="str-unicode-unification">
<h3>str/unicode Unification<a class="headerlink" href="#str-unicode-unification" title="Permalink to this headline">¶</a></h3>
Python 3&#8217;s <a class="reference internal" href="../library/functions.html#str" title="str"><tt class="xref py py-func docutils literal">str()</tt></a> (<tt class="docutils literal">PyString_*</tt> functions in C) type is equivalent to
Python 2&#8217;s <a class="reference internal" href="../library/functions.html#unicode" title="unicode"><tt class="xref py py-func docutils literal">unicode()</tt></a> (<tt class="docutils literal">PyUnicode_*</tt>). The old 8-bit string type has
become <tt class="xref py py-func docutils literal">bytes()</tt>. Python 2.6 and later provide a compatibility header,
<tt class="file docutils literal">bytesobject.h</tt>, mapping <tt class="docutils literal">PyBytes</tt> names to <tt class="docutils literal">PyString</tt> ones. For best
compatibility with Python 3, <tt class="xref c c-type docutils literal">PyUnicode</tt> should be used for textual data and
<tt class="xref c c-type docutils literal">PyBytes</tt> for binary data. It&#8217;s also important to remember that
<tt class="xref c c-type docutils literal">PyBytes</tt> and <tt class="xref c c-type docutils literal">PyUnicode</tt> in Python 3 are not interchangeable like
<tt class="xref c c-type docutils literal">PyString</tt> and <tt class="xref c c-type docutils literal">PyUnicode</tt> are in Python 2. The following example
shows best practices with regards to <tt class="xref c c-type docutils literal">PyUnicode</tt>, <tt class="xref c c-type docutils literal">PyString</tt>,
and <tt class="xref c c-type docutils literal">PyBytes</tt>.
<div class="highlight-c"><div class="highlight"><pre>#include &quot;stdlib.h&quot;
#include &quot;Python.h&quot;
#include &quot;bytesobject.h&quot;

/* text example */
static PyObject *
say_hello(PyObject *self, PyObject *args) {
 PyObject *name, *result;

if (!PyArg_ParseTuple(args, &quot;U:say_hello&quot;, &amp;name))
 return NULL;

result = PyUnicode_FromFormat(&quot;Hello, %S!&quot;, name);
 return result;
}

/* just a forward */
static char * do_encode(PyObject *);

/* bytes example */
static PyObject *
encode_object(PyObject *self, PyObject *args) {
 char *encoded;
 PyObject *result, *myobj;

if (!PyArg_ParseTuple(args, &quot;O:encode_object&quot;, &amp;myobj))
 return NULL;

encoded = do_encode(myobj);
 if (encoded == NULL)
 return NULL;
 result = PyBytes_FromString(encoded);
 free(encoded);
 return result;
}
</pre></div>
</div>
</div>
<div class="section" id="long-int-unification">
<h3>long/int Unification<a class="headerlink" href="#long-int-unification" title="Permalink to this headline">¶</a></h3>
Python 3 has only one integer type, <a class="reference internal" href="../library/functions.html#int" title="int"><tt class="xref py py-func docutils literal">int()</tt></a>. But it actually
corresponds to Python 2&#8217;s <a class="reference internal" href="../library/functions.html#long" title="long"><tt class="xref py py-func docutils literal">long()</tt></a> type&#8211;the <a class="reference internal" href="../library/functions.html#int" title="int"><tt class="xref py py-func docutils literal">int()</tt></a> type
used in Python 2 was removed. In the C-API, <tt class="docutils literal">PyInt_*</tt> functions
are replaced by their <tt class="docutils literal">PyLong_*</tt> equivalents.
</div>
</div>
<div class="section" id="module-initialization-and-state">
<h2>Module initialization and state<a class="headerlink" href="#module-initialization-and-state" title="Permalink to this headline">¶</a></h2>
Python 3 has a revamped extension module initialization system. (See
<a class="pep reference external" href="http://www.python.org/dev/peps/pep-3121">PEP 3121</a>.) Instead of storing module state in globals, they should
be stored in an interpreter specific structure. Creating modules that
act correctly in both Python 2 and Python 3 is tricky. The following
simple example demonstrates how.
<div class="highlight-c"><div class="highlight"><pre>#include &quot;Python.h&quot;

struct module_state {
 PyObject *error;
};

#if PY_MAJOR_VERSION &gt;= 3
#define GETSTATE(m) ((struct module_state*)PyModule_GetState(m))
#else
#define GETSTATE(m) (&amp;_state)
static struct module_state _state;
#endif

static PyObject *
error_out(PyObject *m) {
 struct module_state *st = GETSTATE(m);
 PyErr_SetString(st-&gt;error, &quot;something bad happened&quot;);
 return NULL;
}

static PyMethodDef myextension_methods[] = {
 {&quot;error_out&quot;, (PyCFunction)error_out, METH_NOARGS, NULL},
 {NULL, NULL}
};

#if PY_MAJOR_VERSION &gt;= 3

static int myextension_traverse(PyObject *m, visitproc visit, void *arg) {
 Py_VISIT(GETSTATE(m)-&gt;error);
 return 0;
}

static int myextension_clear(PyObject *m) {
 Py_CLEAR(GETSTATE(m)-&gt;error);
 return 0;
}

static struct PyModuleDef moduledef = {
 PyModuleDef_HEAD_INIT,
 &quot;myextension&quot;,
 NULL,
 sizeof(struct module_state),
 myextension_methods,
 NULL,
 myextension_traverse,
 myextension_clear,
 NULL
};

#define INITERROR return NULL

PyObject *
PyInit_myextension(void)

#else
#define INITERROR return

void
initmyextension(void)
#endif
{
#if PY_MAJOR_VERSION &gt;= 3
 PyObject *module = PyModule_Create(&amp;moduledef);
#else
 PyObject *module = Py_InitModule(&quot;myextension&quot;, myextension_methods);
#endif

if (module == NULL)
 INITERROR;
 struct module_state *st = GETSTATE(module);

st-&gt;error = PyErr_NewException(&quot;myextension.Error&quot;, NULL, NULL);
 if (st-&gt;error == NULL) {
 Py_DECREF(module);
 INITERROR;
 }

#if PY_MAJOR_VERSION &gt;= 3
 return module;
#endif
}
</pre></div>
</div>
</div>
<div class="section" id="cobject-replaced-with-capsule">
<h2>CObject replaced with Capsule<a class="headerlink" href="#cobject-replaced-with-capsule" title="Permalink to this headline">¶</a></h2>
The <tt class="xref c c-type docutils literal">Capsule</tt> object was introduced in Python 3.1 and 2.7 to replace
<tt class="xref c c-type docutils literal">CObject</tt>. CObjects were useful,
but the <tt class="xref c c-type docutils literal">CObject</tt> API was problematic: it didn&#8217;t permit distinguishing
between valid CObjects, which allowed mismatched CObjects to crash the
interpreter, and some of its APIs relied on undefined behavior in C.
(For further reading on the rationale behind Capsules, please see <a class="reference external" href="http://bugs.python.org/issue5630">issue 5630</a>.)
If you&#8217;re currently using CObjects, and you want to migrate to 3.1 or newer,
you&#8217;ll need to switch to Capsules.
<tt class="xref c c-type docutils literal">CObject</tt> was deprecated in 3.1 and 2.7 and completely removed in
Python 3.2. If you only support 2.7, or 3.1 and above, you
can simply switch to <tt class="xref c c-type docutils literal">Capsule</tt>. If you need to support Python 3.0,
or versions of Python earlier than 2.7,
you&#8217;ll have to support both CObjects and Capsules.
(Note that Python 3.0 is no longer supported, and it is not recommended
for production use.)
The following example header file <tt class="file docutils literal">capsulethunk.h</tt> may
solve the problem for you. Simply write your code against the
<tt class="xref c c-type docutils literal">Capsule</tt> API and include this header file after
<tt class="file docutils literal">Python.h</tt>. Your code will automatically use Capsules
in versions of Python with Capsules, and switch to CObjects
when Capsules are unavailable.
<tt class="file docutils literal">capsulethunk.h</tt> simulates Capsules using CObjects. However,
<tt class="xref c c-type docutils literal">CObject</tt> provides no place to store the capsule&#8217;s &#8220;name&#8221;. As a
result the simulated <tt class="xref c c-type docutils literal">Capsule</tt> objects created by <tt class="file docutils literal">capsulethunk.h</tt>
behave slightly differently from real Capsules. Specifically:
<blockquote>
<div><ul class="simple">
<li>The name parameter passed in to <a class="reference internal" href="../c-api/capsule.html#PyCapsule_New" title="PyCapsule_New"><tt class="xref c c-func docutils literal">PyCapsule_New()</tt></a> is ignored.</li>
<li>The name parameter passed in to <a class="reference internal" href="../c-api/capsule.html#PyCapsule_IsValid" title="PyCapsule_IsValid"><tt class="xref c c-func docutils literal">PyCapsule_IsValid()</tt></a> and
<a class="reference internal" href="../c-api/capsule.html#PyCapsule_GetPointer" title="PyCapsule_GetPointer"><tt class="xref c c-func docutils literal">PyCapsule_GetPointer()</tt></a> is ignored, and no error checking
of the name is performed.</li>
<li><a class="reference internal" href="../c-api/capsule.html#PyCapsule_GetName" title="PyCapsule_GetName"><tt class="xref c c-func docutils literal">PyCapsule_GetName()</tt></a> always returns NULL.</li>
<li><a class="reference internal" href="../c-api/capsule.html#PyCapsule_SetName" title="PyCapsule_SetName"><tt class="xref c c-func docutils literal">PyCapsule_SetName()</tt></a> always raises an exception and
returns failure. (Since there&#8217;s no way to store a name
in a CObject, noisy failure of <a class="reference internal" href="../c-api/capsule.html#PyCapsule_SetName" title="PyCapsule_SetName"><tt class="xref c c-func docutils literal">PyCapsule_SetName()</tt></a>
was deemed preferable to silent failure here. If this is
inconvenient, feel free to modify your local
copy as you see fit.)</li>
</ul>
</div></blockquote>
You can find <tt class="file docutils literal">capsulethunk.h</tt> in the Python source distribution
as <a class="reference external" href="http://hg.python.org/cpython/file/2.7/Doc/includes/capsulethunk.h">Doc/includes/capsulethunk.h</a>. We also include it here for
your convenience:
<div class="highlight-c"><div class="highlight"><pre>#ifndef __CAPSULETHUNK_H
#define __CAPSULETHUNK_H

#if ( (PY_VERSION_HEX &lt; 0x02070000) \
 || ((PY_VERSION_HEX &gt;= 0x03000000) \
 &amp;&amp; (PY_VERSION_HEX &lt; 0x03010000)) )

#define __PyCapsule_GetField(capsule, field, default_value) \
 ( PyCapsule_CheckExact(capsule) \
 ? (((PyCObject *)capsule)-&gt;field) \
 : (default_value) \
 ) \

#define __PyCapsule_SetField(capsule, field, value) \
 ( PyCapsule_CheckExact(capsule) \
 ? (((PyCObject *)capsule)-&gt;field = value), 1 \
 : 0 \
 ) \

#define PyCapsule_Type PyCObject_Type

#define PyCapsule_CheckExact(capsule) (PyCObject_Check(capsule))
#define PyCapsule_IsValid(capsule, name) (PyCObject_Check(capsule))

#define PyCapsule_New(pointer, name, destructor) \
 (PyCObject_FromVoidPtr(pointer, destructor))

#define PyCapsule_GetPointer(capsule, name) \
 (PyCObject_AsVoidPtr(capsule))

/* Don&#39;t call PyCObject_SetPointer here, it fails if there&#39;s a destructor */
#define PyCapsule_SetPointer(capsule, pointer) \
 __PyCapsule_SetField(capsule, cobject, pointer)

#define PyCapsule_GetDestructor(capsule) \
 __PyCapsule_GetField(capsule, destructor)

#define PyCapsule_SetDestructor(capsule, dtor) \
 __PyCapsule_SetField(capsule, destructor, dtor)

/*
 * Sorry, there&#39;s simply no place
 * to store a Capsule &quot;name&quot; in a CObject.
 */
#define PyCapsule_GetName(capsule) NULL

static int
PyCapsule_SetName(PyObject *capsule, const char *unused)
{
 unused = unused;
 PyErr_SetString(PyExc_NotImplementedError,
 &quot;can&#39;t use PyCapsule_SetName with CObjects&quot;);
 return 1;
}

#define PyCapsule_GetContext(capsule) \
 __PyCapsule_GetField(capsule, descr)

#define PyCapsule_SetContext(capsule, context) \
 __PyCapsule_SetField(capsule, descr, context)

static void *
PyCapsule_Import(const char *name, int no_block)
{
 PyObject *object = NULL;
 void *return_value = NULL;
 char *trace;
 size_t name_length = (strlen(name) + 1) * sizeof(char);
 char *name_dup = (char *)PyMem_MALLOC(name_length);

if (!name_dup) {
 return NULL;
 }

memcpy(name_dup, name, name_length);

trace = name_dup;
 while (trace) {
 char *dot = strchr(trace, &#39;.&#39;);
 if (dot) {
 *dot++ = &#39;\0&#39;;
 }

if (object == NULL) {
 if (no_block) {
 object = PyImport_ImportModuleNoBlock(trace);
 } else {
 object = PyImport_ImportModule(trace);
 if (!object) {
 PyErr_Format(PyExc_ImportError,
 &quot;PyCapsule_Import could not &quot;
 &quot;import module \&quot;%s\&quot;&quot;, trace);
 }
 }
 } else {
 PyObject *object2 = PyObject_GetAttrString(object, trace);
 Py_DECREF(object);
 object = object2;
 }
 if (!object) {
 goto EXIT;
 }

trace = dot;
 }

if (PyCObject_Check(object)) {
 PyCObject *cobject = (PyCObject *)object;
 return_value = cobject-&gt;cobject;
 } else {
 PyErr_Format(PyExc_AttributeError,
 &quot;PyCapsule_Import \&quot;%s\&quot; is not valid&quot;,
 name);
 }

EXIT:
 Py_XDECREF(object);
 if (name_dup) {
 PyMem_FREE(name_dup);
 }
 return return_value;
}

#endif /* #if PY_VERSION_HEX &lt; 0x02070000 */

#endif /* __CAPSULETHUNK_H */
</pre></div>
</div>
</div>
<div class="section" id="other-options">
<h2>Other options<a class="headerlink" href="#other-options" title="Permalink to this headline">¶</a></h2>
If you are writing a new extension module, you might consider <a class="reference external" href="http://www.cython.org">Cython</a>. It translates a Python-like language to C. The
extension modules it creates are compatible with Python 3 and Python 2.
</div>
</div>

</div>
 </div>
 </div>
 <div class="sphinxsidebar">
 <div class="sphinxsidebarwrapper">
 <h3><a href="../contents.html">Table Of Contents</a></h3>
 <ul>
<li><a class="reference internal" href="#">Porting Extension Modules to Python 3</a><ul>
<li><a class="reference internal" href="#conditional-compilation">Conditional compilation</a></li>
<li><a class="reference internal" href="#changes-to-object-apis">Changes to Object APIs</a><ul>
<li><a class="reference internal" href="#str-unicode-unification">str/unicode Unification</a></li>
<li><a class="reference internal" href="#long-int-unification">long/int Unification</a></li>
</ul>
</li>
<li><a class="reference internal" href="#module-initialization-and-state">Module initialization and state</a></li>
<li><a class="reference internal" href="#cobject-replaced-with-capsule">CObject replaced with Capsule</a></li>
<li><a class="reference internal" href="#other-options">Other options</a></li>
</ul>
</li>
</ul>

<h4>Previous topic</h4>
 <a href="pyporting.html"
 title="previous chapter">Porting Python 2 Code to Python 3</a>
 <h4>Next topic</h4>
 <a href="curses.html"
 title="next chapter">Curses Programming with Python</a>
<h3>This Page</h3>
<ul class="this-page-menu">
 <li><a href="../bugs.html">Report a Bug</a></li>
 <li><a href="../_sources/howto/cporting.txt"
 rel="nofollow">Show Source</a></li>
</ul>

<div id="searchbox" style="display: none">
 <h3>Quick search</h3>
 <form class="search" action="../search.html" method="get">
 <input type="text" name="q" />
 <input type="submit" value="Go" />
 <input type="hidden" name="check_keywords" value="yes" />
 <input type="hidden" name="area" value="default" />
 </form>
 
 Enter search terms or a module, class or function name.
 
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
 </div>
 </div>
 <div class="clearer"></div>
 </div>
 <div class="related">
 <h3>Navigation</h3>
 <ul>
 <li class="right" style="margin-right: 10px">
 <a href="../genindex.html" title="General Index"
 >index</a></li>
 <li class="right" >
 <a href="../py-modindex.html" title="Python Module Index"
 >modules</a> |</li>
 <li class="right" >
 <a href="curses.html" title="Curses Programming with Python"
 >next</a> |</li>
 <li class="right" >
 <a href="pyporting.html" title="Porting Python 2 Code to Python 3"
 >previous</a> |</li>
 <li><img src="../_static/py.png" alt=""
 style="vertical-align: middle; margin-top: -1px"/></li>
 <li><a href="http://www.python.org/">Python</a> &raquo;</li>
 <li>
 <a href="../index.html">Python 2.7.5 documentation</a> &raquo;
 </li>

<li><a href="index.html" >Python HOWTOs</a> &raquo;</li> 
 </ul>
 </div>
 <div class="footer">
 &copy; <a href="../copyright.html">Copyright</a> 1990-2019, Python Software Foundation.
 
 The Python Software Foundation is a non-profit corporation.
 <a href="http://www.python.org/psf/donations/">Please donate.</a>
 
 Last updated on Jul 03, 2019.
 <a href="../bugs.html">Found a bug</a>?
 
 Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.
 </div>

</body>
</html>

Cancel