Cython and the internals of scikit-learn#

The extension modules of scikit-lr are written in Cython and build on the internals of scikit-learn instead of copying them. This page explains how they are used and what it implies, and the conventions of the Cython code. The scikit-learn guides on Cython and on performance also apply.

Using the internals of scikit-learn#

The extension modules cimport the declarations of the .pxd files that scikit-learn installs, and extend its extension types by subclassing them. The meson.build of sklr passes the directory where scikit-learn is installed to Cython, so that it finds them. Only the .pxd files included in the wheels of scikit-learn can be used, so check the installed package rather than its repository.

Code is only copied from scikit-learn when what is needed is not declared in a .pxd file, or when it must diverge too much to be extended, and only after discussing it in an issue. The copied file starts with a comment that tells where it comes from, and the license notice of scikit-learn is distributed along with scikit-lr.

The Python code follows the same idea with the private API of scikit-learn, which is only imported through sklr/utils/_sklearn_compat.py (see Coding guidelines).

Coupling with the version of scikit-learn#

The layout of the extension types and structures of scikit-learn may change between its minor releases, without any deprecation. An extension module compiled against one of them may then crash with another one, instead of raising an error. For this reason:

  • scikit-lr is built against a single minor release of scikit-learn, which pyproject.toml requires both to build and to run it.

  • The version of scikit-learn used to build scikit-lr is recorded in the generated module sklr/_build_info.py. Importing sklr with another minor release raises an ImportError.

  • The workflow “Upcoming scikit-learn” builds and tests scikit-lr every night against the next release of scikit-learn, to find the changes that affect it before that release is published.

Each minor release of scikit-learn thus has its own release line of scikit-lr.

Writing Cython code#

The Cython code follows these conventions:

  • Arrays are passed as typed memoryviews, declared const when they are only read.

  • The inner loops run without the GIL and do not use Python objects.

  • The boundscheck, wraparound and cdivision directives are only disabled where it is safe, with a comment that says why.

Before considering a module finished, read the annotated report that Cython generates for it, which highlights the lines that interact with the Python interpreter:

$ SITE=$(python -c "import sysconfig; print(sysconfig.get_path('purelib'))")
$ cython -a --include-dir "$SITE" sklr/<subpackage>/<module>.pyx

The report is written next to the source, as an .html file that git ignores.