Talk outline

                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                   TESTING NATIVE BINARIES USING CFFI

                             Noufal Ibrahim
                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


Table of Contents
─────────────────

1 Section 1 : Introduction to FFI in python (approx. 7 minutes)
.. 1.1 Calling code from across languages
.. 1.2 Ctypes in python
..... 1.2.1 Standard library
..... 1.2.2 Handles most things.
..... 1.2.3 Directly calls functions from compiled files.
..... 1.2.4 A simple example
..... 1.2.5 TODO Problems with ctypes
.. 1.3 CFFI
..... 1.3.1 A project by the pypy team
..... 1.3.2 More flexible and fixes several issues with ctypes.
..... 1.3.3 Simple example
..... 1.3.4 TODO Different ways of using the library
..... 1.3.5 Almost obviates raw C extensions
..... 1.3.6 TODO Advantages with pypy
2 Section 2 : My use case (approx. 15 minutes)
.. 2.1 cpslib
..... 2.1.1 A low level C rewrite of psutil
..... 2.1.2 Cross platform (currently supports OSX and Linux - Windows support in the works).
..... 2.1.3 A simple API example
.. 2.2 cpslib python bindings using cffi
..... 2.2.1 TODO `cffi_modules' in setup.py
..... 2.2.2 Example usage (Demo)
3 Section 3 : Writing tests in python (approx 5 minutes)
.. 3.1 Now, we can write tests for this
.. 3.2 This validates the C implementation against psutils
.. 3.3 TODO Example of doing the same thing with rust libraries (if time permits)
4 Section 4 : Wrap up and Q/A
.. 4.1 Stuff that I've skipped





1 Section 1 : Introduction to FFI in python (approx. 7 minutes)
═══════════════════════════════════════════════════════════════

1.1 Calling code from across languages
──────────────────────────────────────

  • Loose coupling (e.g. XML-RPC, RESTful services)
    • Slow
    • Serialisation
    • Infrastructure
  • Common layers (e.g. languages that run on the JVM)
    • Some minor API quirks
    • But generally great
    • I have no experience with this.
  • Calling native binaries
    • FFI. libffi
    • Issues - Translation of types, ABIs etc.
    • Different from things like Cython etc. (which compile


1.2 Ctypes in python
────────────────────

1.2.1 Standard library
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


1.2.2 Handles most things.
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


1.2.3 Directly calls functions from compiled files.
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


1.2.4 A simple example
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  ┌────
  │ int
  │ add(int x, int y)
  │ {
  │   return x + y;
  │ }
  └────

  ┌────
  │ gcc -c -o example.o example.c
  │ gcc -shared -o example.so example.o
  └────

  ┌────
  │ import ctypes
  │ ctypes.CDLL("./example.so")
  │
  │ print (example.add(4, 5)) # Prints 9
  └────


1.2.5 TODO Problems with ctypes
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


1.3 CFFI
────────

1.3.1 A project by the pypy team
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


1.3.2 More flexible and fixes several issues with ctypes.
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


1.3.3 Simple example
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  ┌────
  │ f = cffi.FFI()
  │ f.cdef("int add(int, int);")
  │ ffi = f.dlopen("./example.so")
  │ ffi.add(4, 5) # prints 9
  └────


1.3.4 TODO Different ways of using the library
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


1.3.5 Almost obviates raw C extensions
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


1.3.6 TODO Advantages with pypy
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


2 Section 2 : My use case (approx. 15 minutes)
══════════════════════════════════════════════

2.1 cpslib
──────────

2.1.1 A low level C rewrite of psutil
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  • [https://github.com/nibrahim/cpslib]
  • Contributions welcome.


2.1.2 Cross platform (currently supports OSX and Linux - Windows support in the works).
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


2.1.3 A simple API example
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

  ┌────
  │ uint32_t cpu_count(bool);
  │
  │ void
  │ test_cpu_count() {
  │   uint32_t logical = cpu_count(true);
  │   uint32_t physical = cpu_count(false);
  │   printf("Logical : %" PRIu32 "\nPhysical : %" PRIu32 "\n", logical, physical);
  │   printf("\n");
  │ }
  └────


2.2 cpslib python bindings using cffi
─────────────────────────────────────

2.2.1 TODO `cffi_modules' in setup.py
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌


2.2.2 Example usage (Demo)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

◊ 2.2.2.1 In C

  ┌────
  │ #include <inttypes.h>
  │ #include <stdio.h>
  │
  │ #include "pslib.h"
  │
  │ int
  │ main()
  │ {
  │   uint32_t logical = cpu_count(true);
  │   printf("Number of logical CPUs %" PRIu32 "\n", logical);
  │   return 0;
  │ }
  └────


◊ 2.2.2.2 In Python

  ┌────
  │ from pycpslib import lib as P
  │
  │ print ("Number of logical CPUs {}".format(P.cpu_count(True)))
  └────


  ◊ 2.2.2.2.1


3 Section 3 : Writing tests in python (approx 5 minutes)
════════════════════════════════════════════════════════

3.1 Now, we can write tests for this
────────────────────────────────────


3.2 This validates the C implementation against psutils
───────────────────────────────────────────────────────

  ┌────
  │ from pycpslib import lib as P
  │ import psutil
  │
  │ def test_cpu_count(flush):
  │     assert P.cpu_count(1) == psutil.cpu_count(True), "Mismatch in number of logical CPUs"
  │     assert P.cpu_count(0) == psutil.cpu_count(False), "Mismatch in number of physical CPUs"
  └────


3.3 TODO Example of doing the same thing with rust libraries (if time permits)
──────────────────────────────────────────────────────────────────────────────


4 Section 4 : Wrap up and Q/A
═════════════════════════════

4.1 Stuff that I've skipped
───────────────────────────

  • APIs to pass in and return compound data types (arrays, structures
    etc.)
  • Different modes of using the library
  • Q/A