Importing with ctypes in Python: fighting overflows

Introduction

On some cold winter night, we've decided to refactor a few examples and tests for python wrapper in Themis , because things have to be not only efficient and useful, but elegant as well. One thing after another, and we ended up revamping Themis error codes a bit.

Internal error and status flags sometimes get less attention than crypto-related code: they are internals for internal use. Problem is, when they fail, they might break something more crucial in a completely invisible way.

Since best mistakes are mistakes which are not justfixed, but properly analyzed, reflected and recorded, we wrote this small report on a completely boring matter: every edge and connection is a challenge. This story is a reflection on a typical issue: different people working on different layers of one large product, and then look around to wipe out the technical debt.

Strange tests behavior

Anytime we touch Themis wrapper code, we touch the tests because pesticide paradox in software development is no small problem.

It all started with Secure Comparator tests:

# test.py
from pythemis.scomparator import scomparator, SCOMPARATOR_CODES
secret = b'some secret'
alice = scomparator(secret)
bob = scomparator(secret)
data = alice.begin_compare()
while (alice.result() == SCOMPARATOR_CODES.NOT_READY and
bob.result() == SCOMPARATOR_CODES.NOT_READY):
data = alice.proceed_compare(bob.proceed_compare(data))
assert alice.result() != SCOMPARATOR_CODES.NOT_MATCH
assert bob.result() != SCOMPARATOR_CODES.NOT_MATCH

This test attempts to run Secure Comparator with aconstant secret, this way making sure that comparison ends in apositive result (flag is called SCOMPARATOR_CODES.MATCH ). If the secret is matched, tests should finish with success.

Secure Comparator can result in neither SCOMPARATOR_CODES.NOT_MATCH or SCOMPARATOR_CODES.MATCH .

But why the assert has to be just a negative comparison if we're testing a feature with boolean state? Checking against non-equality of NOT_MATCH does not automatically mean it matches.

The first reaction is obviously to see if it even works (via example code ). It did.

Here, the verification code tests for equality, thankfully:

if comparator.is_equal():
print("match")
else:
print("not match")

Fine, so the problem touches only tests. Let's rewrite assert so that it compares scomparator.result() against SCOMPARATOR_CODES.MATCH correct expected state:

# test.py
...
assert alice.result() == SCOMPARATOR_CODES.MATCH
assert bob.result() == SCOMPARATOR_CODES.MATCH

... and bump into unexpected error:

# python test.py
Traceback (most recent call last):
File "test.py", line 23, in
assert alice.result() == SCOMPARATOR_CODES.MATCH
AssertionError

A routine fix of testing of absolutely working feature quickly turns into an interesting riddle. We've added variable output for debugging to see what's really going on inside:

# test.py
...
print('alice.result(): {}\nNOT_MATCH: {}\nMATCH: {}'.format(
alice.result(),
SCOMPARATOR_CODES.NOT_MATCH,
SCOMPARATOR_CODES.MATCH
))
assert alice.result() == SCOMPARATOR_CODES.MATCH
assert bob.result() == SCOMPARATOR_CODES.MATCH

... and get the completely unexpected:

# python test.py
alice.result(): -252645136
NOT_MATCH: -1
MATCH: 4042322160
Traceback (most recent call last):
File "test.py", line 23, in
assert alice.result() == SCOMPARATOR_CODES.MATCH
AssertionError
How come?

>>> import sys
>>> sys.int_info
sys.int_info(bits_per_digit=30, sizeof_digit=4)
...
>>> import ctypes
>>> print(ctypes.sizeof(ctypes.c_int))
4

Even though OS, Python and Themis are 64bit, PyThemis wrapper is made using ctypes, which has 32-bit int type.

Accordingly, receiving 0xf0f0f0f0 from C Themis, ctypes expects a32-bit numberbut 0xf0f0f0f0 is a negative number. Then Python attempts to convert it to an integer without any bit length limit, and literal 0xf0f0f0f0 (from SCOMPARATOR_CODES ) turns into 4042322160 .

This is strange. Let's dive a bit into Themis :

src/soter/error.h :

//
...
/** @brief return type */
typedef int soter_status_t;
/**
* @addtogroup SOTER
* @{
* @defgroup SOTER_ERROR_CODES status codes
* @{
*/
#define SOTER_SUCCESS 0
#define SOTER_FAIL -1
#define SOTER_INVALID_PARAMETER -2
#define SOTER_NO_MEMORY -3
#define SOTER_BUFFER_TOO_SMALL -4
#define SOTER_DATA_CORRUPT -5
#define SOTER_INVALID_SIGNATURE -6
#define SOTER_NOT_SUPPORTED -7
#define SOTER_ENGINE_FAIL -8
...
typedef int themis_status_t;
/**
* @addtogroup THEMIS
* @{
* @defgroup SOTER_ERROR_CODES status codes
* @{
*/
#define THEMIS_SSESSION_SEND_OUTPUT_TO_PEER 1
#define THEMIS_SUCCESS SOTER_SUCCESS
#define THEMIS_FAIL SOTER_FAIL
#define THEMIS_INVALID_PARAMETER SOTER_INVALID_PARAMETER
#define THEMIS_NO_MEMORY SOTER_NO_MEMORY
#define THEMIS_BUFFER_TOO_SMALL SOTER_BUFFER_TOO_SMALL
#define THEMIS_DATA_CORRUPT SOTER_DATA_CORRUPT
#define THEMIS_INVALID_SIGNATURE SOTER_INVALID_SIGNATURE
#define THEMIS_NOT_SUPPORTED SOTER_NOT_SUPPORTED
#define THEMIS_SSESSION_KA_NOT_FINISHED -8
#define THEMIS_SSESSION_TRANSPORT_ERROR -9
#define THEMIS_SSESSION_GET_PUB_FOR_ID_CALLBACK_ERROR -10

src/themis/secure_comparator.h :

...
#define THEMIS_SCOMPARE_MATCH 0xf0f0f0f0
#define THEMIS_SCOMPARE_NO_MATCH THEMIS_FAIL
#define THEMIS_SCOMPARE_NOT_READY 0
...
themis_status_t secure_comparator_destroy(secure_comparator_t *comp_ctx);
themis_status_t secure_comparator_append_secret(secure_comparator_t *comp_ctx, const void *secret_data, size_t secret_data_length);
themis_status_t secure_comparator_begin_compare(secure_comparator_t *comp_ctx, void *compare_data, size_t *compare_data_length);
themis_status_t secure_comparator_proceed_compare(secure_comparator_t *comp_ctx, const void *peer_compare_data, size_t peer_compare_data_length, void *compare_data, size_t *compare_data_length);
themis_status_t secure_comparator_get_result(const secure_comparator_t *comp_ctx);

Now let's see PyThemis side at src/wrappers/themis/python/pythemis/exception.py .

All values here correspond to C code, numbers are small and fit any bit length limits:

from enum import IntEnum
class THEMIS_CODES(IntEnum):
NETWORK_ERROR = -2222
BUFFER_TOO_SMALL = -4
FAIL = -1
SUCCESS = 0
SEND_AS_IS = 1
... What about Secure Comparator part? Looking at the src/wrappers/themis/python/pythemis/sco