djbsort: Security

This page covers djbsort's protections against timing attacks, denial-of-service attacks, buffer overflows, and output-ordering bugs.

## Protection against timing attacks

Operations on secrets often influence attacker-observable timing through the branch predictor, the cache state, etc. Timing attacks work backwards to those secrets.

The constant-time discipline protects against these attacks by eliminating data flow from secrets to timings. This matters for sorting software because sorting is a subroutine in various deployed post-quantum cryptosystems, such as Classic McEliece and NTRU Prime.

Part of the constant-time discipline is disabling dynamic frequency-selection mechanisms such as Turbo Boost: those mechanisms expose power information via timing information, and power consumption is influenced via secrets. Another part of the constant-time discipline is writing software that avoids data flow from secrets to timing variations in CPU instructions: for example, avoiding secret branch conditions and secret array indices. This constrains sorting software: many sorting algorithms rely on data flow from array contents to branch conditions and array indices.

djbsort combines simple arithmetic instructions into comparators that sort 2 array elements, and combines comparators into sorting networks to sort arrays of any size. These arithmetic instructions are believed to run in constant time on all supported CPUs. To protect against compilers converting these arithmetic instructions into variable-time instructions, the djbsort test suite includes tests for various array sizes that the compiled software avoids data flow from array contents to branch conditions and array indices. To reduce the chance of compilers introducing these problems in the first place, djbsort uses cryptoint for arithmetic.

The time for sorting still depends on the size of the array being sorted (although one can work around this by padding to a maximum size). This is not an issue in the cryptographic context: the array sizes are standard.

## Protection against denial-of-service attacks

A different security issue caused by timing variations is that attackers influencing inputs can sometimes cause problematic consumption of CPU time. This type of problem is well known in the context of hash flooding but also applies to some sorting algorithms.

In particular, many sorting libraries use quicksort. Quicksort uses Θ(n log n) comparisons on average for random arrays of n integers (each having more than log2n bits), but drastically slows down to Θ(n2) comparisons for many types of non-random arrays. Libraries typically add features to try to avoid slowdowns observed in tests (sample features: checking whether an array consists of many repetitions of the same number; taking the median of the first, middle, and last array elements as a pivot), but these features do not protect against quadratic slowdowns produced by attacker-chosen inputs. Some libraries guarantee that the slowdown will not be worse than an order of magnitude (for example, "introspective sort" automatically falls back to heapsort when quicksort is progressing slowly), but an order-of-magnitude slowdown can still be an attractive attack target.

There does not seem to have been serious investigation of how much slowdown adversarial inputs can produce in current libraries. A brief investigation shows that the `int32` array

    for (i = 0;i < n;++i)
      buf[i] = 10000*(i/66)+((i%66)==0)-((i%66)==1);

makes vqsort slower; the same pattern with 34 instead of 66 makes x86-simd-sort slower; the same pattern with 514 makes the "far" library slower; and a constant array makes vxsort much slower. Most libraries select pivots deterministically, so merely influencing array entries at the pivot positions would make those libraries much slower. The vqsort library seeds its pivot-selection RNG with OS randomness generated once per thread, but there does not seem to have been investigation of whether vqsort's RNG state can be reconstructed via timings, which would then reduce to the deterministic case.

None of this is an issue for djbsort. The sorting networks in djbsort always use Θ(n log2n) comparisons, more precisely about (1/4)n log2n comparisons. CPUs with AVX2 instructions carry out many comparisons per cycle, making djbsort faster for typical array sizes on those CPUs than the average case of the far, vqsort, vxsort, and x86-simd-sort libraries. For larger array sizes, djbsort is only slightly slower, and in any case adversarial array contents cannot cause djbsort to take more time.

## Memory safety

Sorting software, like other software, could have bugs that read or write outside the specified array. For example, the array

    for (i = 0;i < 1291;++i)
      buf[i] = 100000*(4+3*(i/65536)+((i%15)==0)-((i%15)==1));

appears to reliably cause vxsort to crash under asan. In general, such bugs might crash the application, leak sensitive data, or allow an attacker to take control of the application.

For djbsort, array indices are determined entirely by the array size, and the tests include tests under valgrind for many different array sizes.

## Protection against output-ordering bugs

Non-buffer-overflow bugs in sorting algorithms might produce outputs in the wrong order, perhaps causing applications to misbehave in ways that affect security. It is even conceivable that some inputs will be lost (overwritten by other inputs) or corrupted (changed in value).

Beyond traditional tests in its test suite, djbsort includes tools that, given n, verify that the compiled code sorts all size-n inputs correctly. These tools have been run for many choices of n, although this does not rule out the possibility of bugs appearing for other choices of n.

Version: This is version 2026.01.24 of the "Security" web page.