Originally posted by jrtc27View Post
stress-ng is not a meaningful way to compare performance across different operating systems. Many of its stressors are littered with #ifdef's to alter what exactly the benchmark runs based on the system being tested, and therefore comparing raw numbers is not comparing like with like. It's really only useful if you run it on two systems that end up with the same stress-ng configuration, unless you absolutely know what you're doing and look carefully at the implementation of the stressors being benchmarked. In fact, if you go read the Ubuntu wiki page (https://wiki.ubuntu.com/Kernel/Reference/stress-ng, as linked by https://github.com/ColinIanKing/stre...eference-guide), it has the following to say:
Yet here we are with them being used as if they had scientific use. If you read the manpage (0.18.00 at time of writing), it has this to say:
Please stop using stress-ng in this way, it is not meaningful. Using it to compare across kernel upgrades is what it's (somewhat) designed for, not this.
To give an illustrative example:
The context switch test is in stress-switch.c. In its default configuration it uses two processes blocking on reads and writes of a pipe as a means to do a context switch. Firstly, we have the following:
Code:
#if defined(HAVE_PIPE2) && \ defined(O_DIRECT) if (pipe2(pipefds, O_DIRECT) < 0) { /* * Fallback to pipe if pipe2 fails */ if (pipe(pipefds) < 0) { pr_fail("%s: pipe failed, errno=%d (%s)\n", args->name, errno, strerror(errno)); return EXIT_FAILURE; } } buf_size = 1;#else if (pipe(pipefds) < 0) { pr_fail("%s: pipe failed, errno=%d (%s)\n", args->name, errno, strerror(errno)); return EXIT_FAILURE; } buf_size = args->page_size;#endifThis illustrates a bug: if pipe2 fails but is deemed available, we use a buf_size of 1, not a page like we should do. Then there's the following
Code:
#if defined(F_SETPIPE_SZ) if (fcntl(pipefds[0], F_SETPIPE_SZ, buf_size) < 0) { pr_dbg("%s: could not force pipe size to 1 page, " "errno=%d (%s)\n", args->name, errno, strerror(errno)); } if (fcntl(pipefds[1], F_SETPIPE_SZ, buf_size) < 0) { pr_dbg("%s: could not force pipe size to 1 page, " "errno=%d (%s)\n", args->name, errno, strerror(errno)); }#endifThis tries to constrain the pipe to a single page, presumably so that in the non-O_DIRECT case it still forces context switching, as the buffer will be full after a single write. But of course in the buggy case documented above we can do PAGE_SIZE (typically 4096) writes before that happens, which will massively skew the result.
There's also the other issue that, if this fails, or isn't a feature available (such as on FreeBSD), stress-ng just ignores it and continues on, despite the fact that it will no longer get the block-on-every-write behaviour. So on Linux you'll be getting a blocked I/O syscall every iteration (although technically you could skip them in a multicore system if the reader and writer happen to interleave nicely, though in practice that's likely hard to achieve), but on FreeBSD you'll presumably be getting many I/O syscalls before you block, thereby amortising the overhead. For example, the ~13x speed increase of FreeBSD over Ubuntu could be coming from the fact that BIG_PIPE_SIZE / PAGE_SIZE in FreeBSD is 64k / 4k = 16, i.e. that in the steady state for a pipe that's grown to be "big" you can buffer 16 writes before the reader needs to wake up.
This is just one of the stressors. You can find examples just like this littered throughout the code, because it just is not designed to be used this way. The numbers are utterly useless, because they're measuring different things on different systems.
