Never trust other peoples benchmarks - A recent example (exceptions)

Maybe you have read it maybe you didn’t:

Some days ago there was a blog post regarding php exception performance in 5.4 and the numbers got reported all over the place.

The actually numbers are secondary. The main point is: Don’t trust “random” stuff on the Internet when thinking about improving your application performance. You always need to measure things for your self and take care doing so!

I’ve initially trusted the benchmark myself and disgraced the whole post saying: “Well yes, exceptions are slower than if statements but nice that they got faster”.

After a small comment by Nikita Popov during a chat session we had and his comment on the blog post. Quoting:

A quick tip for the next time when you do “benchmarking”: Try to understand the reason behind it.

For example here the reason is that you did your PHP 5.3 benchmarking with xDebug enabled, and the PHP 5.4 benchmarking with XDebug disabled.

With XDebug (5.3.8) the Exception code needs 1.2 seconds on my machine.

Without it (still 5.3.8) it needs only 0.14 seconds. On 5.4RC3 (without XDebug) it needs 0.09. So the improvement is actually quite minimal (only factor of two).

So, next time you do benchmarks, don’t forget to disabled XDebug

So I did copy the script to look at the numbers my self (the script is attached at the bottom. 1 million runs instead of 100k to get nicer numbers)

With xDebug disabled for both versions:

/opt/php5.3.8/bin/php a.php 
odds: 500000, evens 500000Array
(
    [memory] => 72.665935516357
    [microtime] => 1.4795081615448
)
odd: 500000, evens 500000Array
(
    [memory] => -7.62939453125E-6
    [microtime] => 0.42677807807922
)

/opt/php5.4.0RC5/bin/php  a.php 
odds: 500000, evens 500000Array
(
    [memory] => 72.665348052979
    [microtime] => 1.0610599517822
)
odd: 500000, evens 500000Array
(
    [memory] => 0
    [microtime] => 0.47566914558411
)

Showing an improvement from 1.5 seconds to 1.0 seconds.

Turning on xDebug for BOTH versions

/opt/php5.3.8/bin/php a.php 
odds: 500000, evens 500000Array
(
    [memory] => 72.666839599609
    [microtime] => 5.929713010788
)
odd: 500000, evens 500000Array
(
    [memory] => -0.0006256103515625
    [microtime] => 0.6544771194458
)

/opt/php5.4.0RC5/bin/php  a.php 
odds: 500000, evens 500000Array
(
    [memory] => 72.665969848633
    [microtime] => 5.3529570102692
)
odd: 500000, evens 500000Array
(
    [memory] => 6.866455078125E-5
    [microtime] => 0.65565991401672
)

Showing 5.9 seconds versus 5.3 seconds.

So what do we learn from that?

Running stuff with debugging tools is slower than not doing that. Thats why we don’t use xDebug in production :)

More seriously: When measuring take care, when believing measurements take care too :)

The script I’ve used / copied

error_reporting(-1);
$time = microtime(TRUE);
$mem = memory_get_usage();

$even = $odd = array();
foreach (range(1, 1000000) as $i) {
  try {
    if ($i % 2 == 0) {
      throw new Exception("even number");
    } else {
      $odd[] = $i;
    }
  } catch (Exception $e) {
    $even[] = $i;
  }
}
echo "odds: " . count($odd) . ", evens " . count($even);
print_r(array('memory' => (memory_get_usage() - $mem) / (1024 * 1024), 'microtime' => microtime(TRUE) - $time));

error_reporting(-1);
$time = microtime(TRUE);
$mem = memory_get_usage();

$even = $odd = array();
foreach (range(1, 1000000) as $i) {
    if ($i % 2 == 0) {
        $even[] = $i;
    } else {
        $odd[] = $i;
    }
}

echo "odd: " . count($odd) . ", evens " . count($even);
print_r(array('memory' => (memory_get_usage() - $mem) / (1024 * 1024), 'microtime' => microtime(TRUE) - $time));

Tags » php

19 January 2012
Views
4 Comments
Permalink
Favorited 0 Times

Comments 4 Comments

Jan 20, 2012

Frosty Z said...

Interesting, thanks for sharing.

Jan 25, 2012

Roman Marintsenko said...

Well improvement is not really that small. 50% faster is a big difference, actually..

Mar 13, 2012

Gonzalo Ayuso said...

The aim of the original benchmark wasn't to check performance between php5.3 and php5.4. It was to check if the use of exceptions adds a performance lack or not. After checking that usage of exceptions is the same than creating object I ran the script with my php5.4 host and I show the outcomes (because I had one host with a RC). Thanks to a comment I realized that my php5.4 has got xdebug enabled. Then I quit Xdebug and update the outcomes. That's because I like to share my divagations in the blog. The comments help me a lot learn. Anyway I'm agree with you. Try the scripts in your host and perform the benchmarks by yourself. You will enjoy more :)

Apr 26, 2012

Svetlana said...

Useful info, guys. Thanks!

__edorian's posterous