Friday, June 6, 2014

PHP memory consumption with Arrays and Objects (update: HHVM, phpng June-2014)

Lessons learned:
  • PHPng is significantly faster and uses significantly less memory
  • objects need more memory than arrays (+ 5-250 percent)
  • if array values are numeric, don't save them as strings!
  • saving 1M integers takes 33M of memory with PHPng (increase by factor 8)
  • saving 1M integers as strings takes 79M of memory with PHPng (increase by factor 20)
  • using SplFixedArray can reduce memory usage by 20-100 percent
  • avoid big Arrays and Objects in PHP whenever possible
    (don't use file('big_file') or explode("\n", file_get_contents('big_file')), etc.)
  • use streams whenever possible (fopen, fsockopen, etc.)
  • use generators when available with PHP 5.5 (RFC)
  • comparing 32bit to 64bit systems, memory consumption increases by 100-230 percent

If you need to save memory temporarily in your script, you can stringify your array (increasing cpu usage and runtime):
function test(){

$a = array();
for ($i=0; $i<1000000; $i++) $a[] = $i;
$a = implode(',', $a); // or $a = json_encode($a);
echo number_format(memory_get_usage(true)/1048576, 2)."\n";

}
test();
// 7.5 M / 0.64s (php), 10.4 M / 0.55s (hhvm), 7.25 M / 0.20s (phpng)

Here is a small script to show how much memory PHP needs to handle big Arrays:
the total runtime of the script is 15.5s (php), 8.4s (hhvm), 4.6s (phpng)
(tests made with PHP 5.5.9, HHVM 3.2.0-2014-06-06, PHP 5.7.0-dev-2014-06-05, 2.8GHz single-core openvz)
ini_set('memory_limit', '1024M');

function test(){

$a = '';
for ($i=0; $i<1000000; $i++) $a .= '1';
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 1.50 (php), 3.5 (hhvm), 1.50 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = true;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.25 (php), 30.4 (hhvm), 32.75 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = true;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146.25 (php), 80.92 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = true;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54.25 (php), 30.56 (hhvm), 16.50 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = $i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.50 (php), 30.56 (hhvm), 33.25 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = $i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146 (php), 80.97 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = $i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54.25 (php), 30.58 (hhvm), 16.75 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 230 (php), 61.10 (hhvm), 78.50 (phpng)

$a = new stdclass();
for ($i=0; $i<1000000; $i++) $a->$i = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 237.75 (php), 111.50 (hhvm), 128.25 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 145.5 (php), 61.11 (hhvm), 61.75 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[(string)$i] = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 230 (php), 61.12 (hhvm), 78.50 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = "";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.5 (php), 30.62 (hhvm), 33.75 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = "";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146 (php), 81.02 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = "";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54 (php), 30.63 (hhvm), 17.00 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = "hello";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.5 (php), 30.64 (hhvm), 33.75 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = "hello";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146 (php), 81.04 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = "hello";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54 (php), 30.65 (hhvm), 17.25 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = array();
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 222.5 (php), 30.65 (hhvm), 34.00 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = array();
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 230 (php), 81.05 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = array();
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138 (php), 30.66 (hhvm), 17.50 (phpng)


$a = array();
for ($i=0; $i<500000; $i++) $a[$i] = array("");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 214.75 (php), 18.67 (hhvm), 18.25 (phpng)

$a = new stdclass;
for ($i=0; $i<500000; $i++) $a->$i = array("");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 218.25 (php), 41.81 (hhvm), 41.75 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<500000; $i++) $a[$i] = array("");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 176.00 (php), 30.68 (hhvm), 17.50 (phpng)


$a = array();
for ($i=0; $i<500000; $i++) $a[$i] = array("hello");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 214.50 (php), 18.68 (hhvm), 18.25 (phpng)

$a = new stdclass;
for ($i=0; $i<500000; $i++) $a->$i = array("hello");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 218.25 (php), 41.82 (hhvm), 41.75 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<500000; $i++) $a[$i] = array("hello");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 176 (php), 30.69 (hhvm), 17.50 (phpng)

}
test();

More details and technical explanations about PHP's memory usage can be found on nikic's blog.
More details about PHPng can be found here.

Shortly: a string or an integer in PHP are not mapped to a char-array or an integer in C. A more complex structure is used to get from dynamic typing in PHP to static typing in C:

struct _zval_struct {
  zvalue_value value;     // value object
  zend_uint refcount__gc; // number of references to variable
  zend_uchar type;        // type of variable
  zend_uchar is_ref__gc;  // reference? (&$var)
};
typedef union _zvalue_value {
  long lval;              // integer and boolean
  double dval;            // float (double)
  struct {
    char *val;            // string (with zero bytes)
    int len;              // length of the string
  } str;
  HashTable *ht;          // array (hash table)
  zend_object_value obj;  // object
} zvalue_value;

4 comments:

  1. Thanks for the post. It helped us to find a bug in phpng.
    Could you, please, rerun your test with latest phpng and update digits.

    ReplyDelete
    Replies
    1. I did a new test with PHP 5.5.9 (default package from Ubuntu 14.04), hhvm 3.2 nightly and phpng from yesterday and the results are really amazing:

      for ($i=0; $i<500000; $i++) $a[$i] = array("hello");
      // 214.50 (php), 18.68 (hhvm), 184.75 (phpng May'14), 18.25 (phpng June'14)

      The total time for phpng decreased from 10s to 4.6s, which is now much faster than the latest hhvm. I think benchmarks with real applications should also show improvements because most of them put contents from databases in arrays. You're doing an absolutely great job!

      Delete
  2. Nice it seems to be good post... It will get readers engagement on the article since readers engagement plays an vital role in every blog.i am expecting more updated posts from your hands.
    Android App Development Company

    ReplyDelete

Labels

performance (23) benchmark (6) MySQL (5) architecture (5) coding style (5) memory usage (5) HHVM (4) C++ (3) Java (3) Javascript (3) MVC (3) SQL (3) abstraction layer (3) framework (3) maintenance (3) Go (2) Golang (2) HTML5 (2) ORM (2) PDF (2) Slim (2) Symfony (2) Zend Framework (2) Zephir (2) firewall (2) log files (2) loops (2) quality (2) real-time (2) scrum (2) streaming (2) AOP (1) Apache (1) Arrays (1) C (1) DDoS (1) Deployment (1) DoS (1) Dropbox (1) HTML to PDF (1) HipHop (1) OCR (1) OOP (1) Objects (1) PDO (1) PHP extension (1) PhantomJS (1) SPL (1) SQLite (1) Server-Sent Events (1) Silex (1) Smarty (1) SplFixedArray (1) Unicode (1) V8 (1) analytics (1) annotations (1) apc (1) archiving (1) autoloading (1) awk (1) caching (1) code quality (1) column store (1) common mistakes (1) configuration (1) controller (1) decisions (1) design patterns (1) disk space (1) dynamic routing (1) file cache (1) garbage collector (1) good developer (1) html2pdf (1) internationalization (1) invoice (1) just-in-time compiler (1) kiss (1) knockd (1) legacy code (1) legacy systems (1) logtop (1) memcache (1) memcached (1) micro framework (1) ncat (1) node.js (1) openssh (1) pfff (1) php7 (1) phpng (1) procedure models (1) ramdisk (1) recursion (1) refactoring (1) references (1) regular expressions (1) search (1) security (1) sgrep (1) shm (1) sorting (1) spatch (1) ssh (1) strange behavior (1) swig (1) template engine (1) threads (1) translation (1) ubuntu (1) ufw (1) web server (1) whois (1)