Side Channel

Part one, tl, dr: memcache actually made sugarcrm slower.

Part two, tl, dr: Installing php5-mcrypt makes your sugarcrm site runs faster! Wish they'd note this more prominently.

We reinstalled sugarcrm the other day. Looked okay on development machine. Tested, everyone (for some values of "every") liked it. So we deployed it to the production machine. Used it for a while, then the dashboard and front page promptly slowed down to a crawl, taking up to 10 seconds to display when it was very snappy and loaded in sub-seconds.

After some false starts at troubleshooting, we backported the site back to the development machine. Slow. Okay, time to bring out the heavy duty tools and be serious about it.

---

1. Get Xdebug up

Add following to xdebug.ini:

Set up the output directory, restart apache:

... and reload the offending slow sugarcrm page. xdebug spits out the diags at

---

2. Cachegrind Analysis

The cachegrind file is a rather raw output, so we'd rather have a pretty GUI to visualize it. We'll use kcachegrind for this.

Rather than installing the GUI on the development server, we disabled xdebug, and copied the cachegrind output to our workstation. sudo apt-get install kcachegrind on our workstation will get us everything we need.

Start it up. (Applications / Programming / KCacheGrind). Load cachegrind.out.13582 that we copied over. Set the display to display absolute time (Right click on the data, "Show absolute cost"):

There. Out of the total runtime of 7.57 seconds (first row), 6.94 seconds is spent on php::Memcache->set. What?

Memcache making things slower? Wait, we didn't configure sugarcrm to use Memcache! Apparently sugarcrm automagically uses memcache if it detects it:

Sugar automatically enables external cache support once it detects a supported external cache method

---

3. Conclusion & Fix

It's memcache. It has to be. Time to test.

Before stopping memcached:

After stopping memcached:

Q.E.D. More permanent fix:

sudo vi config_override.php

$GLOBALS['sugar_config']['external_cache_disabled_memcache'] = true;

I'm not sure if the problem is related to sugarcrm or the way we set up our server (all signs point to our version of memcache module for php), but this post is more about how we found the problem, and can be used for just about any php app.

P.S. sugarcrm 5.5a, Ubuntu 8.04 LTS (amd64), mysql 5.0.

P.P.S. Spoiler in title, I know.

---

Part 2: Speed, more speed!

Further testing still showed that our development server is still faster than production. More testing showed that the both development and production server called Sugar's PEAR's Blowfish routines(why sugarcrm is crypting so many things is beyond me) alot, but the development machine runs those faster. Hmmm.

Further looking at the source code, PEAR's blowfish has a Blowfish implementation in PHP as a fallback in case it can't load the mcrypt module for php. Installing php5-mcrypt and restarting the webserver fixed that.