While PHP accelerators deliver better server-side performance and scalability, we can also do other things to speed up the end-user experience. HTTP compression is a common technique for improving performance across the network between the server and the client.  Sugar Labs measured the client-side performance of the SugarCRM application both with and without HTTP compression using Yahoo’s slick YSlow plug-in for Firebug on the FireFox browser. We sampled the SugarCRM application under a couple of different scenarios.

What are the benefits of HTTP Compression?

HTTP compression will compress data that is sent between the server and a user’s browser. In most cases it takes just a few lines to enable it on a Web server, and you can compress HTML, JavaScript and CSS. On average you will see the amount of data transferred between a Web server and a user’s browser decrease in size between 3x to 10x.

Depending on the latency between the server and browser, HTTP compression can have a dramatic effect on the performance of round trips.  The greater the latency, the better the impact.  That means implementing HTTP compression across a LAN connection normally doesn’t help and can often negatively impact performance as the time to compress/un-compress on a LAN outweighs the value of fewer network packets.  HTTP compression works best when there are many network hops between the server and the browser such as when your users connect to their Sugar application over a broadband connection.

A common compression utility used by Web servers and browsers is Gzip.  All browsers that Sugar supports also support accepting data that has been compressed using Gzip.  The Sugar Wiki details how to configure Gzip compression for Apache for the best results with SugarCRM.

Here is a brief sampling of the data collected:

This chart shows the amount of data being transferred the very first time a user views a page with an empty browser cache. Each page was tested independently and the browser cache was cleared between page views. This is a worst case scenario. In a real world scenario, many pages would share common JavaScript and CSS libraries that would be cached between round trips. It does however show the amount of savings per page view once you turn on compression.

Overall the average data size without cache headers and without compression came out to 925 Kb from 58 HTTP requests per page. The 58 requests included loading the page’s HTML and the referenced CSS, JavaScript and images. With compression enabled, the average page size reduces to 270 Kb.  Once the CSS, images, and JavaScript are cached in the browser, the majority of data being passed around is just the actual HTML. The following chart reflects the benefits of compression on the HTML payload.

After the initial page hit for each action, the average page size reduced to 101 Kb per page without compression.  Once you enable compression, that number is reduced to 14 Kb per page, but it still makes 58 HTTP requests to the server. Even though the files are cached by the browser, the browser still makes a call to the server asking if the file has changed. That is why the number of requests per page is still the same. The server will only send back data that has changed, but there is still the overhead of making all those HTTP requests.

If you enable cache headers in your web server, the number of HTTP requests made reduces significantly.  On average, the browser will only make 1 request per page view with cache headers enabled.  You can set the cache for as long as you would like. I would recommend about a week. Note that if your users are connecting over HTTPS, the browser will only cache an object for a given session. As soon as a user logs out, the browser cache will clear by default.  Firefox does allow you to change this browser behavior for HTTPS connections by setting the browser.cache.disk_cache_ssl value to true by typing in “about:config” into your Firefox browser and navigating to this setting.

If you want to calculate the overall amount of data being transferred from the server to the client, the formula would be as follows:

for a single user who views z pages of which k are distinct actions

No Compression:

Total Data Transferred = 925 * k + n * (z - k ) * 101

With Compression :

Total Data Transferred =  270 * k + n * (z -k) * 14

So for 1 user viewing 40 pages of which five pages are distinct actions

No Compression:   925 * 5 + 100 * (40 - 5) * 101 = 8,160 Kb per user

With Compression:  270 * 5+ 100 * (40 - 5) * 14 = 1,840 Kb per user

Now assuming that each user views a new page every 30 seconds (i.e. a 30 second “think” time) we can calculate the bandwidth needed

40 page views * 30 seconds/page view = 1,200 seconds

bandwidth = Total Data Transferred/ Seconds

8,160 K/1200 s = 6.80 Kbps without compression per user

1,840 K/1200 s = 1.53 Kbps with compression per user

Now to get the bandwidth for n users, we simply multiply by n

so for 100 users it would be

6.8Kbps * 100  = 680 Kbps without compression for 100 users

1.53Kbps * 100 = 153 Kbps with compression for 100 users

Doesn’t the browser automatically cache files without cache headers?

The browser will try to cache files, but it will still send a request to the server for each file that it loads from cache that does not have a cache header associated to it.  This is to check if the browser should update the file or not. So there is still a slight request load on the server for each file.  Cache headers prevent this. So while you will see the browser caching, it is still beneficial to send cache headers down with XML, JavaScript, images, CSS, TXT, and SWF files.

Does compression work with everything?

For the most part compression works well, but I would recommend not using it for SOAP since there are several SOAP clients out there that do not handle Gzip compressed data very well. Also you may want to disable it for downloads since often times you will be downloading data that has already been compressed which won’t see any benefit from being Gziped again.  Also note the discussion above on the value of HTTP compression when the server and the browser are on the same LAN.

We have been running load test after load test in the Sugar Labs to determine the performance of Sugar 5.1 and where to focus on improving performance in the future.  You will see performance benefits in 5.1 with the new logger utility, SugarLogger, that greatly reduces the CPU load and the revamped internal handling of custom fields that both reduces the amount of code as well as database queries. We already have a list of improvements we’ll be making to 5.2 and 6.0 as well.

We just finished analyzing one of our load tests that we felt the community would find interesting. This test was geared towards showing the benefits of operation code (op-code) caching as well as external data caching with SugarCRM. We ran a load test geared towards the sales automation functionality with a PHP accelerator, and then we ran it again the with the accelerator disabled.

The database was run on a separate server to ensure that it had no impact on the Web server CPU utilization.

What is Operation Code Caching?

When a user requests a PHP page, most PHP Accelerators will compile the PHP code for the requested page into bytecode. It will then save that bytecode in a cache either on the file system or in shared memory that allows the PHP Accelerator to skip the compile step the next time that PHP page is requested. This greatly reduces CPU utilization since a PHP script no longer needs to be compiled on every round trip.

What is External Data Cache?

An External Data Cache is either shared memory or file based storage that many PHP accelerators have that allow for storing data across round trips and sharing data between multiple users. SugarCRM has an API that takes advantage of these and can currently support APC, Zend Platform, and Memcache. This reduces file I/O as well as round trips to the database.

The Results:

We saw that with caching enabled, our test server completed more requests per second and the requests completed faster. Once we disabled caching we saw the average server response time shift from sub .5 second response times to around .75 second response times. You will also note in the graph below how much longer the tail of the chart is without cache. This indicates that there was a noticeable increase in the number of responses that took greater than 1 second response time without the cache enabled.

The biggest factor is the improvement in CPU utilization when you utilize a PHP accelerator. We saw a drop in CPU utilization from around 75% down to around 40% when utilizing both the op-code caching and the external data cache.

Conclusion:

If you are not running SugarCRM with a PHP accelerator, you really should be. You will see dramatic improvements in performance as well as scalability leading to better user productivity and reduced hardware needs. You can check out our wiki page on PHP Accelerators for more information.

I’ve been doing some testing over the last few months on our new Data Center Edition (DCE) of SugarCRM. One of the key components that’s been giving me trouble was our NFS. I could successfully test with hundreds of users on local disk, but switching to NFS would cause a cascading failure where the system would become almost unresponsive.

The tests on NFS started out fine. I routinely had response times around 170ms for the login page. That’s the average response for the local disk tests on these servers. The problem started as soon as my "users" would start going down multiple paths of code. The CPU load would spike and response times would jump. Both would continue to get worse the longer the test ran.

Took some research and talking with one of the core engineers at Zend before I finally isolated the problem. PHP’s realpath_cache_size setting was the culprit.

realpath_cache_size is used by PHP to keep from having to look up file names. Every time you perform any of the various file functions or include/require a file and use a relative path, PHP has to look up where that file really exists. PHP caches those values so it doesn’t have to search the current working directory and include_path for the file you’re working on.

Since the default setting of realpath_cache_size is only 16K, Sugar was filling that up pretty quickly with 50 instances of Sugar running at the same time on one server. I had never seen any ill-effects from this on the local disk runs, but NFS was the perfect breeding ground for this problem. Simple lstat calls take roughly 10x longer to respond than the standard local disk because of the network overhead. Those slight delays—ten-thousandths of a second—were just enough to cause response times to jump.

The fix was easy enough. Up the realpath_cache_size setting. Finding the sweet spot required some trial and error though. My current setting is 128k. I ran tests with 32k and 64k. 32k is still too small and 64k was ok but I still saw random spikes which I think were caused by the cache filling up. Since there’s no way to expose the realpath cache data in PHP, there’s no way to know for sure what is happening inside it. If I have some downtime in the next week or two, I may throw together a quick extension for getting that data so I can make a quick monitor script.

In the meantime, setting realpatch_cache_size to a value between 64k and 128k should work fine for you if you’re serving Sugar off of an NFS. You’ll notice some benefits on a local disk too, but a local disk can keep up with Sugar even without the the optimal realpath caching. Keep in mind that each PHP process allocates memory for the cache, regardless of how much of it is used. For example, if your web server can handle 100 requests to PHP files and all of them run at the same time, PHP will allocate 12.5 megs of memory if your realpatch_cache_size is set to 128k.

I experimented with the realpath_cache_ttl setting some too. My tests ran the same regardless of whether it was configured to have a TTL of a few hours or the default 120 seconds.

Man, giving a talk remote is always hard.  Is my mic too soft?  Can they hear me?  Did some poor guy with a hearing aid just have it blown out ’cause I had my mic too loud?  Am I pacing ok?  What about handling questions?

Well, thanks to the expert help of Susie and Clint, I didn’t have to worry about any of this while giving my talk on Tuning Sugar Databases to SDUG.  Everything functioned smoothly and the only thing I had to worry about was pacing myself, which seemed to work out good. Hopefully those who attended picked up a few new tricks.  For those who didn’t get to attend live, we’ll have a recording of the webinar online soon.  I’ll update when we do.

I promised some links during the presentation, so here you go.  Our wiki is online @ http://www.sugarcrm.com/wiki/.  There’s excellent info throughout the wiki, but the settings I touched on specifically are all outlined in Performance Tweaks for Large Systems, except one.  Someone mentioned that there was no reference to $calculate_response_time on the wiki, that is correct and will be fixed.  In meantime, here’s a quick intro to it. Adding the following line to your config.php:

"calculate_response_time" = true,

Will get you output like:

The other links I mentioned where, in no particular order:

• MySQL Manual
• MySQL GUI Tools
• SugarCRM’s CTO Series

Also, for anyone who’s interested, the MySQL AB tool for automating many of the processes I outlined is MySQL Enterprise. If you’re short on people or expertise, outsourcing your DBA duties to the company that created MySQL is a great option.

Hope that helps get you started on tuning your SugarCRM installs on MySQL.  If you have any questions that we didn’t get covered in the presentation or Q & A, post them here and we’ll get you answers.

Did you know… if you add the following code below to your config_override.php, you will be able to how many resources it took to build a page?

$sugar_config['show_page_resources'] = true;


You’ll find this at the base of every page output:

Resources used to construct this page (queries, files)(14,156)
External cache (hits/total=ratio) local (2/26=8%)

How cool is that?

Sometimes code just doesn’t scale.

One of my favorite tools to keep the code I am writing fast and scalable is xdebug. When I found that Zend Developer Zone did a four part series on xdebug, I felt compelled to pass it along. This article is specific to profiling code:

http://devzone.zend.com/article/2899-Profiling-PHP-Applications-With-xdebug

The above article also highlights some of the tools for analyzing cachegrind files. I’ve searched, but nothing exists to analyze cachegrind files on Mac OS X. If you know of something, post it here!

xDebug does much more than just profiling, so check it out.