Sugar Developer Blog

I’ve been doing some testing over the last few months on our new Data Center Edition (DCE) of SugarCRM. One of the key components that’s been giving me trouble was our NFS. I could successfully test with hundreds of users on local disk, but switching to NFS would cause a cascading failure where the system would become almost unresponsive.

The tests on NFS started out fine. I routinely had response times around 170ms for the login page. That’s the average response for the local disk tests on these servers. The problem started as soon as my "users" would start going down multiple paths of code. The CPU load would spike and response times would jump. Both would continue to get worse the longer the test ran.

Took some research and talking with one of the core engineers at Zend before I finally isolated the problem. PHP’s realpath_cache_size setting was the culprit.

realpath_cache_size is used by PHP to keep from having to look up file names. Every time you perform any of the various file functions or include/require a file and use a relative path, PHP has to look up where that file really exists. PHP caches those values so it doesn’t have to search the current working directory and include_path for the file you’re working on.

Since the default setting of realpath_cache_size is only 16K, Sugar was filling that up pretty quickly with 50 instances of Sugar running at the same time on one server. I had never seen any ill-effects from this on the local disk runs, but NFS was the perfect breeding ground for this problem. Simple lstat calls take roughly 10x longer to respond than the standard local disk because of the network overhead. Those slight delays—ten-thousandths of a second—were just enough to cause response times to jump.

The fix was easy enough. Up the realpath_cache_size setting. Finding the sweet spot required some trial and error though. My current setting is 128k. I ran tests with 32k and 64k. 32k is still too small and 64k was ok but I still saw random spikes which I think were caused by the cache filling up. Since there’s no way to expose the realpath cache data in PHP, there’s no way to know for sure what is happening inside it. If I have some downtime in the next week or two, I may throw together a quick extension for getting that data so I can make a quick monitor script.

In the meantime, setting realpatch_cache_size to a value between 64k and 128k should work fine for you if you’re serving Sugar off of an NFS. You’ll notice some benefits on a local disk too, but a local disk can keep up with Sugar even without the the optimal realpath caching. Keep in mind that each PHP process allocates memory for the cache, regardless of how much of it is used. For example, if your web server can handle 100 requests to PHP files and all of them run at the same time, PHP will allocate 12.5 megs of memory if your realpatch_cache_size is set to 128k.

I experimented with the realpath_cache_ttl setting some too. My tests ran the same regardless of whether it was configured to have a TTL of a few hours or the default 120 seconds.

I just got back from IBM’s Impact 2008 show. This show was a pretty amazing whirlwind tour with a primary focus around SOA. It took place in the MGM Grand hotel in Vegas. I was amazed at the size of the MGM Grand property. The conference spanned multiple buildings and I would say at least a square mile of territory.

The conference spanned most of the buildings in the picture on the right. Many of the meetings were in the larger buildings that are relatively close. An Un-Conference that I participated in was in the second signature tower. The second signature tower is the middle of the three thin tall towers in the middle picture.

One of the highlights of the conference was the announcement of WebSphere sMash. WebSphere sMash is the commercial offering based on the incubator project Project Zero. ProjectZero is an attempt to implement the PHP language on top of a JVM. WebSphere is going to support dynamic languages. The first two dynamic languages that they are going to support are PHP and Groovy. With this project, you can code your application in PHP, Groovy, C, and Java. All four of these languages will run in the same process space making calls between the different languages very easy. They have also included some great tooling. They have an IDE in Eclipse and an IDE in the web browser. They have also been leveraging their own technologies in the development of the project. They have PHPBB running on the project zero site in WebSphere sMash.

I also had the opportunity to sit down with Jerry Cuomo the CTO and IBM Fellow in charge of this project and record a podcast that is available on IBM’s DeveloperWorks site. At this point, they have two software packages running on sMash: PHPBB and SugarCRM. The instructions for installing and configuring Sugar for sMash are still being refined. There is also a 10 minute video of Sugar on Zero being extended to span multiple languages and leveraging a SOAP to REST API conversion layer.

This project started about 10 months ago and is still getting refined rapidly. At this point it is capable of running Sugar pretty well. It is also setup to create separate Java processes for each application that you are hosting on your machine. Each application has its own set of dependencies and configurations. This allows each application to be setup with the resources that it needs to accomplish its mission independently of any other application on the system. They also have a repository system that allows the distribution of projects complete with custom configurations.

At this point, the current code is implemented in a way to get maximum functionality as quick as possible. a the functionality gets further and further towards completion, I am very interested to see how well the IBM team is able to leverage some of the person centuries IBM has invested in tooling, optimization, monitoring, virtualization, and scalability.

Check out ProjectZero and let me know what you think.

Jacob Taylor

CTO and Co-Founder

Did you hear the rumors that SFDC offered itself to Oracle? Maybe it makes sense for Oracle and SFDC, but does it make sense for developers, the community, and customers?

Tom Foremski says that the two businesses are easily integrated because they’re based on standards. Maybe SFDC actually uses HTML, but APEX is far from an industry standard. How does that help the developer community and ultimately the customers / end-users?

Of course I am an open source advocate and a devoted MySQL user, but Oracle is a stellar database that scales. As a developer and also as the Advanced Support Manager for Sugar, I often find myself working with enterprise organizations that are Oracle shops. So why get stuck with APEX when you can have REAL open standards and flexibility?

If you’re going to develop for Sugar and you’re going to use Oracle - I *HIGHLY* recommend Zend Core for Oracle. It makes installation and maintenance and fine-tuning SOOO easy. My favorite part is that it’s FREE!

I’ve also learned that once you’ve installed Zend Core for Oracle, it would behoove you to upgrade to the latest oci8_client that comes with PHP. PHP 5.2.x+ has the best oci8 client from my personal experience.

What do you think about Oracle and its future? What tips have you learned as a PHP developer and from working with Oracle?