With the introduction of 5.1.0a, there are now some additional configuration options available for your Sugar installation with regards to utilizing PHP external caches.  The new options to add to your config_override.php file are:

• external_cache_disabled_zend
• external_cache_disabled_memcache
• external_cache_disabled_apc

Note that by default, these are all set to false.

Prior to Sugar 5.1.0a, the application would check for the existence of functions or extensions to determine which external cache to use (in order):

• zend (tested by existence of the output_cache_get() function)
• memcache (tested by the existence of the memcache extension)
• apc (tested by the existence of the apc_store() function)

If you had all three installed and activated, you would match the check for zend first and thus be using zend.  If you had just memcache and apc installed, you would be using memcache since it is first in the check order.

With these new settings, you now have more granular control over which external cache is used.  For example, if you installed memcache for the purpose of storing your PHP sessions and wanted to use APC as your external cache, you would need to set ‘external_cache_disabled_memcache => true’  in your config_override.php file–it would also be wise to set external_cache_disabled_zend to true for good measure as well.

A recommendation when utilizing an external cache is to disable all of the possible caches you don’t intend to use to ensure that the proper one is being used.

For more information on the particular caches, please visit this Sugar Wiki article on using PHP accelerators with SugarCRM.

Sugar 5.1 brought many improvements to the Sugar application framework.  In this release, we introduced a new method for creating custom relationships in Studio and Module Builder called the Relationship Editor. This new editor consolidates the functionality of the old “relationship fields” which worked as basic one-to-many relationships as well as the many-to-many relationships that are produced from creating relationships in the 5.0 Module Builder.

As there have been questions regarding how to use this new editor and how exactly it works, this article will cover the inner workings of the Relationship Editor.

Read the rest of this entry »

With the Web Services that SugarCRM provides, there are many clients being written to connect to Sugar servers via the SOAP services called SugarSoap.  This article outlines how to improve SOAP performance by caching the SugarSoap WSDL to reduce the overhead on the Sugar server.  Unless you have a specific reason to do otherwise, everybody should implement this approach when writing a SOAP client that connects to SugarSoap.

The example below uses the NuSOAP library for PHP.  The implementation principle of caching the WSDL applies however to whatever client-side SOAP library you are using.

This example code is what I typically have seen and does not cache the WSDL.

$soapurl = 'http://localhost/soap.php?wsdl';  // Note the wsdl parameter

$soapClient = new nusoapclient($soapurl, true);

$soapClient->call('login',
array('user_auth' =>
array('user_name' => 'admin',
'password' => '1234',
'version' => '.01'),
'application_name' => 'SoapTest')
);

This approach passes the wsdl parameter as part of the SOAP URL and then makes subsequent method calls. This is a very inefficient way of calling SugarSoap as for every method call, the WSDL needs to be generated on the Sugar server which results in a slower than necessary response and an extra load on the server.

The following approach caches the WSDL file on the client-side and is a more efficient way of writing a SOAP client to get a faster response.

$soapurl = 'http://localhost/soap.php?wsdl';

$cache = new wsdlcache('C:\temp', 12000);

The above line is the important one. The first parameter is the location where you will be searching for the WSDL file and next parameter is cache life time in seconds or 0 for unlimited.

$wsdl = $cache->get($soapurl );
if (is_null($wsdl)) {
$wsdl = new wsdl($soapurl );
$cache->put($wsdl);
}

$soapClient = new nusoapclient($wsdl, true);

$soapClient->call('login',
array('user_auth' =>
array('user_name' => 'admin',
'password' => '1234',
'version' => '.01'),
'application_name' => 'SoapTest')
);

So in this scenario, you are caching the WSDL after the first call.  Subsequent calls to the Sugar server won’t download the WSDL again and again. Your client code will rather pull the WSDL from your local cache.

My testing has shown that this approach improves the response time by 40% - 50%.

Note that the SoapClient class built into PHP 5 automatically caches the WSDL.

Any comments or code on how to do the above using other languages like JAVA or .net are much appreciated.

Ready to sharpen your coding skills?

Foothill College in Silicon Valley, CA, is the site of this year’s Silicon Valley Code Camp, a free weekend event by and for developers that features 115 sessions spread across two days. To date, 886 people have registered for this event, which takes place on November 8th and 9th, 2008.

A quick glance at the list of sessions shows a lot of interesting topics, from architecture to Open Source to security. Many speakers come from marquee software companies in the valley, so the content promises to be insightful and valuable. More than just a set of lectures, the Code Camp organizers aim to provide dialogue and discussion amongst developers. They have published a manifesto, which reads:

The Code Camp Manifesto consists of six points: (1) by and for the developer community; (2) always free; (3) community developed material; (4) no fluff – only code; (5) community ownership; and (6) never occur during working hours.

Speaker registrations are closed for this year, but you can still register for the event as an attendee. Just fill out this short form and you’re in!

Keeping up-to-date and connected at the event will be easy as well. They’ve set up a companion Wiki for posting session content, a Twitter feed for broadcasting short updates to attendees before and during the event, an RSS feed for news, and a LinkedIn group so people can keep track of new-found friends after it’s all over.

This “Code Camp” idea isn’t confined to the Bay Area. A quick web search turned up camps in Southern California, Raleigh, North Carolina, Boise, Idaho, and many, many more.

If there’s not a Code Camp in your area, consider organizing a Code Camp of your own! Start small…get a couple of local companies to sponsor some food and prizes, and ask around at local user groups to find developers who are willing to share some aspect of their skill set with others. It’s a classic “win-win” for all: developers get to network with their peers and pick up a new skill or two, sponsors get the attention of lots of potential employees and are seen in the community as “good guys”, and the local college gets additional traffic to the campus, exposing others who may not know of its bounty of classes and resources.

I spoke with someone at Foothill College (the site of Code Camp) and he said they willingly donated the rooms and labs so that the developer community would be aware of their small, slightly out of the way college and would hopefully think about taking courses there once they knew about this gem. As a part-time instructor at Foothill, I heartily applaud this effort and hope to see more returning students on campus, especially if they sign up for my class!

If you are on the SugarCRM mailing list, you received an email this week to save the date for SugarCon ‘09.  The next Sugarcon will be on Monday, February 2nd through Wednesday, February 4th at the Palace Hotel in San Francisco, CA.

Registration will open soon. In the meantime, we suggest you:

• Visit the SugarCon web site for more details about the conference
• Nominate yourself or a colleague to share your SugarCRM story
• View video highlights from last year’s conference

.

I visited the Hotel Palace today with the SugarCon planning team.  Wow!  This is a fantastic hotel right in the heart of downtown San Francisco, just minutes from all the top attractions.  I was particularly impressed with the history of the hotel.  For instance when it was built in 1875, it was the largest hotel in the United States.

On top of the stellar location, the Sugar Team is also pulling together a fabulous conference agenda and some really fun activities for the evenings.  Stay tuned for details in the months to come.

There have been many changes to the Sugar Email Module since the Ajax-based email client was introduced in Sugar 5.0. This article describes what we have done to improve Email in the Sugar 5.0 and 5.1 releases.

First, let’s do a quick review of the email functionality in SugarCRM. Email is used in a variety of different ways including 1) sending “personal” emails from individual users to leads and contacts, 2) sending out “campaign” emails to large recipient lists through the Sugar Campaigns module, 3) archiving email from customers via the email client in the Sugar application and through integrations with MS Outlook and Mozilla Thunderbird, 4) monitoring inbound “personal” email addresses via POP3 or IMAP on a “personal” user basis (for example, joe.user@example.com) and 5) monitoring inbound “group” email addresses also via POP3 or IMAP (for example, sales@example.com).

In Sugar 5.0, we introduced an entirely new AJAX-based email client built into SugarCRM for sending and reading email. Also in Sugar 5.0, we expanded on the Sugar concept of Personal Mail Accounts and Group Mail Accounts.

Personal Mail Accounts – Every user in your Sugar system has the ability to set up Sugar as their personal email client using IMAP or POP3 to monitor inbound email and SMTP or sendmail for sending outbound email. This type of email is intended to be read and managed by the individual user and only shared with other users once it is imported into the Sugar DB.

Group Mail Accounts – You can also monitor a group email address such as sales@example.com. Again using IMAP or POP3, Sugar allows you to import this email directly into the Sugar DB and create relationships between the email and other Sugar objects like Opportunities and Cases. Additionally, with emails imported using a Group Mail Account you can trigger Workflow rules (Sugar Professional and Enterprise only).

More details about the changes to email in Sugar 5.1 are discussed below.

Design issues in Sugar 4 and earlier versions

In the Sugar 4.5.1 and prior releases, there were a few design issues that we explicitly set out to address in the 5.0 and 5.1 releases.

Excessive database growth

In 4.5.1, Sugar imported all email content (whether belonging to personal or group accounts) directly into the database. So over a period of time, the Sugar 4.5.1 database could, and often did, grow enormously if you monitored very active email addresses. This was especially unexpected when using Sugar as a personal email client over an IMAP connection as users and sys admins didn’t expect this configuration to automatically archive all email into the Sugar DB.

Additionally, because all email was directly imported into the Sugar DB in 4.5.1, there was no provision to filter email beforehand, especially for spam emails. In contrast, in Sugar 5.x, only email headers are initially downloaded and displayed from the server. The actual email content is obtained only when you read the email or you import the email. No database records are created until the import step in Sugar 5.x.

Also during the 4.5.1 import process, Sugar checked for any Contacts, Leads, Accounts, Users, or Targets with an email address that matched the From email address of the imported email. If there was, Sugar explicitly linked the imported email with those records. Because of the de-normalized schema design in Sugar 4 around email addresses, that meant Sugar created a linking row in the db for each type of link object, which often led to excessive growth of the database.

Lots of stored email with lots of linked relationships meant unnecessarily large databases.  Sugar 5.x addresses this issue by providing a single link from the archived (imported) email to possibly multiple CRM object types.

Slow POP3 Connections

During the 4.5.1 email import process, Sugar would check if an email on the mail server already existed in the Sugar DB by doing a duplicate check function. If the email is not already in the Sugar DB, the email module would import the email from the mail server to the Sugar DB.

For IMAP connections, this approach worked very well. The IMAP protocol provides for querying mail servers to check if any new mail arrived after a given date. This allowed any subsequent call to the IMAP mail server to be very, very fast even if you don’t delete email from the mail server.

However, this design was very slow for POP3 connections. The POP3 protocol does not provide this date range check, i.e., “find new email since we last checked”. This meant Sugar had to do a duplicate check for every email on the POP3 mail server. So imagine if you had thousands of emails in the mail account you want to monitor via POP3 and you left every email on the server, then Sugar had to step through every email every time in order to determine if any new email has arrived.

The best possible design that makes a POP3 configuration fast is to delete email from the mail server after it has been imported into Sugar. Sugar 4 allowed you to configure POP3 connections that left email on the server after importing into Sugar. Sugar 5 no longer allows that configuration for POP3.

Note that starting with Sugar 5.0, all POP3 emails are deleted from the server after they are imported. Sugar 5, however, does provide the “Leave Messages on Server” setting for IMAP connections beginning with Sugar 5.1.0a.

Limited user interface

• The overall UI was not intuitive in 4.5.1.
• The UI would allow you to view only one email account.
• For each Group Mail Account, you could map only one Group Folder to that account.
• The concept of Sent and Trash folders did not exist in the Email UI. For instance, the Sugar system couldn’t automatically save a copy of the email in the Sent folder of an IMAP account.

Sugar 5 delivers a new email system

All of these issues were addressed in Sugar 5 but it is important for developers and sys admins to understand how the Sugar 5 email system works. Here are detailed steps on how to use email in Sugar 5.0 and 5.1.

Making Sugar 5 auto-import similar to 4.5.1

If you want all emails to be automatically imported like in 4.5.1, then simply create a Group Mail Account in the Email Admin screen and define a destination Group Folder for the monitored email (i.e., have an entry in the field “Assign To Group Folder”). Then set up the Sugar Scheduler system and enable the “Check Inbound Mailboxes” scheduler. This process will import the full content of all messages into the Sugar system from the monitored mailbox on a regular basis.

Note that while the Sugar 5.x DB has a much more efficient schema design when compared to the 4.5.1 version, this method will still cause the database to grow, perhaps significantly, as the entire content of imported emails is automatically stored.

Group Mail Accounts

Group mail accounts are designed to have multiple users monitor and respond to email that comes into the mail account. A group mail account of sales@example.com, for instance, may be associated with all your sales representatives.To monitor a group email address, the system administrator needs to configure the monitored email accounts in the Admin screen and configure the Sugar scheduler to process email coming into those accounts.

Group Mail Account WITH Group Folder - The recommended approach for Group Mail is to use a Group Folder. A Group Folder essentially provides storage for incoming emails. It also provides for the convenient administration of access privileges. Users that have visibility to a Group Folder (via the Group Folder “Assign to Team” field available in Sugar Professional and Sugar Enterprise only) will automatically see the Group Folder in the Email tab.

In this configuration, Sugar will periodically connect to the mail server and automatically import the entire email into the Sugar DB. When imported into the Sugar DB, it is possible to automatically create Cases, execute Workflows or create relationships between the email and other Sugar objects like Opportunities and Cases.

This configuration works best when the mail server has a powerful spam filter to help you prevent importing spam emails. You also need to plan to have enough database storage space to store a large number of emails.

Group Mail Account WITHOUT Group Folder - In order to mitigate database growth and still give multiple users access to a Group Mail Account, you can also create a Group Mail Account without a Group Folder. In this scenario, the Sugar system will not automatically import email into the Sugar DB. Rather the inbound email scheduler process will connect to the mail server and cache the headers of email messages in the Group Account.

In order to subscribe to a Group Mail Account without a Group Folder, each user must click on the Settings button in the Email tab, click on the Accounts tab and then add the Group Mail Account to the list of email folders in the Email tab. When using Group Folders as described above, the Group Folder will automatically appear in the user’s folder pane in the Email screen.

Similar to personal email described below, emails in a Group Mail Account without a Group Folder must be manually imported into the Sugar DB. That means Workflows and linking to other objects can only happen once the user clicks the “Import to Sugar” button in the email. The exception to this is with emails that have the Case macro in the subject line. Those emails are automatically imported into the Sugar DB in all circumstances so that the automatic case logic is always executed.

Personal Mail Accounts

In the case of monitoring personal email, each user needs to configure his personal email account in the Email tab. When a user sets up such an account, the Sugar system downloads just the headers of all the email messages, but not the email body, and puts the headers in the email_cache table in the Sugar DB.Once the user clicks on an email to read it, the email message body is now downloaded from the mail server and put into the local file system in the cache directory. However at this point, the email is still not “imported” into Sugar and therefore it is neither linkable to other objects nor visible to other users.

Note that this process of querying the mail server multiple times and storing as little data as possible in the Sugar DB works well for mail servers like qmail, MS Exchange, GMail, Yahoo and others. However there are many mail servers that don’t allow more than certain number of connections within a specified time. So when you click on any message, the mail server may refuse further connections. If you run into this issue, it’s best to use a Group Mail Account with Group Folder and have all of the email imported into the Sugar DB on a periodic basis.

Import Mail - To link the email to other objects and make the email visible to other users, the user must import the email into the Sugar system from the mail system. An “Import to Sugar” button is presented to the user when the email is opened. When clicked, the Sugar system downloads the entire email again from the mail server and puts it in the email table. During the import process (similar to archiving an email in the Sugar Plug-In for Outlook), the user is presented the option of explicitly linking the email to a Sugar object such as an Opportunity or Case.

Check Mail - The “Check Mail” button in the Email screen checks mail for all configured personal email accounts only. So if you have configured more than one account (i.e. GMail or Yahoo), Sugar will do the check mail process for both the accounts and all the folders within the accounts. This check mail process will only get the headers of the messages. Once you click on a particular message, the Sugar system will go to the mail server and download the message.

Synchronize - The purpose of synchronize action is to bring the mailbox in sync with the mail server. So in this scenario, the Sugar system will delete all the emails headers from the email_cache db table and check mail for all the accounts in all the folders.

Using Monitored Folders

For both types of mail accounts (Group and Personal), we have provided the facility to subscribe to multiple folders within the account. When you set up an account and select IMAP as the protocol, next to Monitored Folders text box field, there is a select button. When you click on that, it shows a list of available folders to subscribe to. You can also select a Trash folder where if you delete an email from an IMAP or POP folder, the Sugar system will delete the email from the email_cache table, delete the email from the mail server and for an IMAP account, and move the email to the Trash folder on the mail server you have configured for this account. You can also configure the Sent Folder for an IMAP account in which when you compose an email and if you select this account in the from dropdown field, we will create a copy of that message in the selected Sent folder of an IMAP account.

Relationships between Emails and Contacts/Leads/Targets

If you have a Contact, Lead or Target whose email address matches the email address of the imported email then those emails will automatically appear in the Contact/Lead/Target’s history subpanel. Note that this only happens after an email is imported which happens automatically with Group Mail Accounts with Group Folders or by clicking the “Import to Sugar” button in all other configurations. Unlike in Sugar 4.5.1 and earlier, you do not need to explicitly link the email to the records. Sugar will implicitly link the email to the records via the email address and not by creating a row in the database as used to happen in 4.5.1.

You can also now manage all the relationships of an email starting in Sugar 5.1. This was not possible in Sugar 5.0. Also, beginning in Sugar 5.1.0 patch A, we have provided a “View Relationships” button which will show the DetailView of an email and you can view/add/remove all the related records.

While PHP accelerators deliver better server-side performance and scalability, we can also do other things to speed up the end-user experience. HTTP compression is a common technique for improving performance across the network between the server and the client.  Sugar Labs measured the client-side performance of the SugarCRM application both with and without HTTP compression using Yahoo’s slick YSlow plug-in for Firebug on the FireFox browser. We sampled the SugarCRM application under a couple of different scenarios.

What are the benefits of HTTP Compression?

HTTP compression will compress data that is sent between the server and a user’s browser. In most cases it takes just a few lines to enable it on a Web server, and you can compress HTML, JavaScript and CSS. On average you will see the amount of data transferred between a Web server and a user’s browser decrease in size between 3x to 10x.

Depending on the latency between the server and browser, HTTP compression can have a dramatic effect on the performance of round trips.  The greater the latency, the better the impact.  That means implementing HTTP compression across a LAN connection normally doesn’t help and can often negatively impact performance as the time to compress/un-compress on a LAN outweighs the value of fewer network packets.  HTTP compression works best when there are many network hops between the server and the browser such as when your users connect to their Sugar application over a broadband connection.

A common compression utility used by Web servers and browsers is Gzip.  All browsers that Sugar supports also support accepting data that has been compressed using Gzip.  The Sugar Wiki details how to configure Gzip compression for Apache for the best results with SugarCRM.

Here is a brief sampling of the data collected:

This chart shows the amount of data being transferred the very first time a user views a page with an empty browser cache. Each page was tested independently and the browser cache was cleared between page views. This is a worst case scenario. In a real world scenario, many pages would share common JavaScript and CSS libraries that would be cached between round trips. It does however show the amount of savings per page view once you turn on compression.

Overall the average data size without cache headers and without compression came out to 925 Kb from 58 HTTP requests per page. The 58 requests included loading the page’s HTML and the referenced CSS, JavaScript and images. With compression enabled, the average page size reduces to 270 Kb.  Once the CSS, images, and JavaScript are cached in the browser, the majority of data being passed around is just the actual HTML. The following chart reflects the benefits of compression on the HTML payload.

After the initial page hit for each action, the average page size reduced to 101 Kb per page without compression.  Once you enable compression, that number is reduced to 14 Kb per page, but it still makes 58 HTTP requests to the server. Even though the files are cached by the browser, the browser still makes a call to the server asking if the file has changed. That is why the number of requests per page is still the same. The server will only send back data that has changed, but there is still the overhead of making all those HTTP requests.

If you enable cache headers in your web server, the number of HTTP requests made reduces significantly.  On average, the browser will only make 1 request per page view with cache headers enabled.  You can set the cache for as long as you would like. I would recommend about a week. Note that if your users are connecting over HTTPS, the browser will only cache an object for a given session. As soon as a user logs out, the browser cache will clear by default.  Firefox does allow you to change this browser behavior for HTTPS connections by setting the browser.cache.disk_cache_ssl value to true by typing in “about:config” into your Firefox browser and navigating to this setting.

If you want to calculate the overall amount of data being transferred from the server to the client, the formula would be as follows:

for a single user who views z pages of which k are distinct actions

No Compression:

Total Data Transferred = 925 * k + n * (z - k ) * 101

With Compression :

Total Data Transferred =  270 * k + n * (z -k) * 14

So for 1 user viewing 40 pages of which five pages are distinct actions

No Compression:   925 * 5 + 100 * (40 - 5) * 101 = 8,160 Kb per user

With Compression:  270 * 5+ 100 * (40 - 5) * 14 = 1,840 Kb per user

Now assuming that each user views a new page every 30 seconds (i.e. a 30 second “think” time) we can calculate the bandwidth needed

40 page views * 30 seconds/page view = 1,200 seconds

bandwidth = Total Data Transferred/ Seconds

8,160 K/1200 s = 6.80 Kbps without compression per user

1,840 K/1200 s = 1.53 Kbps with compression per user

Now to get the bandwidth for n users, we simply multiply by n

so for 100 users it would be

6.8Kbps * 100  = 680 Kbps without compression for 100 users

1.53Kbps * 100 = 153 Kbps with compression for 100 users

Doesn’t the browser automatically cache files without cache headers?

The browser will try to cache files, but it will still send a request to the server for each file that it loads from cache that does not have a cache header associated to it.  This is to check if the browser should update the file or not. So there is still a slight request load on the server for each file.  Cache headers prevent this. So while you will see the browser caching, it is still beneficial to send cache headers down with XML, JavaScript, images, CSS, TXT, and SWF files.

Does compression work with everything?

For the most part compression works well, but I would recommend not using it for SOAP since there are several SOAP clients out there that do not handle Gzip compressed data very well. Also you may want to disable it for downloads since often times you will be downloading data that has already been compressed which won’t see any benefit from being Gziped again.  Also note the discussion above on the value of HTTP compression when the server and the browser are on the same LAN.

While most of the Sugar app has worked fine in the Safari web browser for a while now, the patch A release of Sugar 5.1 adds official support for Safari to SugarCRM.  Check out the newly updated supported platform matrix and the details on the 5.1.0a patch in the release announcement.

This article demonstrates adding an asynchronous call from the cases edit view to simulate returning an SLA computed date.  Everything here is shown except creating the actual logic to determine the date a case should be closed by.   There are several steps that must be completed.

•    Create a custom field to hold the date
•    Create a whole bunch of Javascript to make an Asynch call and handle the result.
•    Create the logic to determine SLA date

This document has step by step instructions.

I don’t like to add to the blogosphere echo chamber, but I came across this great post from a new SugarCRM user that’s worth looking at.  Sean has been running Sugar for two months and has some great ideas for new add-ons to Sugar (Track Skype Calls, Auto merge all dupe emails, iCal feed and more) that would turbo charge Sugar for him.  Any takers in the community?