This is my first post in the Sugar Developer Blog, so first a quick introduction. My name is Manoj Jayadevan and I run product management here at SugarCRM. My team works with the community, customers and development team to plan and prioritize the SugarCRM product roadmap. I’ve been with Sugar for over three years now and am really excited to have helped bring you Sugar 5.

Sugar 5.0 has been in the wild for almost 7 months now and you are probably curious what the Sugar team has been cooking up in the meantime. Well, the wait is almost coming to an end. Sugar 5.1 is expected to be released by the end of July 2008 and will have many functional and platform enhancements that the developer community has been waiting for.

The key focus of 5.0 was to make the Sugar platform easier to extend, customize and maintain. We have big plans for SugarCRM. But to create and build advanced features, the foremost requirement is to have a solid and modern platform. Hence in 5.0, we focused all our energies to build the most modern business application platform on the planet. Now that we have a solid platform, it is time to add the bells and whistles to the application.

The functional enhancements in 5.1 are focused around cross-module reporting, import enhancements, tracking system usage, completing the new email client, and taking Studio and Module Builder to the next level. The platform level changes in Sugar 5.1 are very minimal and are primarily focused around our on-demand platform.

Building new features is good, but fixing existing bugs is even better. One of the key drivers for 5.1 is quality improvement. We are addressing over 800 bugs found just in 5.0. These issues are prioritized based on all the feedback from the community and commercial customers. Special quality improvement emphasis has been placed on the Email module and the upgrade process. We are also diligently fixing Sugar 5.0 issues through 5.0 patch releases. In all, we will fix more than 1400 bugs in 5.1 before we ship the GA version next month. Sugar 5.1 will quickly prove to be a very stable application platform.

Now let’s talk features. CRM systems are great at capturing diverse information from customer interactions. This data is no good until aggregated and presented in a meaningful and readable fashion. The Sugar 5.1 reporting engine available in Sugar Professional and Sugar Enterprise is designed to do exactly that. The design philosophy for the 5.1 reporting engine is to provide an easy-to-use yet powerful report builder for end users to generate complex reports. Our goal is not to build yet another general purpose analytics tool, but to provide 90% of CRM reporting use cases using a cross-module report builder. Key enhancements include an AJAX based wizard driven UI, the new matrix report type, tools to create complex filters, support for inner and outer joins, dynamic run-time filters and more.

More reporting improvements ………The Tracker we know today is the small bread crumb bar at the top of every screen that shows the navigation path of the user in the application. But Tracker has more depth to it than what initially meets the eye. In Sugar 5.1, Tracker will be able to capture information like session history, user activities, resource usage etc. Some of the sample reports that can be generated include: who is online, who are my top employees by CRM usage, what modules are used the most. These reports will provide key performance indicators to users and managers for better insight to manage the adoption of SugarCRM within their business. These reports are available only in Sugar Professional and Sugar Enterprise but the tracker engine is available in Sugar CE version as well.

Even more reporting improvements ………To provide additional tools for reporting, we have built a new plug-in for Excel. Why reinvent the wheel when Excel can crunch the numbers for you? This new plug-in installs into Office 2007 and Office 2003 and allows users to move and update data between SugarCRM and Excel without any extra export/import steps. Users can also search for information like reports and accounts from within Excel and transfer the data once the source is located in the CRM system. The Sugar Plug-In for Excel is available in the Professional Edition and Enterprise Editions and is included in Sugar Network for Sugar Community Edition users.

Reporting on data is good…..But how do we get data into the application? ….For that we revamped the import tool in Sugar 5.1. The new tool is completely redesigned and special attention is provided to make imports fast and scalable. The redesign and most of the coding comes from Sugar Community legend, Ken Brill, who has joined the Sugar team and now enjoys spending his time working with Sugar customers. We have observed tremendous improvements in the speed and amount of data that can be imported. Our design goal is to attain an import rate of 10,000 records per minute or more. There is also support for more intuitive duplicate checking, error logging and recovery.

Now let’s talk about our beloved Module Builder and Studio. With Sugar 5.0, we introduced Module Builder. This tool is a huge success. This is evident from the proliferation of modules on the SugarForge and SugarExchange. We are furthering the Module Builder in 5.1 by adding much demanded features like one-to-many, many-to-many and one-to-one relationships. Also, Module Builder and Studio will now support a history and rollback feature for bullet-proofing the development process. Last, but not the least, Studio will now support creating and editing of Sugar Dashlets.

Everyone is going wireless. What about Sugar? Our strategy for Sugar Wireless is to use the new powerful and modern browsers in the latest PDA’s to provide users with a rich client experience. We have re-launched Sugar Wireless with many additional features. This is a rewrite of our first generation client and is based on the new MVC framework for UI generation introduced in Sugar 5.0. The key modules like Accounts, Opportunities, Cases, Employees and Activities are exposed through this browser-based Web application designed for small PDA screens. Our long term goal is to make all the modules available in Sugar Wireless that is creatable by Module Builder and editable by Studio. Imagine this - Any module created by Module Builder or edited by Studio will be magically exposed and through Wireless client.

Email and CRM systems? Why do you need an email client in a CRM system? The simple answer is: to keep the sales folks in the CRM system and streamline CRM adoption and productivity. We built an HTML AJAX client in 5.0 with reasonable success. The community provided tons of feedback. The most common feedback was to fix bugs. So we listened. Substantial investment is made in 5.1 to fix email bugs and also to improve the usability and performance. That said, I would like to take this opportunity to highlight our philosophy in building the email client in 5.0. Our goal is to provide a nice clean and simple way to manage email in CRM. We do not intend to build a MS Outlook-like client in the application. We will be adding a few more simple use cases in the near future, but will not build a full-fledged email client.

So as you can see, there are many reasons to move to Sugar 5.1. We will be adding more features and platform enhancements in the future. And all these new features will be built on the Sugar 5 platform. Hence it’s a great idea to migrate and adopt the Sugar 5 platform and leverage all the great work Sugar developers and community members have done.

Thanks and happy coding.

So you’ve created a new module in Sugar 5.0 with Module Builder, but now you want to import records into that module. Unfortunately in Sugar 5.0, the import function is not automatically enabled for new custom modules. We have addressed this issue in the upcoming Sugar 5.1 but that doesn’t help you today.

For those of you working with Sugar 5.0, here is a link to a sample module where I enabled the import function. You will need to take a look at the code to see how to do the same for your custom modules. I built this in Sugar Community Edition, so it will only install on Sugar 5.0.0 CE. However the approach is exactly the same for the Sugar Professional and Enterprise Editions.

Details
This module package contains a very basic module called “Work”. I built this module in Module Builder (only took 2 minutes, gotta love MB!) based on the Issue template.

When you unzip the module package, you will see directories called EnableImport, icons and SugarModules. Everything in the icons and SugarModules directories is exactly what you get when you create a module with Module Builder and publish the module package.

The EnableImport directory is where I put the necessary code to enable the import function. This directory contains some sub-directories and five files. I will explain those five files below. I also modified the manifest.php for this package to copy over the required files for enabling import and adding in the menu item on the Work module.

Required Files

1. EnableImport\menus\Import_zz_Work.php - This file is used by the manifest.php to add the Import item to the Work module’s shortcuts menu.
2. EnableImport\modules\Import\config.php - This file registers the Work module (zz_Work) as a module that can be imported.
   NOTE: This is the one out-of-the-box file that you must modify which means this change is not upgrade-safe. You will need to manually merge the one-line change in this file back into your code when ever you do upgrades in the future.
3. EnableImport\modules\Import\Importzz_Work.php - Contains the logic for actually doing the import. If you need to do any special logic (data manipulation, creating special relationships, etc), you will create special functions in this file that get executed at import.
4. EnableImport\modules\zz_Work\field_arrays.php - This file is necessary but unfortunately a relic from the past. Up until Sugar 5.1, the Import module does not use the standard vardefs for defining a module’s required fields but rather keys off of this field_arrays.php file that we used back in Sugar 1.0 and 2.0. So you will need to register all of your module’s required fields in this file otherwise your import will fail. Thankfully Sugar 5.1 finally puts this dependency to rest.
5. EnableImport\modules\zz_Work\Import.php - This file actually calls the Import module code when you click on the Import button.

On a final note, be sure to take a look through the manifest.php file for this package to see how I modified it to install these required files.

Have fun and let me know how this works for you.

Clint

I have recently been asked to answer this question a few times, “Does SugarCRM require GUIDs for primary keys in the database? ”

At a high level, Sugar was designed to allow the primary keys to be replaced by any unique string. This could either be a different GUID algorithm, a key that has some meaning (such as bean type first, followed by info), an external key, and/or auto-incrementing numbers.

The GUIDs in Sugar are generally only used as unique primary keys.  They have no other special meaning.  We don’t bother to look at what is in the string and do not rely on what they contain other than the ability to match them against records in the DB and having the include/utils.php function create_guid() return a valid one. Because all of our relationship tables are typed, we link two records (such as an account record with a contact record) with a specified ID in the record type relationship table (e.g. accounts_contacts) instead of just storing the ID and searching to find out what type has that ID.

We chose GUIDs rather than auto-incrementing keys to allow for easier data synchronization across databases. This data synchronization issue comes into play when the Sugar Offline Client (part of Sugar Enterprise) syncs data between the main Sugar installation and the Offline Client or when developers use the Sugar SOAP APIs or a tool like Talend for data synchronization.

The strategy of the Offline Client using GUIDs for primary keys was very easy to implement and we don’t have to worry about data conflicts. If you change the system to use some other ID scheme and do need to accommodate data synchronization across data stores, then you would have to either partition the IDs ahead of time or work out a system similar to what we have for cases and bugs where we have a server ID that is globally unique and a incrementing case or bug number. This is a moderate customization though which would require some careful planning and implementation.

However if you don’t care about data synchronization issues, then you can certainly change the primary key format to some other unique string.

To implement a new primary key method or to import existing data with a different primary key format and then rely on the existing GUID mechanism for new records, there are a few things to look out for:

• The system expects primary keys to be string types and will format the SQL with quotes. If you change the primary key types to an integer type, you might have SQL errors to deal with since we put all ID values in quotes in our generated SQL. The database might be able to be able to ignore this issue. MySQL running in Safe mode will have issues for instance.
• Case-sensitivity can be an issue. IDs “abc” and “ABC” are typically treated the same in MySQL. Some other CRM systems from which people were migrating to SugarCRM were using case sensitive strings as their IDs on export. If this is the case, and you are running MySQL, you need to run an algorithm on the data to make sure all of the IDs are unique. One simple algorithm is to MD5 the ids that they provide. A quick check will let you know if there is a problem. If you imported 80,000 leads and there are only 60,000 in the system, some might have been lost due to non-unique primary keys resulting from this case-sensitivity issue.
• Sugar only tracks the first 36 characters in the primary key. Any replacement primary key will either require changing all of the ID columns with one of an appropriate size or to make sure you don’t run into any truncation or padding issues. MySQL in some versions has had issues with Sugar where the IDs were not matching because it was adding spaces to pad the row out to the full size. MySQL’s handling of char and varchar padding have changed in some of the more recent versions. To protect against this, you will want to make sure the GUIDs are not padded with blanks in the DB.

Jacob

I was reading a PCWorld article today about an Oxford professor Jonathan Zittrain who has written a new book called The Future of the Internet and How to Stop It . He has taken the position that closed Internet appliances like the iPhone are stifling innovation.

Quote: “Zittrain argues that today’s Internet appliances such as the iPhone and Xbox hamper innovation. That’s because these locked-down devices prohibit the kind of tinkering by end users that made PCs and the Internet such a force of economic, political and artistic change.”

I find this interesting from a couple perspectives.

First, Carolyn Duffy Marsan, the PCWorld journalist who wrote the review, doesn’t speak once about open source. Living inside the Open Source bubble, I am amazed at times when somebody in the technology industry writes a story about the evils of closed technology without immediately praising the virtues of open source software. After all, the very core principal behind open source is to revolt against closed source software that prevents the very innovation that Zittrain fears we will lose. For shame Ms. Marsan! Keep preaching Matt Asay!

Secondly, it’s interesting to see that Zittrain’s concern is so similar to the concern that Richard Stallman expressed when he sponsored the recent update to the GPL open source license, GPLv3, and specifically drafted license language against the closed Tivo appliance. His concern, which he calls “tivoization” was that closed Linux-based appliances like the Tivo DVR both stifled innovation and flew in the face of the intent of open source licenses.

What I am happy to see is that others besides us pocket-protector open source enthusiasts are seeing that the true power of technology and the Internet is in the freedom to innovate. One of my favorite quotes is:

“Innovation is the ability to see change as an opportunity - not a threat”

I’m personally not completely convinced that the iPhone, Xboc and Tivo are going to put the brakes on innovation, but I applaud Mr Zittrain and anybody else who points out the virtues of innovation and strive to protect it.

Ok, so I know it’s a bit last-minute, but for those of you who are looking to sharpen your PHP and MySQL skills, there is an online community college course starting this week. It’s taught by Foothill College, one of the local colleges in the Silicon Valley area.

The class is CIS 52N and it’s completely online so you don’t need to be local to take the course, though you will be charged more if you live “out of state” (outside California). You can register for this (or any other) spring quarter course through April 18th.

For more information, or to register for this course, go to Foothill College’s website. Foothill runs classes year around so check out their website (or your own community college) for interesting classes.

I am here today at the Dutch JoomlaDays conference in Utrecht, Netherlands talking Joomla developers about SugarCRM. Great conference with over 200 developers at the event. Kudos to Henk and team for putting on a great show.

I’m listening right now to James Vasile from the Software Freedom talk about building a community and a successful open source project. As an open source lawyer, you would expect him to focus on the open source license as the center of building a community.

Instead his his three building blocks of building a community are:

• Freedom - create community rules that give people the freedom to build, extend and share. While few people typically read an open source license, it is this contract that ensures the freedom.
• Sharing - build an environment where sharing is cool.
• Ecosystem - Focus on growing and invigorating the ecosystem around the software.

I certainly agree with all his points. A couple things I would add are 1) build GREAT software and 2) communicate, communicate, communicate. You only attract people to your community if you build something attractive.  People will only stay if they are working within a community and not yelling into an empty room.

DragonflyMaster, one of our top posters in the Sugar Forums the past several months, has just created a great Wiki article on Building Dynamic Dropdowns. Also called dependent dropdowns, this feature allows you to link two dropdowns together such that the values in the second dropdown are based on the values in the first dropdown.

Great feature!

We have just posted a Developer Survey for the Sugar Community and are asking all SugarCRM developers to give us your feedback. Please take a few minutes to fill out the survey and let us know what tools and resources you would like us to work on next for you.

Man, giving a talk remote is always hard.  Is my mic too soft?  Can they hear me?  Did some poor guy with a hearing aid just have it blown out ’cause I had my mic too loud?  Am I pacing ok?  What about handling questions?

Well, thanks to the expert help of Susie and Clint, I didn’t have to worry about any of this while giving my talk on Tuning Sugar Databases to SDUG.  Everything functioned smoothly and the only thing I had to worry about was pacing myself, which seemed to work out good. Hopefully those who attended picked up a few new tricks.  For those who didn’t get to attend live, we’ll have a recording of the webinar online soon.  I’ll update when we do.

I promised some links during the presentation, so here you go.  Our wiki is online @ http://www.sugarcrm.com/wiki/.  There’s excellent info throughout the wiki, but the settings I touched on specifically are all outlined in Performance Tweaks for Large Systems, except one.  Someone mentioned that there was no reference to $calculate_response_time on the wiki, that is correct and will be fixed.  In meantime, here’s a quick intro to it. Adding the following line to your config.php:

"calculate_response_time" = true,

Will get you output like:

The other links I mentioned where, in no particular order:

• MySQL Manual
• MySQL GUI Tools
• SugarCRM’s CTO Series

Also, for anyone who’s interested, the MySQL AB tool for automating many of the processes I outlined is MySQL Enterprise. If you’re short on people or expertise, outsourcing your DBA duties to the company that created MySQL is a great option.

Hope that helps get you started on tuning your SugarCRM installs on MySQL.  If you have any questions that we didn’t get covered in the presentation or Q & A, post them here and we’ll get you answers.

My good friend and cohort in CRM blogging, Martin Schneider, posted a review yesterday about a recent article from Psychology Today that analyzes why developers participate in open source communities. While I agree with Martin’s thoughts, I think the motivations go a bit deeper…right into the wallet.

First, let’s take the “street cred” idea a bit further. What I’ve seen in the Sugar community is that independent consultants and system integration firms can turn “street cred” into “business cred” or marketability and make some money. Simply put, companies hire you for projects if you are a credible Sugar community member. Also, IT developers NEED to have open source experience on their resumes these days to get the best jobs. Hiring somebody who actually contributes code to an open source project gives companies real competitive advantage for their business.

So being a prominent member of the Sugar community turns into actual dollars (Euros, kroners, Linden dollars, whatever). That’s a VERY powerful concept that is glossed over with the ticky tacky term “street cred”.

Now let’s take the “self-actualization” idea. Yeah, coders code because they like to code. Coding is cool. Believe it or not, one of my favorite ways of relaxing is to fix bugs in Sugar. Now call me wacko (my wife calls me other things), but Sugar is my baby and I want to see her not just sing, but friggen Rock the Cazbah (shout out to John’s current favorite band). So I certainly agree with the point on “self-actualization.”

However, the ability for a company, a project manager and/or a developer implementing SugarCRM to take control of their own destiny around the Sugar app is HUGE. I’ve project managed implementations of proprietary CRM systems in past lives. There’s nothing better at blowing up your project schedule than a showstopper bug or unexpected limitation in an API.

With open source, a company can fix the problem themselves if needed or extend an API to meet their needs. Now SugarCRM works very hard as a company to respond to all of our customer’s reported issues in a timely manner, but when your go-live date is looming ahead of you and your CEO demands no delays, there is nothing better than having the ability to fix any issues yourself and not be tied to the responsiveness of your vendor. And if you are an independent consultant or system integration firm, time = money.  Again, control is a very powerful aspect which drives developers to open source that goes way beyond “self-actualization” or even “altruism”. It drives sanity for the developer and bottom-line value for the business.

Personally, I think these two ideas of marketability and control should have been included in the Psychology Today as these are the motivations I hear about everyday from Sugar community members.  And marketability is about putting money in your wallet and control is about keeping money from flowing out of your wallet.