SugarCRM Quality Assurance, Does it exist?

[alsutton]

One week after 4.5.0 is out the door and we already have a patch fixing things like;

- The system displays an error message when you attempt to open an email attachment.
- The get/post/request parameters have various security issues.
- Sometimes, you cannot log into Sugar Suite through the Sugar Plug-in for Microsoft Outlook.

(and the patch notification started off by saying 4.5.0 in the header and 4.2.1 in the body?!?!?!?!).

So what do we think?, Does SugarCRM have an internal QA group or does it rely on unpaid beta testers who may not test all the corners of the system.... I know what I think...

[longreach]

I think this is too harsh an assessment - and that the mitigating factor is the size of the Sugar app - it is getting to be a very large piece of code, with navigation pathways that produce a crazy number of use cases. Release 4.5 in particular amounted to a re-architecture of the system - as internationlization and new UI touched almost everywhere.

Do they release too quickly after fixing what they think are the final bugs - yes, for sure. They should have at least another week in there of quiet from the testers before release. Do they knowingly use the OS user population as continued agents for QA after a GA release - yes, for sure. But then - it seems a reasonable contribution as you are not paying for the code.

Personally I have always adopted a policy of not using their latest releases until about a month has passed of general release. And I think anyone that uses the latest Sugar code for production within a month of release is not using good judgment. But I think you should have a testing install during that time - to help get the software to where it needs to be.

Do other vendors have better quality on initial releases. Yes. But they also don't have open forums that make their messes quite so public. Nor are they advancing their product so quickly as Sugar has in the last 2 years. Does Sugar need to slow down their releases even more as they mature - yes. Do they need to do something about the mess that is email - yes, for sure. But are they also making huge positive strides in QA with each release - yes, for sure.

[Scott_Savage]

Yeah I agree. They do test it on the community a bit, but they also interact a lot and seem very open to suggestions. It is clear they read our feedback and I definitely think they are headed in the right direction. I think it is a bit harsh to slam them when they release a new product, they are being ambitious with their development speed and they are essentially producing a very good core product, the edges just need some trimming and bug consolidation (along with some more attention to email).

[ruchida]

Good point.

We all know there is no perfect QA. If you like perfect QA, you need to understand no product will be released forever.

If you like more and longer QA, you need to pay more to Sugar to hire additional QA staff, and you need to go back to the expensive "proprietary era".

Sugar Team needs to achieve many feature requests submitted by community members, and at the same time needs to keep a reasonable price. That means we need to share load, and each of us needs to find out how to share load.

Like QA, there is no perfect "how-to-share-load". You need to find it by yourself, and I need to find it too.

[alsutton]

I would accept rough edges, or the odd small bug, but these seem like some howlers to come up in the first week, and there are other bugs which don't seem to get addressed (sort order for the My inbox being one example that bugs the hell out of me).

I've said this several times before; I'm sure that Sugar would get more customers if they stopped trying to play the cool guy and started stabalising things, because at the end of the day looks are nice, but a reliable piece of software is nicer.

Look at the web sites that have succeeded as a model for whats good, you've got things like main sites of eBay, Amazon, and Google, all of which don't have an all singing all dancing user interface, but what they do what they do quickly and reliably.

So if the QA department is overloaded, then Isay cut down the functionality. After all I don't want an AJAX-ed to field in the email section that predicts who wants to send an email to and pre-fills the field when I can't view an embedded screen shot in an incoming mail.

[brendanwalsh]

I'd go with - please clear out some of the longstanding bugs...there is a danger that anything under Urgent isn't getting fixed. And I'd go with - please sort out email.

But 4.5 is strong. The UI needed the move forward - not for show but for useage.

It feels to me that the way Sugar is priced (or not priced for the OS) and the way they go about being open that it serves no purpose to yell harsh words. Better to positively contribute in the forums, raise the bugs, and save the tough comment for when it is due. Without wishing to sound pompous the trick with Sugar appears to be feel more like "we" than jabbing at "them".

And don't use x.0 from any organisation. Just load it up, run it and feedback. Remember Windows 3.0? ;-)

Brendan

[alsutton]

Brendan,

I'm normally a rational, easy going person, but I've suffered for a long time with SugarCRM, and it's not only worn my patience thin, it's also dented my sanity on occasions. I feel that putting the UI before the functionality is like trying to put a ferrari body a cheap kit-car, you can make it look as good as you want, but when people start to use it they'll see the problems, and you'll end up loosing more sales than you gain. I've been told that a person who has a good experience tells, on average, 3 people. A person who has a bad experience tells 10. At the moment web based Email, my main area I have to deal with, is definatley a bad experience.

I've done the discussing in forums, raising bugs, etc., etc., and seen very little happen. You'll see from my very first post that incoming email handling has been a big thing for me, and here we are, over a year later, with promise after promise about incoming Email handling, and we still have a built-in mail reader which can't display in-email images in the way that some of even the most basic email readers can.

I've had patches from SugarCRM which simply didn't work, used alternatives that now don't work because the developer couldn't keep up with Sugars constant upgrades, and even had patches from SugarCRM employees which didn't get included in later releases.

All in all it's not been a fun ride.

[brendanwalsh]

I guess we would agree that you wouldn't use Sugar if the most important thing to you was web-based email. It is of secondary importance to me and our clients.

Sounds like its time for a cool beer and a reassessment. This poll aint going to get your email bugs fixed ;-)

Brendan

[synchro]

Something that struck me by its absence right from the start - there is no test suite for Sugar, at least not in the OS edition. This, combined with comments elsewhere about simple things like lax coding standards and swathes of invalid HTML give the impression that development is very much a seat of the pants affair, so breakages like this seem inevitable. The practice of 'blanking' chunks of code that differentiate OS, pro and enterprise versions seems truly bizarre and smacks of poor strategy decisions and maintainability nightmares. I don't deny that it's hard to do unit testing on web pages, but it is possible, and core functionality should be in the model, not in page controllers. Rapid feature turnover combined with a lack of test driven development means we'll see new stuff quickly, but we can't expect reliability. Though many can live with this situation (I am), and Sugar is still a great product, it's a very risky strategy in the long term.

[alsutton]

One thing I'd disagree with is where you say "It's difficult to unit test web pages". We produce a web based enterprise password safe which uses JUnit and HTTPUnit to test paths through the application. It takes, on average, 10 to 20 minutes to write and test a test, and we enforce that every code change that goes in must be covered by an existing or new unit test that uses the web UI and not just a unit test which exercises the code in question.

I suspect SugarCRM has no unit tests for the web UI, so there would be a chunk of effort to get the ball rolling, but once it is rolling it will save Sugar from the embarassment they currently seem to have to go through every time they release a new version or patch.