First impression from a fellow opensource guy

[nant]

Greetings from the Joomla / Community Builder team at Joomlapolis.com !

We have been experimenting using Sugarcrm / CE for our 300K+ membership on joomlapolis.com.

The goal was to put sugarcrm in place and slowly prepare integrations with our core Joomla/Community Builder environment (also PHP / MYSQL based).

We started by installing version 5.5 on a dedicated powerful machine.

We then tried to import our 300K exported member records to Sugar initially as targets in order to prepare an email marketing campaign celebrating our 300K member.

Well as most of the real large users can most likely agree (unless we have done something very wrong), the import process is a bitch. We had to continuously split import files again and again because things would crash in the middle of the import process for no apparent reason and with no logging of errors. We found that we could not even attempt uploads of more than 30K records no matter what resources we gave to our powerful dedicated machine. It also appeared as if at the end of a successful import the system would try to run some really nasty SQL scripts that basically brought down the entire system and made it impossible to perform another import without prior killing of these nasty scripts. Well after a 15 hours of struggling we managed to finish this import process.

So now we had close to 300K targets in Sugar.

We had previously tested the email campaign with sample test target lists and we felt that the functionality was good enough for our purposes.

Now we needed to assign all these imported targets to a target list. Here we fell into an existing bug (I search the forums and found many discussions about this) where one cannot just select all targets or all targets of a query and put them in a target list. It appears that only a page worth at a time could be added - making things impossible for the size we were dealing with. Increasing the page list size from 20 to 10000 was not really a solution (in order to select more than 20 records at a time) because of all the javascript involved - many browsers would just hang (even safari and chrome). Also, the id columns used made it impossible to directly manipulate the database to overcome this bad UI.

At some point we managed to assign 100K targets of the 300K imported ones to our campaign target list. We then decided to start the sending process.We did not mange to send more than 3000 emails. No real error logging existed so I cannot say what failed. It looks like something was failing in some PHP code but was never properly logged and the cron processes just kept on doing nothing.

We have more technical information available should any Sugar developer want to look closer at this situation.

I believe that should some of the SUgar developers have the time to look closer at our specific environment, then perhaps many usability issues could be addressed (we could help also as much as possible).

I know that development teams are usually swamped and tend to ignore such posts - so no real disappointment if it remains unanswered and if no one contacts us. We really wanted to us Sugar but it looks like it is not really targeted for large databases and mass email campaigns which is what we need.

Thank you for your time reading this rather long post.
Nick (aka nant) from Community Builder Team at Joomlapolis.com

[eggsurplus]

Just in case anyone else runs into the same issues the workaround which I've needed and still do is to do a direct db import for large record sets. Then to add them to the target list I've borrowed from the PowerProspecting idea and added a Target List field to the Mass Update area so that a very large number of contacts can be moved into a target list at once. An alternative to that again is to do a direct db query to move them into a target list.

Not ideal but a work around. I agree though, on the popup select there needs to be a Select->All Records option like on the List View. No reason these areas can't be using the same/similar code on the back end anymore.

[ruchida]

things would crash in the middle of the import process for no apparent reason

This might be caused by lack of memory. How much memory is allocated to PHP?

The number - 300K does not seem to be realistic in a general mamory usage.

[nant]

@Both: Some extra information

> I agree though, on the popup select there needs to be a Select->All Records option like on the List View

It is there, but selects only displayed page.

> How much memory is allocated to PHP?

1.5 GigaBytes for PHP max_mem allows importing only 10,000 records in a page at a time!

(this dedicated server being used only by me at that time)

Browser-side: Safari 4 mac on 8 Gigabytes RAM


As you can see the machine has the resources - the code is failing.

I am really hoping the development team has some time to take a closer look at this (I find that debugging in an environment that illustrates the problem can save much time in the long run).

[ruchida]

1.5 GigaBytes for PHP max_mem

Hi,

I think 1.5GB is too small for for SugarCRM to import 300K records.

[nant]

Hi,

I think 1.5GB is too small for for SugarCRM to import 300K records.

Well as I said with this memory only 10K imports were possible.

Anyone care to comment on the other issues encountered?

[eggsurplus]

> I agree though, on the popup select there needs to be a Select->All Records option like on the List View

It is there, but selects only displayed page.

That option only selects the current page. What it needs is an additional option like on the List View where you specifically ask for all records and not just the current page.

Honestly, 300k records is a huge amount to expect to be able to move around through the front end. I've had issues with a number of apps trying to work with that large of data sets. This isn't Sugar specific. Data set sizes like this is best managed behind the scenes.

[nant]

That option only selects the current page. What it needs is an additional option like on the List View where you specifically ask for all records and not just the current page.

Honestly, 300k records is a huge amount to expect to be able to move around through the front end. I've had issues with a number of apps trying to work with that large of data sets. This isn't Sugar specific. Data set sizes like this is best managed behind the scenes.

Agree with missing option.

Forget the 300K import issue. I am sure there are many performance issues that could be fixed to get the import process to be able to handle larger files, but that's for the developers to investigate.

The mass email issues are real and have not been address. Has anyone had success with email campaigning of some large size, e.g., 50K?

[SugarDev.net]

I assume as Joomla community leaders, you are used to programming PHP?

[sugarcane]

Has anyone had success with email campaigning of some large size, e.g., 50K?

Absolutely. We have many clients who send out much larger campaigns than this. The campaign batches and speed rely on a few factors:
-Scheduler being properly set up
-Setting for '# Campaign emails per batch'
-Email Server limitations

Many times, we find that campaigns fail because the email server is not configured to handle such large amounts of volume in a shorter period of time.

FYI - we do provide email outbound SMTP settings as a solution to our clients for the purposes of sending out campaign emails. Please feel free to contact us if this is something you are interested in.