Importing of leads fails with memory exhaustion

[GaryWayneSmith]

Hello,

We have a csv export from an access database that holds about 10k leads. These are pretty much direct mail leads. When trying to import them, Apache dies with an error about memory exhaustion. The error specifically happens when memory gets to 100mb. I've looked through a couple of the existing threads and they recommend tweaking memory and cutting down on the number of records imported. I've tweaked out php.ini to allow for 300mb, but still it dies at 100mb. I've tried pruning it down into batches of 1000 messages and it still fails, batches of 500 and have had mixes results.

Is it possible to do a import directly into MySQL via SQL? Someone mentioned in a thread that the ID field must be created through SugarCRM but since these are new records and don't have a reference as of yet, I don't understand why this would be, when it comes to leads that is.

What's the best approach to resolve this problem?

[Angel]

Hello,

We have a csv export from an access database that holds about 10k leads. These are pretty much direct mail leads. When trying to import them, Apache dies with an error about memory exhaustion. The error specifically happens when memory gets to 100mb. I've looked through a couple of the existing threads and they recommend tweaking memory and cutting down on the number of records imported. I've tweaked out php.ini to allow for 300mb, but still it dies at 100mb. I've tried pruning it down into batches of 1000 messages and it still fails, batches of 500 and have had mixes results.

Did you reboot Apache after modifying your PHP.INI file?

Is it possible to do a import directly into MySQL via SQL? Someone mentioned in a thread that the ID field must be created through SugarCRM but since these are new records and don't have a reference as of yet, I don't understand why this would be, when it comes to leads that is.

The ID value is the unique identifier within the table, thus you cannot simply ignore it. If you do, you wouldn't be able to import more than one lead because the second one with a blank ID value would violate the restriction that the value has to be unique.

If you must import via external tools, you can create the ID for each row ahead of time using the create_guid() function in utils.php.

Another option would be to leverage the SOAP API to insert the data into Sugar. The API would handle the ID portion for you.

[GaryWayneSmith]

Did you reboot Apache after modifying your PHP.INI file?

The ID value is the unique identifier within the table, thus you cannot simply ignore it. If you do, you wouldn't be able to import more than one lead because the second one with a blank ID value would violate the restriction that the value has to be unique.

If you must import via external tools, you can create the ID for each row ahead of time using the create_guid() function in utils.php.

Another option would be to leverage the SOAP API to insert the data into Sugar. The API would handle the ID portion for you.

Angel,

Thanks for the follow up.

I tried several things, I showdown the entire Xen instance, allocated 2GB of ram, etc, still no luck. What I did, since I'm pretty fluent at sql, was to create a cusotm insert statement batch and imported them then all directly to mysql. I know that's not supported. I used a unique, incrementing sequence, and didn't find any ill effects thus far.

Gary

[sacramentojoe]

Angel,

Thanks for the follow up.

I tried several things, I showdown the entire Xen instance, allocated 2GB of ram, etc, still no luck. What I did, since I'm pretty fluent at sql, was to create a cusotm insert statement batch and imported them then all directly to mysql. I know that's not supported. I used a unique, incrementing sequence, and didn't find any ill effects thus far.

Gary

Having the same issue.
Can you share your script?