Importing from Comma Delimitted file.

[NPayette]

Hey I am trying to import from a file containing about 26k contact,
It`s been more then 30 minute and it`s still not finished, is that normal and also wouldn`t that be a good idea to give a Status update on the import...

For now it`s only telling me that it`s started but it doesnt give me a clue about where it is in the process...

Anyway, hoping for some help.

Thanks,
Nic.

[VeonConsulting]

Hi Nic,

We also faced such a problem while importing.
The best thing is, though some time consuming, divide the contacts into 9-10 parts and import one part at a time.

Regards,

[NPayette]

Hi Nic,

We also faced such a problem while importing.
The best thing is, though some time consuming, divide the contacts into 9-10 parts and import one part at a time.

Regards,

By any chance you didn`t develop a little tool to split one CSV file into multiple file containing X nb of row max?
Anyway thanks... But It`s not normal that it`s that slow...

Can anyone point me to the php file which is doing the import? I would like to take a peek at it

Thanks.

[sbantz]

This won't help you much, but I think there is a maximum amount of time the process will run before it just stalls. There is a file you can edit to increase this timeout. Search the forums a bit and you may find what I am talking about. I know I read it a few days go.

I like the idea of splitting up your imports. Mine took < minute to import 1350 records.

[NPayette]

This won't help you much, but I think there is a maximum amount of time the process will run before it just stalls. There is a file you can edit to increase this timeout. Search the forums a bit and you may find what I am talking about. I know I read it a few days go.

I like the idea of splitting up your imports. Mine took < minute to import 1350 records.

Yeah I like the idea too, to problem is that for the initial integration of SugarCRM which is going to happen in the upcoming days, I will have to import over 2 million Contact or Account in a timely matter, but if I split each CSV file into 1000 records one it's just going to make the manual import impossible and I will have to develop my own importer (Which I will have to anyway...)

Anybody knows of anyone who developed a custom FAST direct importer made for VERY LARGE import?

[Angel]

Anybody knows of anyone who developed a custom FAST direct importer made for VERY LARGE import?

I've built a few before for various needs. I really don't like messing with the built-in import utility for large volumes of data.

If you are considering building your own, you could use the SOAP API to do it and not have to worry about knowing the structure in Sugar. See the Wiki for further info on the SOAP API: http://www.sugarcrm.com/wiki

I should forewarn you that it (SOAP API) is not the speediest thing in the world.

If you are .NET developer, you could also get a copy of this library (http://www.sugarforge.org/projects/guiddll/)that I built which is essentially a port of the PHP function that Sugar uses to generate the unique IDs within the Sugar tables. You could then use that to create your own utility that writes directly to the database with appropriately formatted ID values.

[NPayette]

I've built a few before for various needs. I really don't like messing with the built-in import utility for large volumes of data.

If you are considering building your own, you could use the SOAP API to do it and not have to worry about knowing the structure in Sugar. See the Wiki for further info on the SOAP API: http://www.sugarcrm.com/wiki

I should forewarn you that it (SOAP API) is not the speediest thing in the world.

If you are .NET developer, you could also get a copy of this library (http://www.sugarforge.org/projects/guiddll/)that I built which is essentially a port of the PHP function that Sugar uses to generate the unique IDs within the Sugar tables. You could then use that to create your own utility that writes directly to the database with appropriately formatted ID values.

That my friend is a great news as I am building the Sync/Import project in C#, do you have any other usefull .Net Library I may want to take a peek at?

I will release my Comma Delimiter custom importer to you after

In fact my current project is all about filling the future SugarCRM with current contact/account we have on 3 different server/database.

Then use the same importer and integrate it in a Sync Services that will monitor each entity for new or edited contact/account and keep them all Sync (With a DataEntry team that will verify duplicate data with different set of data)

It will have to be able to read different source (CSV, XML, etc.), parse and map it to good field. I"m pretty sure some people have the same kind of need somewhere so I will release anything Non-Company specific

Thanks for future help or idea on my project.

[baigeman]

Import Speed Increase

I was just having an issue with importing Leads that got resolved here: http://www.sugarcrm.com/forums/showthread.php?p=93065

Since then I noticed a MAJOR MAJOR MAJOR change in importation when I changed to importing the salutation, first name, and last name separately versus importing the full name. I also got rid of all of the "empty" rows in my .csv file that were created by the program I exported from.

This changed my upload time from over 15 minutes to INSTANT! for 41 leads on an old slow server

So if you are having problems try playing with what data you are feeding Sugar. If anyone out there knows more about what data issues will bog down Sugar please share.

Hope this helps

[cerhorn]

I use the Pentaho open source ETL tool (kettle)that works very well and is relatively easy to use. It is far less complex than Talend (another open source integration tool) and you can be functional with Pentaho in about 4 hours with the extensive user manual.

[GameboyRMH]

As baigeman said import the name using separate fields, this will give you better results as well.

You can change your maximum execution time in php.ini (max_execution_time = ##)

You should also see some dots moving across your screen if the import is progressing (depending on your version).

And a data issue that will bog down sugar - in 5RC, possibly earlier versions as well, if you're importing an Account and you try to map anything to SugarCRM's parent ID field, the import will hang. So sadly the only way around it is to lose your account parent/child relationships I reported the bug and it should be fixed in v5 GA though.