importing over 1 million records

[heebeha6]

I have over 400,000 contacts I need to impiort. I have tried importing it serverl times and dont get an error message but no records gert imported. I set tmy max upload to 99999999 butr still no good.

Any help here would be greatly appeciated.

[Skeeve]

Why not feed them directly to the database?

As far as I know you can't import more than 6000 records at a time.

[kbrill]

The only way you are going to get that much data into MySQL is to import it directly. The limit in 5.1 is more than 6,000 but no where near 1,000,000. There are a few tools made for importing data (navicat and jitterbit off the top of my head) but with the relationship data that needs to be created too you might want to get help with a job this big.

[roblaus]

Ken is right. There is no way importing this big number through Sugar directly. This is not necessarily Sugar's fault but you're violating all kinds of php limitations.
Doing it via the backdoor (directly into MySQL) isn't too complicated if you're careful - it's just the emails that may cause you some headache because you need to create the aforementioned relations. DO NOT try it on a production system!!

rgds

[kbrill]

Doing it via the backdoor (directly into MySQL) isn't too complicated if you're careful - it's just the emails that may cause you some headache because you need to create the aforementioned relations. DO NOT try it on a production system!!

rgds

All the modules have relationships that need to be created, contacts (the module he mentioned) needs to have a relationship with Accounts created in the accounts_contacts table most of the time. Just about any module you choose is going to have it's own special nuances to importing.

Although emails would be the hardest one I can think of.

[mrmmm]

I've run into similar problems in the past and another solution (besides going directly through the DB) is to separate your file into smaller chunks. I have figured out (by trial and error) the largest amount of records that I can safely import and just break up the file into those chunks. Obviously a rather time consuming task with a 1M records (if the max is at around 6K), but I am sure it is possible to hack up a small script that will divide your files.

[stevec]

Another tool to look at is Talend.

I'm not recommending this over any other product or process - it's just one I came across recently.

When I set up our system, I used navicat (a mysl gui front-end) but only because both jitterbit and talend were not available at that point. the latter two are data processing tools for heterogeneous systems so they are designed exactly for what you are doing.

However, you sill still need to understand the sugarcrm database relationships and structure. The two systems have sugarcrm modules but they might not be complete enough to do everything.