Problem with character sets when importing

[itso]

Hi all,

I have just installed 5.1 on a shared host with a typical install - everything is UFT-8. I created an account whose name contains Cyrillic characters. Everything shows up great on the screen. Then I export this account, delete it from the system and try to import it again. Everything works well, I can see the correct name in the preview - the last step before importing. After the import, however, the cyrillic characters in the name get messed up. Since this file is produced by the same installation, I am certain that the output and input codec is correct - UTF-8, besides I have checked visually in the user profile.

My only explanation is that during the import action, Sugar tansforms the characters in some other character set and the result is that non-UFT-8 is written in mysql. I have even looked inside the accounts table. So I think that this is a bug. I have tried this in Windows XP and Ubuntu on that shared host and also in a local installation on xampp. You can see the phpinfo at:
http://mmdpartners.com/sugar51/phpinfo.php

Could someone please help me. This is very weird. I am fairly technical but still an enthusiast - not a professional - and any ideas are appreciated.

Greetings from Bulgaria.

-Itso

[clint]

Hello Itso -

Click on My Account and take a look at the "Import/Export Character Set:" setting. Is that also UTF8?

Clint

[itso]

Hi Clint,

Yes, it is UTF-8. I do an export from this installation of one account. I do not change anything. Then I try to import it and it comes in with jumbled characters. I am doing this with the admin user. No others have been created yet. I want to try this out before fully committing to 5.1.

Do you need any other diagnostic info. you can download the diagnostic file from:
http://www.mmdpartners.com/sugar51/diagnostic1.zip

-i

[roblaus]

Did you check the import details (underneath the mappings, you need to open it)? Try to change the character set there.

[itso]

could you please tell me where to look and what exactly... I have the source code downloaded, I could also search a dump of the database.

please note that I am trying to export and import on the same installation, so encodings should not matter. But still cyrillic characters get jumbled.

Thank you for your help/
-i

[roblaus]

Not so complicated. Third page of the import wizard, where you set the mappings. See attached.
Hope it helps

[itso]

Thank you, Robert,

I just noticed from your response that the import screen has advanced settings. Of course they were the problem. For some reason, they by default assume a CP1252 character set for my CSV file. Why is this the default value here if I have set everything in Sugar to UTF? Can't it just take that and not show initiative ? I guess this allows for some flexibility if one cannot reproduce the export file in the desired char set. Still, the lack of consistency is confusing, especially since the advanced import options are hidden behind a button.

Once I put UTF-8 there everything works.

Thank you very much for your help.

-Hristo Stefanov

[clint]

Thank you, Robert,

I just noticed from your response that the import screen has advanced settings. Of course they were the problem. For some reason, they by default assume a CP1252 character set for my CSV file. Why is this the default value here if I have set everything in Sugar to UTF? Can't it just take that and not show initiative ? I guess this allows for some flexibility if one cannot reproduce the export file in the desired char set. Still, the lack of consistency is confusing, especially since the advanced import options are hidden behind a button.

Once I put UTF-8 there everything works.

Thank you very much for your help.

-Hristo Stefanov

Bug 25927 filed to track this issue.

[jyim]

FYI, Bug 25927 was fixed in the 5.1.0c release.