How can I make bounce handling work on CE 5.5.0?

[blasphemonium]

Hello

I am trying to use email campaigns but now bounce handling doesn't seem to work properly.

Emails are already going out properly, and my bounced emails are being received in my bounce email account (I directly checked through webmail), however Sugar doesn't associate bounced messages with my campaign.

Schedulers are running and show to have been succesfully executed....

I have seen a lot of threads but most of them refer to former Sugar editions so I wouldn't really want to change any settings to my system not being sure about possible consequences.

Do you have an idea of possible reasons which make my bounce handling not to work properly?

I use:
Sugar CE 5.5.0 (Build 6655)
Linux (CentOS 5.4)
MySQL 5
PHP 5

Thanks in advance for any assistance you could give me!!

[sugarcane]

Were you previously on another version of Sugar where this was working? We have noticed in the past on 5x versions that the system will only record the bounced emails within the CRM when the Username for the Bounce account is the same as the From name. We have not yet tested this on 5.5.

Other things you might want to check: by default, the scheduler item for bounced emails only runs from 2-6AM, so you want to make sure that this item is both activated and maybe expand the time in which it runs. Otherwise, you will see delays between when the emails actually bounce and when the bounced emails are recorded within the system.

[blasphemonium]

Thanks for your feedback sugarcane.

This is the first version of Sugar that I am testing.

According with your suggestion, I set "From Name" and even "From Address" same as bouce account username (in my case campaigns@ebc-hq.com).

campaigns@ebc-hq.com mailbox received bounced email as I could check it from webmail, however Sugar didn't report it. In fact "Process Bounced Campaigns Emails" ran "successfully" for 2 times....

Regarding your other hint, scheduler was initially configured to run as originally every hour from 2-6am, however bounce email was received at 2:05am, so it should detect it at 3:00am.

I have read some threads related to trackers, but I am not sure if these bugs were solved with 5.5 release.

Any other ideas?

Thanks in advance!

[Bloom_IT]

Hi
Sugar can only detect an email as bounce if:

• The original email contains a remove me link (the link that contains a tracker identifier)
• The bounced message body also contains the remove me link

The tracker is a unique id that is assigned to any email sent by a sugar campaign and is unique. This way, when one of the targets performs an action (like clicking on the remove me link or clicking on any other tracker link), sugar knows which user performed that action.

When an email bounces back, sugar needs that the bounced back email has the remove me tracker so it can find out which target did not receive the email
What happens is that usually the bounced emails have only the email header and not the body. Therefore, sugar can’t know which target failed to receive the email and just discards that email.

So your bounced email content should look like
================================================== ====
Delivery to the following recipient failed permanently:

ghost@failed_delivery_domain.com

Technical details of permanent failure:
DNS Error: DNS server returned answer with no data

----- Original message -----

Received: by 10.216.163.7 with SMTP id z7mr1538727wek.123.1271414202336;
Fri, 16 Apr 2010 03:36:42 -0700 (PDT)
Return-Path: <no-reply@xxxxx.com>
Received: from xxxxxxxxxx
by xxxx with ESMTPS id xxxx
(version=TLSv1/SSLv3 cipher=RC4-MD5);
Fri, 16 Apr 2010 03:36:39 -0700 (PDT)
Date: Fri, 16 Apr 2010 13:35:00 +0300
Return-Path: no-reply@xxxxx.com
To: Ghost Rider <ghost@failed_delivery_domain.com>
From: no-reply@xxxxxx.com
Reply-to:
Subject: [Test]: Newsletter 05
Message-ID: <9a9d09739ee756170d34160ed043ade3@xxxxxxx.com>
X-Priority: 3
X-Mailer: PHPMailer (phpmailer.codeworxtech.com) [version 2.3]
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="b1_9a9d09739ee756170d34160ed043ade3"

----- End of message -----
================================================== =====================

To solve this, I implemented a solution that writes the sugar tracker directly into the email header. Since headers are usually always available on the body of the bounce message, you just need to search for it.
To implement the work around, follow these steps:

1) edit ..\sugarcrm\modules\EmailMan\EmailMan.php and add the bold line

$mail->ClearAllRecipients();
$mail->ClearReplyTos();
$mail->Sender = $this->mailbox_from_addr;
$mail->From = $this->mailbox_from_addr;
$mail->FromName = $this->current_emailmarketing->from_name;

// I added the following line to get target_tracker_key into the Header of the EMail
$mail->AddCustomHeader("X-SugarCRM-TrackingID:".$this->target_tracker_key)



2) edit ..\sugarcrm\modules\Campaigns\ProcessBouncedEmails .php and change the lines below to the ones shown in bold

//do we have the identifier tag in the email?

$email_description=quoted_printable_decode($email_ description);
$matches=array();
if (preg_match('/index.php\?entryPoint=removeme&identifier=[a-z0-9\-]*/',$email_description,$matches)) {

$identifiers=preg_split('/index.php\?entryPoint=removeme&identifier=/',$matches[0],-1,PREG_SPLIT_NO_EMPTY);

CHANGE TO

//commented this line because the quoted_printable_decode was messing with the remove-me id and bouncing failed because of that
//$email_description=quoted_printable_decode($email_ description);
$matches=array();
if (preg_match('/X-SugarCRM-TrackingID: [a-z0-9\-]*/',$email_description,$matches)) {

$identifiers=preg_split('/X-SugarCRM-TrackingID: /',$matches[0],-1,PREG_SPLIT_NO_EMPTY);

3) additionally, you can enhance the detection of invalid email address by changing the line below to the one shown in bold

//do we have the phrase permanent error in the email body.
if (preg_match('/permanent[ ]*error/',$email_description)) {

CHANGE TO

//do we have the phrase permanent error in the email body.
if (preg_match('/permanent[ ]*error|dns[ ]*error|permanent[ ]*failure/i',$email_description)) {



Your bounced emails should now look like

================================================== ====
Delivery to the following recipient failed permanently:

ghost@failed_delivery_domain.com

Technical details of permanent failure:
DNS Error: DNS server returned answer with no data

----- Original message -----

Received: by 10.216.163.7 with SMTP id z7mr1538727wek.123.1271414202336;
Fri, 16 Apr 2010 03:36:42 -0700 (PDT)
Return-Path: <no-reply@xxxxx.com>
Received: from xxxxxxxxxx
by xxxx with ESMTPS id xxxx
(version=TLSv1/SSLv3 cipher=RC4-MD5);
Fri, 16 Apr 2010 03:36:39 -0700 (PDT)
Date: Fri, 16 Apr 2010 13:35:00 +0300
Return-Path: no-reply@xxxxx.com
To: Ghost Rider <ghost@failed_delivery_domain.com>
From: no-reply@xxxxxx.com
Reply-to:
Subject: [Test]: Newsletter 05
Message-ID: <9a9d09739ee756170d34160ed043ade3@xxxxxxx.com>
X-Priority: 3
X-Mailer: PHPMailer (phpmailer.codeworxtech.com) [version 2.3]
X-SugarCRM-TrackingID: 3b3b0b45-0e20-f230-bc29-4bc83d69d17c
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="b1_9a9d09739ee756170d34160ed043ade3"

----- End of message -----
================================================== =====================

And sugar should detect the bounced emails

This is working perfectly with Sugar 5.5.1. Just remember that it is not upgrade safe and every time there is a new sugar release you will have to perform these code changes.

Cheers

[blasphemonium]

Hello Bloom_IT

Thank you so much for your feedback.

I was trying a similar procedure which I got from this post: http://www.sugarcrm.com/forums/showt...emoveme&page=8
but for some reason ID was not being included in message header.

I followed your instructions to change EmailMan.php and in fact now it is including the TrackingID within message headers, however it is being included more than once:

Message-ID: <8e23c2de1ae87ecd9c9e70e3b312f767@node01.tmdhostin g210.com>
X-Priority: 3
X-Mailer: PHPMailer (phpmailer.codeworxtech.com) [version 2.3]
X-SugarCRM-TrackingID: 9608030b-9684-41cc-3f5d-4bc89718f295
X-SugarCRM-TrackingID: 9077024d-922e-b436-a029-4bc897580ff3
X-SugarCRM-TrackingID: 6bccbd47-4b65-fbbc-8511-4bc8976a9ec6
X-SugarCRM-TrackingID: c5190e81-25fc-b511-b3f5-4bc89753f4a8
X-SugarCRM-TrackingID: a92f28aa-a535-a553-7616-4bc8979c2ca2
X-SugarCRM-TrackingID: 1608e279-7e7a-4814-5de3-4bc8974d98fb
X-SugarCRM-TrackingID: 71e77859-04ab-3647-0a5e-4bc89746f8f2
X-SugarCRM-TrackingID: e093a32b-940c-0de0-adf7-4bc897dacba7
X-SugarCRM-TrackingID: 82bf42d2-21db-6b34-b51b-4bc8979ad130
X-SugarCRM-TrackingID: e3e4ddc7-a9be-7b2f-6cc7-4bc897b17d3d
X-SugarCRM-TrackingID: a630eaa4-02eb-1edf-3303-4bc897ba1b8d
X-SugarCRM-TrackingID: ec84f92f-079c-aff6-6a65-4bc8973c40ca
X-SugarCRM-TrackingID: aea851c2-b8a8-8de3-8cb5-4bc897db0607
X-SugarCRM-TrackingID: 402157bf-d71e-2a55-2f24-4bc8977a8981

From the other post I found another line which was included before the AddCustomerHeader statement. This line is following:

$mail->ClearCustomHeaders();

Could it be related with including only one tracking ID for each message?

or maybe

Is it normal that all these tracking ID headers be included ih messages?

Thanks in advance for your help!

[blasphemonium]

Hello

I followed instructions but I have the following questions:

1. TrackingIDs are now being included in messages, however I am not sure if they are being included in header or body. When I check a bounce message in MS Outlook I can see my TrackingID (Right click -> Options -> Internet Headings).
However, when I click "Show headers" from Sugar email client it just tries to open a new window but nothing is shown.
When I click "Show email source" I can see first a list of headers like this:

Return-path: <>
Envelope-to: crm@ebc-hq.com
Delivery-date: Fri, 16 Apr 2010 14:30:53 -0500
Received: from mail.networksolutionsemail.com ([205.178.146.50])
by node01.tmdhosting210.com with smtp (Exim 4.69)
....more come here....

Then I can see the error descriptions which is the one you can see in your bounce message body

And finally I see another "headers" block with some similar information like the first one, however this block is including my TrackingID information.

Does it mean that I need to make Sugar to save emails raw source to find TrackingID?

2. In fact I tried to make Sugar save emails raw source, but when I check the sugar log I find the following entry:

Fri Apr 16 14:00:05 2010 [30661][1][DEBUG] *** INBOUNDEMAIL not saving raw due to system settings

Does it mean that my check bounces scheduler is not being able to work because of raw sources are not being saved?

3. A stupid question but I didn't find this anywhere:
I have 2 email accounts: one for inbound which is associated with a group folder, and another account for bounce checking but is not associated with any folders. Do I need to add a folder so that bounces be scanned or just with setting up the account it can work?

I am really confused about it. Any help is very welcome!

[Bloom_IT]

Hi again.

So, going over your questions one by one:

1) From the other post I found another line which was included before the AddCustomerHeader statement. This line is following:

$mail->ClearCustomHeaders();

Well, I found no use for this sentence and since I can’t be sure if any customer header has been added in some other part of the code, I decided not to clear it.

2) Is it normal that all these tracking ID headers be included in messages?

Not at all! And since the tracking ids are all different, there must be something wrong in your code. Have you rechecked it?

3) Does it mean that I need to make Sugar to save emails raw source to find TrackingID?

When emails are sent, the tracking Id is only part of the email header and therefore only visible when you open it with, for instance, outlook (Right click -> Options -> Internet Headings).
When the email bounces back, the original header should be available on the body of the bounced email. Of course your email provider could, in theory, have just a simple message without the original headers being shown in the body but I have never seen that happen.
Bottom line, if you can read the X-SugarCRM-TrackingID: xxxx in the bounced message, it is being written to the body. I would trust on your regular email client (outlook or the web interface) but I would not pay much attention to what is shown on Sugar email client (personally cause I can’t be sure if it is works properly =P )

4) Does it mean that my check bounces scheduler is not being able to work because of raw sources are not being saved?

Before coming to this solution, I tried the approach described on the other thread with the raw emails but never got the raw source being saved. The emails were detected and written into the “emails_text” table (raw_source column was always empty).
The scheduler should be working and the emails being loaded by sugar because you have that entry in the log. Whether it is saved as raw or not does not affects the how the scheduler works.

3. I have 2 email accounts …
Not followed your question. I have this working with the following configuration:
> 1 email account of type “group” and “Bounce Handling”
> Additional configs as follow:



When sending my campaign emails I use this (and other two accounts). Remember that the account that you use to send the email must be of type “Bounce Handling” and you must have configured the “inbox” folder to be monitored for bounces.

Does this clarify your questions?

Cheers

[blasphemonium]

I am going insane now. I can not make it work

Well, I discarted problems with schedulers because I checked the log and it is working.
In fact when I check the bounce account through webmail I find that bounced emails were marked as read, but those which are not bounced remain as unread.

From the log I also found these entries:

*********** NO duplicate found, continuing with processing.
*********** Importing RAW email.
...
InboundEmail found multipart email - saving attachments if found.
Query:INSERT INTO emails_text (email_id, from_addr, to_addrs, cc_addrs, bcc_addrs, reply_to_addr, description, description_html, raw_source, deleted) VALUES('96a9.........

Which make me thin that sugar is retrieving information from bounced emails, and in fact for the "raw_source" variable, retrieved value is:

.......\nX-SugarCRM-TrackingID: f41cb267-82f9-056d-74c0-4bc9c5457695

Bounced emails are read, and trackingID is also read, however few lines later in log file I found this entry:

Warning: skipping bounced email because it does not have the removeme link.

So now I am really desperated. Any body has got an idea on how to make it work?

Thanks in advance for any help.

[blasphemonium]

Now I am checking ProcessBouncedEmails.php file, but unfortunatelly I don't have any idea about PHP. However I found something interesting:

When function begins, variables are declared:

function campaign_process_bounced_emails(&$email, &$email_header) {
global $sugar_config;
$emailFromAddress = $email_header->fromaddress;
$email_description = $email->description;

As you can see $email_description was assigned value from certain thing called "$email", and something stored inside it named "description".

The initial file included this line, which I commented according with forum:

//if (preg_match('/index.php\?entryPoint=removeme&identifier=[a-z0-9\-]*/',$email_description,$matches)) {

As you can see it was calling this variable, but now my line looks like this:

if (preg_match('/X-SugarCRM-TrackingID: [a-z0-9\-]*/',$raw_source,$matches)) {

Which is reading variable $raw_source, but it was not initialized above like $email_description.

Could it be the reason that sugar is not being able to read the trackerID?
How could I initialize the $raw_source variable in case it is what is causing the problem?

I checked beyond this if, and else option is:

} else {
$GLOBALS['log']->info("Warning: skipping bounced email because it does not have the removeme link.");

which is the same error message that I am getting from log file :O

Which makes me think that the problem is really related with the former if.

Thanks in advance for any assistance

[Bloom_IT]

Hi

Sorry for the delay but it has been a busy time over here. I'm short on time but i can try two help you and i'll provide 3 quick updates.

1) the ClearCustomHeaders function is actually needed. I just realize that a few days ago

2) can you tell me witch email server are you using?

3) can you upload somewhere your ProcessBouncedEmails.php and EmailMan.php so i can run a diff and check what are the differences? Maybe something else is missing

Cheers