Sharepoint workflow - The email message cannot be sent

The issue really got me big time. I spent a long time trying to figure out what went wrong so I am hoping this post can help you. 

Our Expense Claim workflow was unable to send out email notifications and the error message was "The email message cannot be sent. Make sure the outgoing email settings for the server are configured correctly"

I was damn sure that I have configured the email settings on sharepoint server. I double checked, and triple checked it and confirmed that it is there. For you who doesn't know where to check, it is on your Sharepoint Central Administration - System Settings - Configure outgoing email settings.

The next thing I needed to double check is the setting on our exchange server, but again I was pretty sure that I have added the Sharepoint server to the allowed list. Sure enough when I checked it is already listed there as you can see in the picture below (highlighted)

For long I was struggling to pinpoint the issue. I thought it was the workflow issue because it's trying to send the email out using System Account. Even when we called Microsoft support guy, he confirmed that the System Account sometimes does "crazy things" and that's it - no more suggestions...

After about three weeks battling, I decided to look at the issue from a different angle. I turned on the logging feature on our Exchange Server for the SMTP traffic. By default on Exchange 2010, the SMTP logging is off and you need to set it to verbose to capture the SMTP traffic.

Sure enough the log helped me pin point the issue. It shows me that the one that initiated the email out is our SQL Server - not the Sharepoint itself. 

Default log location is on \Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\ProtocolLog\SmtpReceive

I then added the SQL Server IP address to the allowed list on our Exchange Server receive connector and it solves the issue.

Moral of the story: Never assume you know or have done everything. Always check your log first.