Day 5 - How to fight and fix spam. Common problems and best tools.

Written By: Pablo Hinojosa (@pablohn6)
Edited By: Brendan Murtagh (@bmurts)

The Best Tools to Combat and Fix Common Spam Problems

This article summarizes from a 30,000 foot view of what is spam, anti-spam, and how to fix common problems. This is not an article where you are going to find the command(s) to fix spam problems for your MTA. With the help of this article you will understand why you are suffering spam problems and how to identify the root cause. This article is not intended as a how-to, but provide a foundation for troubleshooting and implementing a configuration to help rectify a spam/bad reputation problem.

What is Spam?

Obviously you know what Spam is but, do you know what that represents in global terms? According to Kaspersky Lab Spam and Phishing in Q3 report, “Six in ten of all emails received are now unsolicited spam”. Imagine visiting ten webpages and six of them be unsolicited? What if they were phone calls, sms, or clients of your business? This century’s primary form of communication is email, business-related or not, communications are electronic. Do you imagine start each day 6 of each 10 conversations in an unsolicited way? Systems administrators responsibility is to change that number 10 by number 100, 1000 or as much zeroes you are able to reach.

One thing to know to help understand the scale of spam is that spam is a huge business. Unsolicited emails is one of the most common methods to promote hundreds of legal and illegal business. From a small bikes shop to a huge phishing or ransomware criminal network.

Spam is also a huge consumer of resources, both electronic and human. The SMTP protocol was designed from a naive perspective. Old protocol designers did not take much into account on how to cheat in a communication, which is why big providers design and implement several protocols to authenticate and limit cheating in email delivery (composed by 2 MTA exchanging messages). It is important to learn what those protocols are and how to properly configure them to not be flagged.

However, there are instances where our servers are sending actual Spam. Obviously this is not our intention but we need to quickly identify the issue and begin remediation immediately. In the next section, we will focus on how to detect the sending of spam and discuss techniques to resolve the problem.

Are you a spammer?

Whether or not you are a spammer is a matter of trust. The receiving MTA will question your trustworthiness and you will have to show your reasons. Let’s see what we have to configure to respond no and be trusted to that question.

The most important record is an MX record. As RFC 1034 states, it “identifies a mail exchange for the domain”. You can send emails and be trusted from non MX records servers, but the best way to be trusted (because it is the first thing to be checked) is to send emails from your MX records servers. This is not always possible or desirable. Sometimes another MTA is sending emails spoofing the sending domain. This is typically unauthorized which is why the “SPF record” was created. As RFC 7208 states, with an SPF record you can “explicitly authorize the hosts/IPs that are allowed to use your domain name, and a receiving host can check such authorization”. An SPF record is a TXT record you should create (I recommend this website ) to tell the world which servers are allowed to send on behalf of your domain name.

Some MTAs require more validation. They need the email signed by your MTA to trust you and then be able to verify that signature. This is implemnted by using DKIM. DKIM “defines a domain-level authentication framework for email using public-key cryptography and key server technology to permit verification of the source and contents of messages by either Mail Transfer Agents (MTAs)”. As Wikipedia says: DKIM resulted in 2004 from merging two similar efforts, “enhanced DomainKeys” from Yahoo and “Identified Internet Mail” from Cisco. The configuration of DKIM depends on your MTA and your OS. Generally speaking the steps include, but aren’t limited to generating public-key cryptographic keys, set up your MTA and a TXT record). A simple Google search for your MTA, OS, and DKIM should get you started. You can verify your configuration with this tool.

There are times when a MTA can say, hey! you are cheating me! I reject your email and you should know you are a cheater. That is why DMARC was created. “DMARC is a scalable mechanism by which a mail-originating organization can express domain-level policies and preferences for message validation, disposition, and reporting, that a mail-receiving organization can use to improve mail handling”. It basically uses SPF and DKIM records to make a decision and accept or reject (and notify if you wish) your email. It is just a TXT record, you can use this generator, but there are several tools to create, validate your DNS record or email and read DMARC reports.

If you have not had spam problems previously and you configure MX, SPF, DKIM and DMARC is 99% sure you are going to respond “no” to “Are you spammer?” question and you will be trusted. If you are not trusted, feel free to send me an email and I will help you figure out the reason why with that configuration you are not trusted. Be sure you check all your configuration is OK with this amazing tool. But, wait, what happens if you are a spammer?

You are spammer.

First of all, with “you” I mean your IP. And sometimes, usually in shared hosting services, you have the problem, but you are not the cause. Or worse, your IP is not sending spam now, but it did before. And spam problems are so serious that once you have been flagged for spamming, they do not easily give you the chance to be forgiven. It is all about a reputation. Your reputation is based on your IP spam problems history and even your IP range spam problems history. Yes, the IP 7 bits away from your IP is sending spam and your reputation could be affected. To find out, I recommend you visit this website.

Most of the time, the problem is on your IP and thus your ip is blacklisted. I recommend this tool to check if your IP is blacklisted. But be careful, sometimes you may appear blacklisted, but not because of sending actual spam, but you because do not respect some RFC. That is why this tool shows only main and most famous blacklist. If you are blacklisted you will have to:

1. Be able to respond “no” to the “Are you a spammer?” question.
2. Fix your “you are spammer” problems (locate the spam source and fix it).
3. Request a delist to blacklist.

If you do just step 3, it will actually be worse because there is a strong possibility you will be blacklisted again, spam is so serious that blacklists sometimes only forgive you once, but sometimes not twice.

Often enough this scenario happens, when you send an email which is rejected or the email goes directly to the Receipent’s Spam folder. In the first case, NDR can sometimes tell you the reason (or the blacklist) for why they rejected the email. This level of detail depends on the receiving side’s MTA configuration.

However in second case, anti-spam software and major providers work in a different fashion. Typically service providers will flag your IP as a spammer, which results in all email originating from that host/IP go directly to the Spam folder or are rejected and it negatively affects to your “internal reputation”. The reputation of an email is calculated as a score using a mathematical formula in conjunction with pattern detection and defined rules that are analyzed by the mail server. This tool can show the score of your email content. This is very important when you are sending newsletters because they have a high probability to be marked as spam. This is why many companies and people use dedicated email marketing services like MailChimp and AWeber.

With major providers, that internal reputation depends on additional “secret” factors, but we could also say it helps when a person (not a robot or a mathematical formula) says: this is not spam. Do you remember that button?

If you are having Spam problems (mainly rejects) only with one provider, the next information will help you. If your problem is with Yahoo, you can use this form to say: hey please forgive me. Gmail also has this form. Microsoft (Outlook and Hotmail) has also this form but they also have an internal reputation tool to show you what do they think about you. They are named SNDS and JMRP, and if you are having problems with Hotmail (too often) they will help you a lot.

With small providers sometimes the best option is to send an email to postmaster requesting for a whitelist of your IP.

When you are sending too much spam, sometimes anti-spam software / services or major providers just reject your emails because they have no doubt you are a spammer. If from your MTA IP you cannot telnet to port 25 of MX record IP (timeout), you will not be able to send any email to them and then your emails will be queued. This is the worst symptom. Somebody can send emails to a provider, sometimes nobody can send emails to anybody, our telephone is ringing and everybody screaming. If you came here in that situation, I hope this article has helped you to understand how a serious of a problem spam can be. Remember to validate your configuration and always work as fast as possible to find the source of the spamming.

Locating a spam source is sometimes a hard, but necessary task for System Administrators. if you have read this far, you will understand how anti-spam technically works, so you will have more weapons to fight it. It is also a security research task, because usually a compromise has occurred to one of your clients or your server which was used to send unsolicited email all around of the world. In that case you will have to find the malicious code and also close the point of entry. In this situation, I suggest you to do the following:

• Study your mail logs to find out if it is a single email account or not. If it is one email account, maybe malware or a cracked email password is the root cause. Changing the password may fix the problem. However, if other malware is still on the client email device, the password could be compromised again. A re-image is the safest method to ensure a clean device or machine.
• If the FROM email is generated, that could be an internal malicious code generating Spam emails. You can create a wrapper for your MTA, OS and your platform stack to log the source of each email that is sent. For example this is a wrapper for Sendmail, Apache and PHP. Special attention if your platform is Wordpress or Joomla. Bots can try old bugs of non updated plugins or malicious free (but not free) themes to insert the malware.

As conclusion, we can say Spam is a huge problem that affects all email providers. That problem could be caused because a lack of configuration to increase ip reputation or because an actual spam sending due to malware. That is why it is important to take in account your ip reputation and also the security of your infrastructure to skip future problems.

Pablo Hinojosa is a Linux System Administrator that worked at Gigas Hosting Support Team assisting to thousands of clients affected by Spam.