sendmail sender queue groups
2012/01/27
Sendmail provides for queue groups where one can have messages that stay in queue be placed in separate queues which are treated differently according to rules described in the queuegroup ruleset. FEATURE(queuegroup) helps managing such queues via the access database but unfortunately deals only with recipient addresses. But what if one wants to place messages in a separate (slower) queue based on sender’s address?
QUEUE_GROUP(`newsletter', `N=10, I=31m, P=/storage/queues/n.*')dnl LOCAL_RULESETS Squeuegroup R$* $: $>canonify $&f R$* < @ $* > $* $: $1 Rowner-newsletter $# newsletter
The above trick does not make use of the access database. In fact you must not use FEATURE(queuegroup) in your sendmail.mc with it. The queuegroup ruleset is called with the recipient address as an argument. The first line replaces it with the sender’s address ($&f) canonified. In this particular newsletter case, we are only interested in the left hand side of the email address ($1). Others may be interested in the sender’s domain ($2). The third line checks to see whether the left hand side matches what we expect (owner-newsletter) and if so, it selects the corresponding queue. Otherwise the default queue, named mqueue, is selected.
For a more complete ruleset that can treat combinations of senders and recipients and via the access database see “Sendmail Extended Queue Groups“.
check_compat vs MIMEDefang
2011/09/27
We have a user that wishes to have messages sent from sender@host-xyzw.etp.eu.example.com discarded by our mailservers. The natural choice for such blocks seems to be FEATURE(compat_check). In fact we had a number of other users with similar requests that were serviced this way. The problem in this case was that the xyzw part of host-xyzw.etp.eu.example.com was not constant or predictable and finite. Naturally I thought that a local version of the check_compat ruleset would suffice, since $*.eu.etp.example.com matches all possible such hostnames. But it seems that according to the bat book this cannot be done while also using FEATURE(compat_check):
Note that although with V8.12 and later you can still write your own check_compat rule set, doing so has been made unnecessary by the FEATURE(compat_check) (§7.5.7 on page 288). But also note that, as of V8.12, you cannot both declare the FEATURE(compat_check) and use this check_compat rule set.
Since I did not wish to tamper with our sendmail.mc this time, MIMEDefang came to the rescue: filter_relay is called with arguments both the sender and the receiver and that took care of it. But again, had I chosen to write this using sendmail’s language, it might have looked ugly, but it would also have been a one-liner (ugly but elegant in its own way).
mail hosted at Google, web server elsewhere
2011/03/14
This post aims to cover two sets of questions that frequently appear on Serverfault:
“I have the email of my organization hosted at Google and the web server at a hosting provider. When the web server sends email (when a form is completed for example), email is received by everyone except when the recipient is in our domain. Then sendmail tries to deliver locally and not over at Google”. Or, “certain recipients, including Google, reject email from the web server (or servers withing our LAN) as spam”.
There are answers at Serverfault recommending the use of ssmtp in order to forward all sending email via Google, but this requires SMTP authentication and a password saved in a file.
For the purposes of this post the domain example.com will be used.
Configure SPF for example.com
SPF is framework that allows the domain name owners notify the world who they believe the appropriate servers sending mail on behalf of their domain are. Google support pages note that the SPF record should at least be in the form of v=spf1 include:_spf.google.com ~all. However, it is also needed that server.example.org be able to send email on behalf of example.com. So the appropriate record becomes:
v=spf1 a:server.example.org include:_spf.google.com ~all
Note: example.org is not the same domain as example.com
Configure sendmail for server.example.org
example.com is included in /etc/mail/local-host-names, which means that server.example.org treats this a local domain and will try to deliver locally, instead of Google. The following additions to the sendmail configuration file (sendmail.mc) take care of this:
LOCAL_CONFIG
Kbestmx bestmx -T.TMP
LOCAL_RULE_0
R $* < @ example.com. > $*
$#esmtp $@ [$(bestmx example.com. $)] $: $1 < @ example.com. > $2
The line is broken in two for readability. As always remember that the LHS and the RHS of the rule are separated with tabs and not spaces. So do not copy-paste. Build and install sendmail.cf, restart sendmail and check.
I would welcome additions on how the same can be achieved with postfix or exim.
An alternative to FEATURE(mailertable)
2010/07/09
Using FEATURE(mailertable) one can instruct sendmail to route email for certain destination via a specific relay. A mailertable is essentially a static map that instructs sendmail where to route email for certain destinations ignoring DNS MX RRs (or other information). Example:
yahoo.com smtp:[server.example.com] yahoo.com.hk smtp:[server.example.com] yahoo.com.mx smtp:[server.example.com] yahoo.com.br smtp:[server.example.com] yahoo.com.cn smtp:[server.example.com] yahoo.com.sg smtp:[server.example.com]
Why would one want to do that? Your customers may have been hit by a botnet and as a result your outgoing mail server may have sent enormous amount of spam. Since most high-profile mail hubs use some kind of reputation scheme on the IP addresses that contact them, it is quite probable that your outgoing mail server is experiencing delays, or worse denied delivery despite the fact that in the meantime you have done your best to stop the botnet and clear your queues. I know for it has happened to me.
A mailertable is a quick solution to route email through another mail server just for recipient domains that implement such policies. But it is far from perfect for the Postmaster has no way to know all the domains that Yahoo! Mail in the above example hosts in order to construct a mailer table. Luckily, when high-profile mail hubs (like Gmail, Yahoo! Mail and Hotmail) implement good patterns on their DNS MX RRs, a programmatic (instead of a static) solution can be deployed:
LOCAL_CONFIG Kbestmx bestmx -T.TMP LOCAL_RULE_0 R$+ < @ $+ > $* $: $(bestmx $2 $: NOTFOUND $) $| $1 < @ $2 > $3 R$+.hotmail.com. $| $+ < @ $+ > $* $#esmtp $@ [server.example.com] $: $2 < @ $3 > $4 R$+ $| $+ < @ $+ > $* $: $2 < @ $3 > $4
In the above snippet, any email that is directed to a domain that is served by Hotmail’s servers is routed via server.example.com. For the record, our outgoing webmail server achieved a senderscore of 50, and although a filter stopped the plaque, Hotmail silently discarded email originating from it. Using the above solution restored communications for our users.
Using bestmx for discarding outgoing email
2010/07/07
The following ruleset discards email that originates from domains for which we are not best MX. It is meant to be applied on outgoing email servers:
LOCAL_CONFIG Kbestmx bestmx -T.TMP LOCAL_RULESETS SLocal_check_mail R$* $: $>canonify $1 # You may (or may not) want to comment the following line R < @ > $#OK R$* < @ $+. > $* $1 < @ $2 > $3 R$* < @ $+ > $* $: $2 # Short circuit certain domains (and host names) Rexample.com $#OK R$* . example.com $#OK R$* $: $(bestmx $1 $: NO $) # If a temporary error occurs, do not block R$*.TMP $#OK Rserver.example.com. $#OK R$* $#discard $: $1
This works for as long as spammers do not use domains for which they do not control the DNS zones. If they do control the DNS zones they can easily add your relays as MX to them. In such cases the above ruleset must be modified to lookup the name servers for domains that server.example.com is best MX and then decide to discard. However the above trick erased thousands of outgoing spams yesterday.
PS: Like I posted on twitter: I rewrote the above filter in ~35 lines of Perl (subroutine filter_sender for MIMEDefang’s mimedefang-filter). The sendmail version is both more compact and readable (at least to me).
on picking an MTA
2010/06/23
Sometimes I get asked on what is my MTA (Mail Transfer Agent) of choice. Almost always I am asking “What do you want to do with it?”. Personally, in most places I install sendmail. There are cases (cases where one would use FEATURE(nullclient) or similar) where I install nullmailer, for I find it unnecessary to run sendmail.
People sometimes ask me why do I choose sendmail and not Postfix (or Qmail in the old days) or even Exim since we are running a mostly Debian shop. Leaving the monolithic argument aside (which is kind of funny when most people that use it are using a monolithic kernel OS anyway) I am using sendmail because of its expressive power. I can find a way to express what I am thinking (filtering, routing, etc) in its modem noise of a programming language or milters like MIMEDefang (IIRC, there’s a wonderful PDF presentation by Ricudis on the Turing completeness of the sendmail.cf language but I have no link to it).
It is not that I have not used other MTAs. Hell, I was even running Postfix alpha versions right after it was renamed from VMailer. And occasionally I am running MeTA1 instances. But I always return to sendmail. If it does not suit you, it is OK. Pick the one MTA that can help you build the setup that you have in mind, be it Exim, Postfix, netqmail, commercial software like Exchange or CommuniGate, whatever. If it works for you and your team, then it is the right choice. Endless debates are for people who have too much free time.
However, if there is one recommendation that I can share, this is it: If you are serious about email (routing) invest some time reading the bat book. He who can understand a complex piece of software like sendmail, can guide himself through any email system.
(triggered by a brief conversation I had with a friend this afternoon)
sendmail load configuration
2010/06/05
This post is about a neat trick that I have not seen many times discussed. According to the configuration README the default values for controlling load averages are:
- confQUEUE_LA (QueueLA) Load average at which queue-only function kicks in. Default values is (8 * numproc) where numproc is the number of processors online (if that can be determined).
- confREFUSE_LA (RefuseLA) Load average at which incoming SMTP connections are refused. Default values is (12 * numproc) where numproc is the number of processors online (if that can be determined).
However in “Sendmail Theory and Practice” (I am a proud owner of both editions) Paul Vixie and Fred Avolio propose a different approach:
“Astute readers will note that the value shown for Ox (QueueLA) is larger than the value shown for OX (RefuseLA), and that this is opposite from the configuration files you may have seen elsewhere. Setting them as shown here gives Sendmail a range of load average in which it is capable of delivering messages from its queue but incapable of receiving new messages. This is intentional. If you set Ox to be less than OX, Sendemail has instead a range of load average in which it can receive new mail (thus adding to the queue) but cannot deliver any queued mail. We believe that mail queues should become smaller or stay the same size when the load average is high. After watching our large mail gateway computers melt down many times over the years, we have learned that it is better to let other hosts’ mail stay where it was -on other hosts- when our load average is high, than to accept it even though we don’t plan to do anything with it until load average becomes low again.”
In other words although the defaults suggest otherwise, it may be wiser to have QueueLA > RefuseLA. This piece of advice is on both the 1995 (1st) and 2002 (2nd) editions of the book. A pearl that comes from 1995 that is still relevant.
sendmail: Should I use $*, $+ or $- ?
2009/11/09
When writing a sendmail.mc rule, you can use some operators on the left hand side, like $- (match exactly one token), $+ (match one or more tokens) and $* (match zero or more tokens). You may find yourself in a situation where for example you want to use a certain delivery agent for some of your users. Normally you would write something like the following:
LOCAL_CONFIG Kmitsos btree -m -a.mitsos /etc/mail/mitsosusers LOCAL_RULE_0 : R$- < @ $=w . > $* $: $(mitsos $1 $) $3 R$- . mitsos $* $#mitsos $: $1
The above example seems to be correct, right? But what if you have a user in mitsosusers that contains a dot (.) in the user name (for example yiorgos.adamopoulos)? Because the . is a token separator (see the OperatorChars definition in your sendmail.cf), $- will not match the name. So the correct ruleset in this case is:
R$+ < @ $=w . > $* $: $(mitsos $1 $) $3 R$+ . mitsos $* $#mitsos $: $1
I got bitten by this sometime ago, and that is why I am sharing it.
Poor man’s milter-ahead
2009/08/21
I have blogged before that the reason that I like MIMEDefang is that it gives the Postmaster a Perl interpreter (a programming language that is) and a library of functions that can be used to filter and manipulate incoming and outgoing email.
Of the functions available I believe that md_check_against_smtp_server() deserves special mention since it can be used to quickly implement a poor man’s milter-ahead or milter-sender. Of course milter-ahead implements many features (caching among others), but with some effort most (if not all) of the functionality can be implemented withing mimedefang-filter.
Then again, milter-ahead does not cost much (€90) even for small organizations, so the not invented here syndrome can be supressed.
sendmail, libsnert and BerkeleyDB
2009/04/04
This has come up (and has been answered) a number of times in the SnertSoft mailing list:
Always make sure that you compile and link libsnert with the same version of BerkeleyDB that your sendmail binary is linked with. Otherwise you will experience access.db errors (it fails to open).
