Using Formail to remove headers

[Stephen Allen]

I'm a newbie to pocmail/formail so please forgive anything that seems
overwhelmingly obvious!

I write to several mailing lists, each of which needs a different return
address (eg. ***@mydomain.com, or ***@mydomain.com).
Because I need to be able to post messages from different places, I have to
send the messages to myself and let Procmail/Formail re-send it to the right
place. To keep things tidy I also remove a few non-essential headers. This
is an example:

:0
* ^X-Original-To:.procmail-***@mydomain\.com
| formail \
-I Received: \
-I Reply-To: \
-I "From: Stephen Allen <***@mydomain.com>" \
-I X-Original-To: \
-I Delivered-To: \
-I "To: ***@lists.RWTH-Aachen.DE" \
| $SENDMAIL "$SENDMAILFLAGS" \
***@mydomain.com \
***@lists.RWTH-Aachen.DE

I would like to remove all headers except Subject: and the MIME-related
ones, then set the To: and From: as above. So, I'm stuck! Would anyone be
able to help me find a solution please?

Many thanks,
Steve :)

[Michael J Wise]

Post by Stephen Allen
To keep things tidy I also remove a few non-essential headers.

I hope you never have to do any diagnostics.

Post by Stephen Allen
Would anyone be able to help me find a solution please?

Build a special filter in PERL, and pipe the message thru that.

But personally, I'd just use the aliases at mydomain.com, and point
them all to your main account, and ... well, I'd arrange things a bit
differently, is all, without sending the messages off to another
account somewhere else.

In short, not knowing what you are really doing, my suspicion is you're
making your life too complex for no good reason.

Aloha mai Nai`a!

[Stephen Allen]

This post might be inappropriate. Click to display it.

[Stephen Allen]

On Sunday, July 31, 2005 10:03 PM

Post by Stephen Allen
Currently, I remove all Received: headers as a courtesy to others,
but I ws thinking I don't actually need any header except To:, From:,
Subject: and anything to do with MIME.

Oops, I would need to keep the Message-ID: header too.

Steve :)

[Ruud H.G. van Tol]

Post by Stephen Allen

Post by Stephen Allen
Currently, I remove all Received: headers as a courtesy to others,
but I ws thinking I don't actually need any header except To:, From:,
Subject: and anything to do with MIME.

Oops, I would need to keep the Message-ID: header too.

Don't forget the References: and In-Reply-To: header fields.

[Dallman Ross]

Post by Stephen Allen
Currently, I remove all Received: headers as a courtesy to
others, but I ws thinking I don't actually need any header except
To:, From:, Subject: and anything to do with MIME.
So, the million dollar question, how to remove all headers except the above?

One way: Save the headers you want into vars, remove all headers,
put the headers you want back.


:0
* ^\/From:.*
{ H_FROM = $MATCH }


:0
* ^\/Subject:.*
{ H_SUBJ = $MATCH }


:0
* ^\/Content-Type:.*
{ H_CTYPE = $MATCH }

:0
* ^\/MIME-Version:.*
{ H_MIME = $MATCH }


:0
* ^\/Content-Transfer-Encoding:.*
{ H_CTE = $MATCH }


:0
* ^\/Content-Disposition:.*
{ H_DISPO = $MATCH }



:0
| formail -I "" \
-I "$H_FROM" \
-I "$H_SUBJ" \
-I "$H_CTYPE" \
-I "$H_MIME" \
-I "$H_CTE" \
-I "$H_DISPO" \
| $SENDMAIL $SENDMAILFLAGS -t ***@somewhere


Tested, but for the sendmail pipe.


Dallman

[Stephen Allen]

On Sunday, July 31, 2005 11:05 PM

Post by Dallman Ross
One way: Save the headers you want into vars, remove all headers,
put the headers you want back.
:0
| formail -I "" \
-I "$H_FROM" \
-I "$H_SUBJ" \
-I "$H_CTYPE" \
-I "$H_MIME" \
-I "$H_CTE" \
-I "$H_DISPO" \

Thanks Dallman, that looks perfect! I didn't know you could save vars like
that. My next project would be to set up an alias file type of thing
instead of using all this for each and every mailling list. I'll let you
(everyone) know if I need help.

Once again, thanks very much.
Steve :)

[David W. Tamkin]

Dallman suggested to Stephen,

Post by Dallman Ross
One way: Save the headers you want into vars, remove all headers,
put the headers you want back.
:0
* ^\/From:.*
{ H_FROM = $MATCH }
:0
* ^\/Subject:.*
{ H_SUBJ = $MATCH }
:0
* ^\/Content-Type:.*
{ H_CTYPE = $MATCH }
:0
* ^\/MIME-Version:.*
{ H_MIME = $MATCH }
:0
* ^\/Content-Transfer-Encoding:.*
{ H_CTE = $MATCH }
:0
* ^\/Content-Disposition:.*
{ H_DISPO = $MATCH }
:0
| formail -I "" \
-I "$H_FROM" \
-I "$H_SUBJ" \
-I "$H_CTYPE" \
-I "$H_MIME" \
-I "$H_CTE" \
-I "$H_DISPO" \

Oy. Why all the saving into variables?

:0
| formail -kz -X From: -X Subject: -X Content-Type: \
-X MIME-Version: -X Content-Transfer-Encoding: \
-X Content-Disposition: | \
"$SENDMAIL" $SENDMAILFLAGS -t ***@somewhere.else

[Stephen Allen]

On Monday, August 01, 2005 12:19 AM

Post by David W. Tamkin
:0
| formail -kz -X From: -X Subject: -X Content-Type: \
-X MIME-Version: -X Content-Transfer-Encoding: \
-X Content-Disposition: | \

I adapted your idea and laid it out as below I've done it like this so it's
easier for me to change). It seems to work except that I can't get the
From: and To: fields added. The message is delievered correctly, so the
SENDMAIL part is working, but the From: & To: are being set to my user
default rather than what I want as shown.

Thank you,
Steve :)

:0
* ^X-Original-To:.test-***@rowyerboat\.com
| formail -kz \
-X Subject: \
-X In-Reply-To: \
-X Content-Type: \
-X MIME-Version: \
-X Content-Transfer-Encoding: \
-X Content-Disposition: \
-X Message-ID: \
-X References: \
-A "From: Stephen Allen <***@rowyerboat.com>" \
-A "To: ***@anotherdomain.com" \
| $SENDMAIL "$SENDMAILFLAGS" \
***@anotherdomain.com

[David W. Tamkin]

Post by Stephen Allen
I adapted your idea and laid it out as below I've done it like this so it's
easier for me to change). It seems to work except that I can't get the
From: and To: fields added. The message is delievered correctly, so the
SENDMAIL part is working, but the From: & To: are being set to my user
default rather than what I want as shown.
:0
| formail -kz \
-X Subject: \
-X In-Reply-To: \
-X Content-Type: \
-X MIME-Version: \
-X Content-Transfer-Encoding: \
-X Content-Disposition: \
-X Message-ID: \
-X References: \
| $SENDMAIL "$SENDMAILFLAGS" \

The lack of -X overrides -A, and we really should have loop detection
for any forwarding, and it's $SENDMAILFLAGS, not $SENDMAIL, that needs
to be without quotes, so ...

:0
* ^X-Original-To:.test-***@rowyerboat\.com
* ! $ ^X-Loop: $\LOGNAME@$\HOST
| formail -kz \
-X Subject: \
-X In-Reply-To: \
-X Content-Type: \
-X MIME-Version: \
-X Content-Transfer-Encoding: \
-X Content-Disposition: \
-X Message-ID: \
-X References: \
-A "From: Stephen Allen <***@rowyerboat.com>" -X From: \
-A "To: ***@anotherdomain.com" -X To: \
-A "X-Loop: $LOGNAME@$HOST" -X X-Loop: \
| "$SENDMAIL" $SENDMAILFLAGS \
***@anotherdomain.com

[Dallman Ross]

Post by David W. Tamkin
The lack of -X overrides -A, and we really should have loop
detection for any forwarding, and it's $SENDMAILFLAGS, not
$SENDMAIL, that needs to be without quotes, so ...

Ah, yes. I'll have to remember the "negative" override.
(Sort of reminds me of the Dormant Commerce Clause.)[1] :-)


[1] Joke only works if you're somewhat of a legal beagle.

Dallman

[Dallman Ross]

I adapted your [David Tamkin's] idea and laid it out as below
I've done it like this so it's easier for me to change). It
seems to work except that I can't get the From: and To: fields
added. The message is delievered correctly, so the SENDMAIL part
is working, but the From: & To: are being set to my user default
rather than what I want as shown.
:0
| formail -kz \
-X Subject: \
-X In-Reply-To: \
-X Content-Type: \
-X MIME-Version: \
-X Content-Transfer-Encoding: \
-X Content-Disposition: \
-X Message-ID: \
-X References: \
| $SENDMAIL "$SENDMAILFLAGS" \

Try changing the -A to -I, in addition to using the -f option to
sendmail (with an address thereafter for the sender), the latter
point being a repeat of one I just wrote in a companion message
following up to David.

Good that he pointed out the need for an X-Loop check, as well.

Dallman

[David W. Tamkin]

Dallman suggested to Stephen,

Try changing the -A to -I ...

Yes, except for X-Loop:, which should always be added with -A; and also,
Stephen still needs the -X options for the other headers, or even -I
won't survive the other -X's.

[David W. Tamkin]

Dallman suggested to Stephen,

Try changing the -A to -I ...

Yes, except for X-Loop:, which should always be added with -A; and also,
Stephen still needs the -X options for the other headers, or even -I
won't survive the other -X's.

[Dallman Ross]

Post by David W. Tamkin
Dallman suggested to Stephen,

Post by Dallman Ross
One way: Save the headers you want into vars, remove all headers,
put the headers you want back.
:0
* ^\/From:.*
{ H_FROM = $MATCH }
:0
* ^\/Subject:.*
{ H_SUBJ = $MATCH }
[. . . .]
:0
| formail -I "" \
-I "$H_FROM" \
-I "$H_SUBJ" \
-I "$H_CTYPE" \
-I "$H_MIME" \
-I "$H_CTE" \
-I "$H_DISPO" \

Oy. Why all the saving into variables?
:0
| formail -kz -X From: -X Subject: -X Content-Type: \
-X MIME-Version: -X Content-Transfer-Encoding: \
-X Content-Disposition: | \

Yes, very clever; thanks, David. I actually tried with the -k
flag along with the -r flag, as my first idea. I couldn't figure
out how to get -k not to prepend a quote character on the body
lines, though, so I gave up on that idea. I think your -z switch
is pushing what otherwise would be a space back over to the left
margin? Good. Anyway, I wasn't thinking at the time that formail
would have read in the original headers, anyway, though, so know
what their contents is, even though we are deleting them in the
same action. Good thinking.

As for the OP's issue with the From header's being the original
user, the solution for him is to use the -f option with sendmail,
if his system will honor it. (Mine does.)

| "$SENDMAIL" "$SENDMAILFLAGS" -f "$MYADDRESS" -t

The quotes aren't really necessary, as you implied in a follow-up
message. But I have quotes around "$SENDMAILFLAGS" in working
operation, and it does not cause a problem; so your inference
in that companion posting about the quote marks there being an
issue does not seem to be the case. Do you have other experience
with that?

Dallman

[Dallman Ross]

b) Why do $LOGNAME and $HOSTNAME have a \ after the $ symbols?

"$\name will be substituted by the
all-magic-regular-expression-characters-disarmed equivalent of $name"
That is 'cryptic' for: "use $\var in stead of $var when you use a
variable inside a condition, unless you set up $var especially for usage
inside a condition".

And *that* is cryptic for, "'$\var' quotes all the chars in '$var'
that would otherwise be regex magic chars or metachars."

Suppose $LOGNAME is "john.smith". Well, then $\LOGNAME is
"john\.smith". (Actually, it's "()john\.smith", but the "()"
gesture is meaningless in this case.)
There really should be, but Microsoft Outlook Express doesn't
put one in, in most cases, so spits in the face of the standard.
Btw, it is a standard, but not an actual requirement of the RFCs.
They advise that it's a good idea, is all. At least that's my
recollection from my last read-through.

I have anti-spam recipes that look carefully at the Message-ID.
I have to write in exceptions for OE and a couple of other lame
MUAs.

Dallman

[Ruud H.G. van Tol]

Post by Dallman Ross

b) Why do $LOGNAME and $HOSTNAME have a \ after the $ symbols?

"$\name will be substituted by the
all-magic-regular-expression-characters-disarmed equivalent of $name"
That is 'cryptic' for: "use $\var in stead of $var when you use a
variable inside a condition, unless you set up $var especially for
usage inside a condition".

And *that* is cryptic for, "'$\var' quotes all the chars in '$var'
that would otherwise be regex magic chars or metachars."

Since when are you allowed to spoil my jokes?
(Stephen started, by calling the man-pages cryptic.)

Post by Dallman Ross
There really should be, but Microsoft Outlook Express doesn't
put one in, in most cases, so spits in the face of the standard.

That totally depends on how you set up Windows Networking. Amongst many
programs, I use W2k+OE, my Message-ID contains "@isolution.nl".
Righ click on My Computer -> Properties -> Network Identification ->
Properties (see Full computer name) -> More -> Primary DNS suffix of
this computer.

Post by Dallman Ross
Btw, it is a standard, but not an actual requirement of the RFCs.
They advise that it's a good idea, is all. At least that's my
recollection from my last read-through.

It is a bit stronger than that. RFC 2822, section 3.6.4:
Though other algorithms will work, it is RECOMMENDED that the right
hand side contain some domain identifier (either of the host itself
or otherwise) such that the generator of the message identifier can
guarantee the uniqueness of the left hand side within the scope of
that domain.

When the left-side is random enough, you might think that you get away
with a right-side that is not exclusively yours, but then there will
always be a risk of collisions, which goes against "will work". If
everybody would use a proper FQDN (host.domain.tld) at the right-side,
and a proper date.time.counter.random-or-hash at the left-side of the @,
then there is no quest for collisions.

Post by Dallman Ross
I have anti-spam recipes that look carefully at the Message-ID.
I have to write in exceptions for OE and a couple of other lame
MUAs.

Sometimes it just ends in "@localhost>" or "@[127.0.0.1]>" or even
"@127.0.0.1>".

[David W. Tamkin]

Ruud explained to Stephen,

That is 'cryptic' for: "use $\var in stead of $var when you use a
variable inside a condition, unless you set up $var especially for usage
inside a condition".

Or essentially, "if $var contains literal text rather than a regexp, use
$\var."

[David W. Tamkin]

Stephen also asked,

a) I've used -I instead of -A, although it seems I could have used either
(or even -i or -a). Is -I the 'normal' option to use?

If there's no preexisting header with the same name, there's no
difference. If there is, -I will remove the old one and leave only the
new one; -A will leave the old one and add the new one.

With X-Loop: you should use -A. It's very unfriendly to use -I. If
someone else's X-Loop: header is already there and the message gets
forwarded back to that person, but someone along the way has removed the
earlier X-Loop:, the removal victim's setup will forward it again.

With To:, well, if you're going to use $SENDMAIL -t, you should add the
new To: with -I so that you aren't sending a copy to the old To: address.

| "$SENDMAIL" $SENDMAILFLAGS -t \
-F "Stephen Allen" \
a) I've ommitted the From: header and used -f in the SENDMAIL command
instead. Is this OK?

Not the same. -f sents the From_ envelope-sender address, not the From:
header address.

b) I've used -t in the SENDMAIL command so it relies on the TO fields, is
this OK too?

In this particular case, yes.

The one last little bit I'm not sure about is how to correct the Message-ID.
I was informed by someone that I could probably do with changing that to
reflect the actual source domain (ie. mydomain.com). At the moment it's
just pegged with the name of my Windows machine.

Just some more options on the call to formail you're already making:

-a Message-Id: -XMessage-Id: \

and that should do it.

[Ruud H.G. van Tol]

a) I don't understand ! $ ^X-Loop.. etc... What does the $ mean here?

The loose '$' in (the start of) the condition means 'reparse from here'.
It is necessary for expanding the variables in the condition.

Normally a '$' inside a condition matches a newline-character
(ASCII-10). But starting at a loose '$', the rest of the line is parsed
for things to expand, like $LOGNAME and $HOSTNAME. (Shouldn't HOSTNAME
be HOST?)

b) Why do $LOGNAME and $HOSTNAME have a \ after the $ symbols?

In 'man procmailrc' this is said about '$\':
"$\name will be substituted by the
all-magic-regular-expression-characters-disarmed equivalent of $name"

That is 'cryptic' for: "use $\var in stead of $var when you use a
variable inside a condition, unless you set up $var especially for usage
inside a condition".
There really should be a FQDN after the @.

[David W. Tamkin]

I actually tried with the -k flag along with the -r flag, as my first
idea. I couldn't figure out how to get -k not to prepend a quote
character on the body lines, though, so I gave up on that idea.

-b prevents the body quoting of -kr, but we don't want -r anyway.

I think your -z switch is pushing what otherwise would be a space
back over to the left margin?

With -X (rather than -x), -z just canonicalizes all the retained headers
to have exactly one space and zero tabs between the colon and the first
non-blank character after it.

But I have quotes around "$SENDMAILFLAGS" in working operation, and
it does not cause a problem; so your inference in that companion
posting about the quote marks there being an issue does not seem to
be the case. Do you have other experience with that?

If you have more than one option in $SENDMAILFLAGS, quoting it will make
$SENDMAIL get it as one option with a hard space in the middle instead
of two options.

[Dallman Ross]

Post by David W. Tamkin

I have quotes around "$SENDMAILFLAGS" in working operation,
and it does not cause a problem; so your inference in that
companion posting about the quote marks there being an issue
does not seem to be the case. Do you have other experience
with that?

If you have more than one option in $SENDMAILFLAGS, quoting it
will make $SENDMAIL get it as one option with a hard space in the
middle instead of two options.

Ah. Good point. I'm going to take my quotes away. But won't
that cause a problem if the var has an unquoted space?

Dallman

[Stephen Allen]

Thank you again for your many replies. I've actually learnt quite a lot
through watching what you all have to say (the manpages alone are, at best,
cryptic). My finished article is below (with email address munged to help
stop sniffers), but I do have a few questions still (see after)...

:0
* ^X-Original-To:.procmail-***@mydomain\.com
* ! $ ^X-Loop: $\LOGNAME@$\HOSTNAME
| formail -kz \
-X Subject: \
-X In-Reply-To: \
-X Content-Type: \
-X MIME-Version: \
-X Content-Transfer-Encoding: \
-X Content-Disposition: \
-X Message-ID: \
-X References: \
-I "X-Loop: $LOGNAME@$HOSTNAME" -X X-Loop: \
-I "To: ***@lists.RWTH-Aachen.DE" -X To: \
| "$SENDMAIL" $SENDMAILFLAGS -t \
-F "Stephen Allen" \
-f ***@mydomain.com


Questions...


* ! $ ^X-Loop: $\LOGNAME@$\HOSTNAME

a) I don't understand ! $ ^X-Loop.. etc... What does the $ mean here?
b) Why do $LOGNAME and $HOSTNAME have a \ after the $ symbols?


-I "X-Loop: $LOGNAME@$HOSTNAME" -X X-Loop: \
-I "To: ***@lists.RWTH-Aachen.DE" -X To: \

a) I've used -I instead of -A, although it seems I could have used either
(or even -i or -a). Is -I the 'normal' option to use?


| "$SENDMAIL" $SENDMAILFLAGS -t \
-F "Stephen Allen" \
-f ***@mydomain.com

a) I've ommitted the From: header and used -f in the SENDMAIL command
instead. Is this OK?
b) I've used -t in the SENDMAIL command so it relies on the TO fields, is
this OK too?


The one last little bit I'm not sure about is how to correct the Message-ID.
I was informed by someone that I could probably do with changing that to
reflect the actual source domain (ie. mydomain.com). At the moment it's
just pegged with the name of my Windows machine.

Message-ID: <017401c5952c$70bd9720$***@dangermouse>

I can only think that all my mail (and 99.9% of the rest of the worldss) is
sent like this, so is it a problem, or just normal?

Thank you very much again,
Steve :)

[David W. Tamkin]

Post by Dallman Ross
Ah. Good point. I'm going to take my quotes away. But won't
that cause a problem if the var has an unquoted space?

That's the idea; the expansion should be broken at that unquoted space.
If it has a space that needs to be kept, you can always set

SENDMAILFLAGS="ar' 'g1 arg2"

and it will come through as "ar g1" "arg2" when $SENDMAIL is invoked,
I'd expect.