The mailing list for listmasters using Sympa

[CSS <address@concealed>]

On Sep 9, 2011, at 5:53 AM, Elwyn Davies wrote:

> Hi, Charles.
> 
> CSS wrote:
>> On Sep 7, 2011, at 2:51 PM, Elwyn Davies wrote:
>> 
>>  
>>> Hi, Charles.
>>> 
>>> I am glad you found my stuff useful!
>>>    
>> 
>> I'm top-posting a bit here, but I wonder if you could confirm if this 
>> explanation of how the postfix side of this works seems correct to you?
>>  
> It seems to be essentially correct.
> This piece of the Postfix documentation gives a useful overview of the 
> rewriting process
> http://www.postfix.org/ADDRESS_REWRITING_README.html
> .. but I agree it is very complicated!
> I spent best part of a month back in 2008 sorting out the whole process in 
> detail!
>> Here's how I think this works. Domain is lists.hostcompanion.com. List is 
>> "address@concealed". Inbound email is a subscribe request via 
>> email, so "address@concealed". Delivery and rewrite 
>> path goes like this then:
>> 
>>      • address@concealed comes in - Postfix knows it 
>> receives "lists.hostcompanion.com" email from the main.cf line 
>> "virtual_alias_domains = lists.hostcompanion.com"
>> 
>>      • Postfix then checks if the user exists. The main.cf line 
>> "virtual_alias_maps = 
>> regexp:/usr/local/etc/sympa/postfix/lists.hostcompanion.com" tells it to 
>> check the referenced file, which supports regex matching, to see if the 
>> user matches.
>> 
>>      • In that regex file, this line matches: 
>> "/^(test1)-(request|editor|owner|subscribe|unsubscribe)\@lists\.hostcompanion\.com$/
>>  $1+lists.hostcompanion.com@sympa$2.". The match is first on "test1", and 
>> the parens means to save "test1" in $1. The "-" matches, and again, the 
>> parens tell it to save the match, this time in $2 (second match). Lastly, 
>> the "@lists.hostcompanion.com" matches. This is therefore deliverable, and 
>> postfix accepts the message.
>>  
> Yes.  This is pretty much standard regular expression grammar.
>>      • That same regex file not only does a match to confirm the address 
>> is valid, but it has rewritten it based on the 
>> "$1+lists.hostcompanion.com@sympa$2." half of the regex.
>> 
>>      • The address is now "test1+lists.hostcompanion.com@sympasubscribe." 
>> (note the trailing ".")
>>  
> I think that the trailing . is not maintained internally.  It is only 
> needed in the virtual alias right hand side format to suppress the addition 
> of the local domain.  If you look at the output form 'sendmail -bv' shown 
> in the howto you will see that the trailing dot is not present there after 
> the mapping.
>>      • The transports match comes next. The line "/^.*\@sympasubscribe$/ 
>> sympasubscribe:" matches the "@sympasubscribe." that was the result of the 
>> rewrite.
>> 
>>      • The transport match tells Postfix to hand this to the 
>> "sympasubscribe" transport defined in master.cf.
>> 
>>      • The master.cf line "flags=RF user=sympa 
>> argv=/usr/local/libexec/sympa/queue ${user}-subscribe@${extension}" is 
>> used.
>>  
> Actually the whole of the two lines describing the transport (the dsecond 
> line is just a continuation).
>>      • master.cf is referencing "user" and "extension". "user" is the 
>> address to the left of the "+" in our last rewrite (ie: test1) and 
>> "extension" is everything between the "+" and the "@" in the last rewrite 
>> (ie: lists.hostcompanion.com).
>>  
> Note that this '+' separator is configurable.  '+' is the default.
>>      • This means /usr/local/libexec/sympa/queue is called like so: 
>> "/usr/local/libexec/sympa/queue address@concealed" 
>> (and the queue has access to the envelope-from to know *who* to subscribe 
>> of course).
>> 
>> I'm not so sure about the transport matching, and I'm also unclear on the 
>> "sympasubscribe." domain and where postfix learns that it's local.  Other 
>> than that, do I basically have the message flow down?
>>  
> See the address rewriting HOWTO mentioned above for the transport matching.
> The pseudo-domains used in the transport are indeed peculiar.  The point 
> here seems to be that all the checking of domains (like making sure we 
> aren't doing relay for illegal domains) is done on the input queuing side.  
> The Virtual aliasing happens there.  The transport mapping happens on the 
> output side and I *think* that no checking is done - it just matches.  I 
> presume that Postfix accepts that domains arising out of its internal 
> mapping process are 'kosher' and don't need any further checking.  And we 
> don't actually want any aliasing etc to happen for these pseudo-domains, so 
> they just don't need to appear in any of the checking or mapping 
> configuration.  I guess the Postfix authors could have put in another 
> configuration line, maybe 'other legitimate domains' but I think we are 
> maybe explouting a 'feature;.

Well, now I'm deep into "where does the rewriting happen again" mess.  I'm 
just skipping the VERP stuff now, this will do.  But I did want to leverage 
my amavis install on our primary mxer.  I've got the amavis side all sorted 
out - put a policy bank in place to match mail from this host, set it to send 
that email back to the right host, and normal deliveries with amavis in place 
work fine.

However, when I activate my "content_filter" for amavis, sympa seems to fall 
apart.

Do you (or anyone here) have any tips just how to put the mail through a 
content filter in postfix without mangling the rewrites?

In main.cf, I just have this for amavis:

content_filter=smtp-amavis:[207.99.53.216]:11024

And in master.cf, two chunks, one to deal with the above filter:


smtp-amavis unix -      -       -       -       2  smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20

And for the return path from amavis:

207.99.53.221:10027 inet n  -       -       -       -  smtpd
    -o content_filter=
    -o local_recipient_maps=
    -o relay_recipient_maps=
    -o smtpd_restriction_classes=
    -o smtpd_client_restrictions=
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o strict_rfc821_envelopes=yes
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o 
receive_override_options=no_header_body_checks,no_unknown_recipient_checks

It's a long list, but I think most of that is just to override things to 
defaults.

I've tried adding the "receive_override_options=no_address_mappings" to both 
in turn and that doesn't seem to alter things.

Any ideas?

Thanks,

Charles


>>  
>>> As far as I can see, you are pretty much done.
>>>    
>> 
>> Let's hope so, this hurt my brain.
>>  
> That was my feeling back in 2008! Hence the long winded howto in case I 
> forgot any of the hard earned integration of knowledge.
> 
> Regards,
> Elwyn
> 
> PS
> Thanks for the files.  I'll do a few updates to the HOWTO shortly.
> 
> /E
>> Thanks,
>> 
>> Charles
>> 
>>  
>>> You seem to think you have a problem here in that you have to paste this 
>>> in manually..
>>> 
>>>    
>>>>> And finally, the hack that I stick in the top of that same file to 
>>>>> handle the
>>>>> bounces, posts, etc.:
>>>>> 
>>>>> # this seems to be necessary, probably belongs somewhere else
>>>>> /^(sympa)\@lists\.hostcompanion\.com$/
>>>>> $1+lists.hostcompanion.com@sympalist.
>>>>> /^(listmaster)\@lists\.hostcompanion\.com$/
>>>>> $1+lists.hostcompanion.com@sympalist.
>>>>> 
>>>>>        
>>> Well you could do some extra stuff to generate this automatically with
>>> the first instance of the list, but I didn't.
>>> 
>>> Its a once per virtual host operation, so I just left it for manual
>>> config.  I only really cared about making extra lists (I only had one
>>> host to worry about so it wasn't a big deal).  You will have to do it
>>> for each virtual host.
>>> 
>>> This line is *NOT* required (and will probably screw things up)
>>> /^(.*+owner\@lists\.hostcompanion\.com$/
>>>    
>>>>> $1+lists.hostcompanion.com@sympaowner.
>>>>> 
>>>>>        
>>> The aliases already cope with address@concealed etc
>>> 
>>> One point which is not clear form your mail is that it is essential that
>>> there are no extra newlines in the added lines.. if they are really like
>>> this in the virtual host aliases file, then they won't work.  You need
>>> something like:
>>> /^(root|postmaster)\@lists\.hostcompanion\.com$/      
>>> address@concealed
>>> /^(sympa|listmaster)\@lists\.hostcompanion\.com$/     
>>> $1+lists.hostcompanion.com@sympalist.
>>> 
>>> where there is no newline after .com$/
>>> 
>>> BTW
>>> If you are willing to send me the updated conf.pm etc files, I'll add
>>> them to the description so that people can use them with the latest
>>> version.  I haven't had occasion to upversion my installation yet as it
>>> does everything we needed (very simple requirements) and the password
>>> improvement was not essential.  For our purposes the shared documents
>>> folders were really helpful!
>>> 
>>> Regards,
>>> Elwyn Davies
>>> 
>>> On Wed, 2011-09-07 at 09:14 +0200, address@concealed wrote:
>>>    
>>>> Hello,
>>>> 
>>>> I'm having a tough time getting this setup properly for a non-profit I'm
>>>> working with.
>>>> 
>>>> I read up on Sympa's features and it looks like the right choice - it 
>>>> covers
>>>> three things they need - web-based posting, discussion lists, and 
>>>> announce
>>>> lists.  And it covers things I need like a decent permissions system and
>>>> bounce processing.
>>>> 
>>>> After reading about the various Postfix integration methods, this one 
>>>> seemed
>>>> like it was the best in that it would not make me a backscatter source 
>>>> (the
>>>> other options for virtual setups do rejects after the message enters the 
>>>> queue
>>>> due to their greedy regex matching):
>>>> 
>>>> http://www.folly.org.uk/sympa/sympa_config_03.html
>>>> 
>>>> It's a nice setup, but the changes to Sympa since that was released left 
>>>> me
>>>> having to patch up Conf.pm, alias_manage.pl, and confdefs.pm.
>>>> 
>>>> This seems to work however - all my address@concealed mappings 
>>>> work,
>>>> those aliases are generated automatically by the patched alias manager.  
>>>> The
>>>> Conf.pm changes are working fine (set flags in each virthost robot.conf).
>>>> 
>>>> What I seem to be missing since I'm looking at so many Postfix 
>>>> tutorials, many
>>>> of which are now outdated, is how to handle my address@concealed 
>>>> mappings.
>>>> I can't find a decent reference for that.  Below is my current postfix 
>>>> setup,
>>>> which seems to work, but I need to add a mapping manually for each host 
>>>> as
>>>> noted in the config.
>>>> 
>>>> Let's start with main.cf:
>>>> 
>>>> #
>>>> # SYMPA STUFF
>>>> #
>>>> transport_maps = regexp:/usr/local/etc/postfix/transport_sympa
>>>> 
>>>> # Set the concurrency for delivery to Sympa
>>>> sympalist_destination_recipient_limit = 1
>>>> symparequest_destination_recipient_limit = 1
>>>> sympaeditor_destination_recipient_limit = 1
>>>> sympasubscribe_destination_recipient_limit = 1
>>>> sympaunsubscribe_destination_recipient_limit = 1
>>>> sympabounce_destination_recipient_limit = 1
>>>> 
>>>> # Specify the virtual alias domain(s)
>>>> virtual_alias_domains = lists.hostcompanion.com,lists.o4onyc.org
>>>> 
>>>> # Specify the virtual alias file(s) - this is the only per virthost 
>>>> config
>>>> virtual_alias_maps = regexp:/usr/local/etc/sympa/postfix/
>>>> lists.hostcompanion.com,regexp:/usr/local/etc/sympa/postfix/lists.o4onyc.org
>>>> 
>>>> Then the transport maps:
>>>> 
>>>> # Transport map for Sympa virtual domain transports
>>>> /^.*\@sympalist$/         sympalist:
>>>> /^.*\@symparequest$/      symparequest:
>>>> /^.*\@sympaeditor$/       sympaeditor:
>>>> /^.*\@sympasubscribe$/    sympasubscribe:
>>>> /^.*\@sympaunsubscribe$/  sympaunsubscribe:
>>>> /^.*\@sympaowner$/       sympabounce:
>>>> 
>>>> And master.cf:
>>>> 
>>>> # Sympa mailing list manager transports
>>>> sympalist     unix  -       n       n       -       -       pipe
>>>> flags=RF user=sympa argv=/usr/local/libexec/sympa/queue 
>>>> ${user}@${extension}
>>>> symparequest     unix  -       n       n       -       -       pipe
>>>> flags=RF user=sympa argv=/usr/local/libexec/sympa/queue ${user}-request@
>>>> ${extension}
>>>> sympaeditor     unix  -       n       n       -       -       pipe
>>>> flags=RF user=sympa argv=/usr/local/libexec/sympa/queue ${user}-editor@
>>>> ${extension}
>>>> sympasubscribe     unix  -       n       n       -       -       pipe
>>>> flags=RF user=sympa argv=/usr/local/libexec/sympa/queue 
>>>> ${user}-subscribe@
>>>> ${extension}
>>>> sympaunsubscribe     unix  -       n       n       -       -       pipe
>>>> flags=RF user=sympa argv=/usr/local/libexec/sympa/queue 
>>>> ${user}-unsubscribe@
>>>> ${extension}
>>>> sympabounce     unix  -       n       n       -       -       pipe
>>>> flags=RF user=sympa argv=/usr/local/libexec/sympa/bouncequeue ${user}
>>>> 
>>>> And now the aliases that are autogenerated for one virthost:
>>>> 
>>>> ## This virtual aliases file for domain lists.hostcompanion.com is 
>>>> dedicated
>>>> to Sympa Mailing List Manager
>>>> ## You should edit your Postfix main.cf file to declare it
>>>> ##
>>>> /^lists\.hostcompanion\.com$/            xxx
>>>> ##
>>>> #------------------------------ test1: list alias created 06 Sep 2011
>>>> /^(test1)-(request|editor|owner|subscribe|unsubscribe)\@lists\.hostcompanion
>>>> \.com$/    $1+lists.hostcompanion.com@sympa$2.
>>>> /^(test1)\@lists\.hostcompanion\.com$/
>>>> $1+lists.hostcompanion.com@sympalist.
>>>> 
>>>> And finally, the hack that I stick in the top of that same file to 
>>>> handle the
>>>> bounces, posts, etc.:
>>>> 
>>>> # this seems to be necessary, probably belongs somewhere else
>>>> /^(sympa)\@lists\.hostcompanion\.com$/
>>>> $1+lists.hostcompanion.com@sympalist.
>>>> /^(listmaster)\@lists\.hostcompanion\.com$/
>>>> $1+lists.hostcompanion.com@sympalist.
>>>> /^(.*+owner\@lists\.hostcompanion\.com$/
>>>> $1+lists.hostcompanion.com@sympaowner.
>>>> 
>>>> I'm pretty lost here - any input is appreciated.
>>>> 
>>>> Thanks,
>>>> 
>>>> Charles
>>>>      
>> 
>>  
>