The mailing list for listmasters using Sympa

[Elwyn Davies <address@concealed>]

Hi, Charles.

CSS wrote:
> On Sep 7, 2011, at 2:51 PM, Elwyn Davies wrote:
>
>   
> > Hi, Charles.
> >
> > I am glad you found my stuff useful!
> >     
>
> I'm top-posting a bit here, but I wonder if you could confirm if this 
> explanation of how the postfix side of this works seems correct to you?
>   
It seems to be essentially correct.
This piece of the Postfix documentation gives a useful overview of the 
rewriting process
http://www.postfix.org/ADDRESS_REWRITING_README.html
.. but I agree it is very complicated!
I spent best part of a month back in 2008 sorting out the whole process 
in detail!
> Here's how I think this works. Domain is lists.hostcompanion.com. List is 
> "address@concealed". Inbound email is a subscribe request via email, so 
> "address@concealed". Delivery and rewrite path goes like this 
> then:
>
>         • address@concealed comes in - Postfix knows it receives 
> "lists.hostcompanion.com" email from the main.cf line "virtual_alias_domains = 
> lists.hostcompanion.com"
>
>         • Postfix then checks if the user exists. The main.cf line 
> "virtual_alias_maps = 
> regexp:/usr/local/etc/sympa/postfix/lists.hostcompanion.com" tells it to check the 
> referenced file, which supports regex matching, to see if the user matches.
>
>         • In that regex file, this line matches: 
> "/^(test1)-(request|editor|owner|subscribe|unsubscribe)\@lists\.hostcompanion\.com$/ 
> $1+lists.hostcompanion.com@sympa$2.". The match is first on "test1", and the parens means to save 
> "test1" in $1. The "-" matches, and again, the parens tell it to save the match, this time in $2 (second 
> match). Lastly, the "@lists.hostcompanion.com" matches. This is therefore deliverable, and postfix accepts the 
> message.
>   
Yes.  This is pretty much standard regular expression grammar.
>         • That same regex file not only does a match to confirm the address is valid, 
> but it has rewritten it based on the "$1+lists.hostcompanion.com@sympa$2." 
> half of the regex.
>
>         • The address is now "test1+lists.hostcompanion.com@sympasubscribe." (note the 
> trailing ".")
>   
I think that the trailing . is not maintained internally.  It is only 
needed in the virtual alias right hand side format to suppress the 
addition of the local domain.  If you look at the output form 'sendmail 
-bv' shown in the howto you will see that the trailing dot is not 
present there after the mapping.
>         • The transports match comes next. The line "/^.*\@sympasubscribe$/ 
> sympasubscribe:" matches the "@sympasubscribe." that was the result of the rewrite.
>
>         • The transport match tells Postfix to hand this to the 
> "sympasubscribe" transport defined in master.cf.
>
>         • The master.cf line "flags=RF user=sympa 
> argv=/usr/local/libexec/sympa/queue ${user}-subscribe@${extension}" is used.
>   
Actually the whole of the two lines describing the transport (the 
dsecond line is just a continuation).
>         • master.cf is referencing "user" and "extension". "user" is the address to the left of the "+" in 
> our last rewrite (ie: test1) and "extension" is everything between the "+" and the "@" in the last rewrite (ie: 
> lists.hostcompanion.com).
>   
Note that this '+' separator is configurable.  '+' is the default.
>         • This means /usr/local/libexec/sympa/queue is called like so: 
> "/usr/local/libexec/sympa/queue address@concealed" (and 
> the queue has access to the envelope-from to know *who* to subscribe of course).
>
> I'm not so sure about the transport matching, and I'm also unclear on the 
> "sympasubscribe." domain and where postfix learns that it's local.  Other 
> than that, do I basically have the message flow down?
>   
See the address rewriting HOWTO mentioned above for the transport matching.
The pseudo-domains used in the transport are indeed peculiar.  The point 
here seems to be that all the checking of domains (like making sure we 
aren't doing relay for illegal domains) is done on the input queuing 
side.  The Virtual aliasing happens there.  The transport mapping 
happens on the output side and I *think* that no checking is done - it 
just matches.  I presume that Postfix accepts that domains arising out 
of its internal mapping process are 'kosher' and don't need any further 
checking.  And we don't actually want any aliasing etc to happen for 
these pseudo-domains, so they just don't need to appear in any of the 
checking or mapping configuration.  I guess the Postfix authors could 
have put in another configuration line, maybe 'other legitimate domains' 
but I think we are maybe explouting a 'feature;.
>   
> > As far as I can see, you are pretty much done.
> >     
>
> Let's hope so, this hurt my brain.
>   
That was my feeling back in 2008! Hence the long winded howto in case I 
forgot any of the hard earned integration of knowledge.

Regards,
Elwyn

PS
Thanks for the files.  I'll do a few updates to the HOWTO shortly.

/E
> Thanks,
>
> Charles
>
>   
> > You seem to think you have a problem here in that you have to paste this in 
> > manually..
> >
> >     
> > > > And finally, the hack that I stick in the top of that same file to handle the
> > > > bounces, posts, etc.:
> > > >
> > > > # this seems to be necessary, probably belongs somewhere else
> > > > /^(sympa)\@lists\.hostcompanion\.com$/
> > > > $1+lists.hostcompanion.com@sympalist.
> > > > /^(listmaster)\@lists\.hostcompanion\.com$/
> > > > $1+lists.hostcompanion.com@sympalist.
> > > >
> > > >         
> > Well you could do some extra stuff to generate this automatically with
> > the first instance of the list, but I didn't.
> >
> > Its a once per virtual host operation, so I just left it for manual
> > config.  I only really cared about making extra lists (I only had one
> > host to worry about so it wasn't a big deal).  You will have to do it
> > for each virtual host.
> >
> > This line is *NOT* required (and will probably screw things up)
> > /^(.*+owner\@lists\.hostcompanion\.com$/
> >     
> > > > $1+lists.hostcompanion.com@sympaowner.
> > > >
> > > >         
> > The aliases already cope with address@concealed etc
> >
> > One point which is not clear form your mail is that it is essential that
> > there are no extra newlines in the added lines.. if they are really like
> > this in the virtual host aliases file, then they won't work.  You need
> > something like:
> > /^(root|postmaster)\@lists\.hostcompanion\.com$/      
> > address@concealed
> > /^(sympa|listmaster)\@lists\.hostcompanion\.com$/     
> > $1+lists.hostcompanion.com@sympalist.
> >
> > where there is no newline after .com$/
> >
> > BTW
> > If you are willing to send me the updated conf.pm etc files, I'll add
> > them to the description so that people can use them with the latest
> > version.  I haven't had occasion to upversion my installation yet as it
> > does everything we needed (very simple requirements) and the password
> > improvement was not essential.  For our purposes the shared documents
> > folders were really helpful!
> >
> > Regards,
> > Elwyn Davies
> >
> > On Wed, 2011-09-07 at 09:14 +0200, address@concealed wrote:
> >     
> > > Hello,
> > >
> > > I'm having a tough time getting this setup properly for a non-profit I'm
> > > working with.
> > >
> > > I read up on Sympa's features and it looks like the right choice - it covers
> > > three things they need - web-based posting, discussion lists, and announce
> > > lists.  And it covers things I need like a decent permissions system and
> > > bounce processing.
> > >
> > > After reading about the various Postfix integration methods, this one seemed
> > > like it was the best in that it would not make me a backscatter source (the
> > > other options for virtual setups do rejects after the message enters the queue
> > > due to their greedy regex matching):
> > >
> > > http://www.folly.org.uk/sympa/sympa_config_03.html
> > >
> > > It's a nice setup, but the changes to Sympa since that was released left me
> > > having to patch up Conf.pm, alias_manage.pl, and confdefs.pm.
> > >
> > > This seems to work however - all my address@concealed mappings work,
> > > those aliases are generated automatically by the patched alias manager.  The
> > > Conf.pm changes are working fine (set flags in each virthost robot.conf).
> > >
> > > What I seem to be missing since I'm looking at so many Postfix tutorials, many
> > > of which are now outdated, is how to handle my address@concealed mappings.
> > > I can't find a decent reference for that.  Below is my current postfix setup,
> > > which seems to work, but I need to add a mapping manually for each host as
> > > noted in the config.
> > >
> > > Let's start with main.cf:
> > >
> > > #
> > > # SYMPA STUFF
> > > #
> > > transport_maps = regexp:/usr/local/etc/postfix/transport_sympa
> > >
> > > # Set the concurrency for delivery to Sympa
> > > sympalist_destination_recipient_limit = 1
> > > symparequest_destination_recipient_limit = 1
> > > sympaeditor_destination_recipient_limit = 1
> > > sympasubscribe_destination_recipient_limit = 1
> > > sympaunsubscribe_destination_recipient_limit = 1
> > > sympabounce_destination_recipient_limit = 1
> > >
> > > # Specify the virtual alias domain(s)
> > > virtual_alias_domains = lists.hostcompanion.com,lists.o4onyc.org
> > >
> > > # Specify the virtual alias file(s) - this is the only per virthost config
> > > virtual_alias_maps = regexp:/usr/local/etc/sympa/postfix/
> > > lists.hostcompanion.com,regexp:/usr/local/etc/sympa/postfix/lists.o4onyc.org
> > >
> > > Then the transport maps:
> > >
> > > # Transport map for Sympa virtual domain transports
> > > /^.*\@sympalist$/         sympalist:
> > > /^.*\@symparequest$/      symparequest:
> > > /^.*\@sympaeditor$/       sympaeditor:
> > > /^.*\@sympasubscribe$/    sympasubscribe:
> > > /^.*\@sympaunsubscribe$/  sympaunsubscribe:
> > > /^.*\@sympaowner$/       sympabounce:
> > >
> > > And master.cf:
> > >
> > > # Sympa mailing list manager transports
> > > sympalist     unix  -       n       n       -       -       pipe
> > >  flags=RF user=sympa argv=/usr/local/libexec/sympa/queue ${user}@${extension}
> > > symparequest     unix  -       n       n       -       -       pipe
> > >  flags=RF user=sympa argv=/usr/local/libexec/sympa/queue ${user}-request@
> > > ${extension}
> > > sympaeditor     unix  -       n       n       -       -       pipe
> > >  flags=RF user=sympa argv=/usr/local/libexec/sympa/queue ${user}-editor@
> > > ${extension}
> > > sympasubscribe     unix  -       n       n       -       -       pipe
> > >  flags=RF user=sympa argv=/usr/local/libexec/sympa/queue ${user}-subscribe@
> > > ${extension}
> > > sympaunsubscribe     unix  -       n       n       -       -       pipe
> > >  flags=RF user=sympa argv=/usr/local/libexec/sympa/queue ${user}-unsubscribe@
> > > ${extension}
> > > sympabounce     unix  -       n       n       -       -       pipe
> > >  flags=RF user=sympa argv=/usr/local/libexec/sympa/bouncequeue ${user}
> > >
> > > And now the aliases that are autogenerated for one virthost:
> > >
> > > ## This virtual aliases file for domain lists.hostcompanion.com is dedicated
> > > to Sympa Mailing List Manager
> > > ## You should edit your Postfix main.cf file to declare it
> > > ##
> > > /^lists\.hostcompanion\.com$/            xxx
> > > ##
> > > #------------------------------ test1: list alias created 06 Sep 2011
> > > /^(test1)-(request|editor|owner|subscribe|unsubscribe)\@lists\.hostcompanion
> > > \.com$/    $1+lists.hostcompanion.com@sympa$2.
> > > /^(test1)\@lists\.hostcompanion\.com$/
> > > $1+lists.hostcompanion.com@sympalist.
> > >
> > > And finally, the hack that I stick in the top of that same file to handle the
> > > bounces, posts, etc.:
> > >
> > > # this seems to be necessary, probably belongs somewhere else
> > > /^(sympa)\@lists\.hostcompanion\.com$/
> > > $1+lists.hostcompanion.com@sympalist.
> > > /^(listmaster)\@lists\.hostcompanion\.com$/
> > > $1+lists.hostcompanion.com@sympalist.
> > > /^(.*+owner\@lists\.hostcompanion\.com$/
> > > $1+lists.hostcompanion.com@sympaowner.
> > >
> > > I'm pretty lost here - any input is appreciated.
> > >
> > > Thanks,
> > >
> > > Charles
> > >       
>
>