The mailing list for listmasters using Sympa

["e.c." <address@concealed>]

Thanks, Steve. I think that answers my question about any possible
effects on other lists at the same site who don't want to use
rate-limiting. All the lists don't have to use all the scenarios,
right?

Ed

On Sat, Jun 28, 2014 at 8:35 PM, Steve Shipway <address@concealed> 
wrote:
> Using the -02 version, the $WINDOW seting is in the rate_limit.pm file (I 
> think -02 defaults to per day anyway).
>
> The CustomCondition line to limit to 60 per day for the list, and 6 per day 
> per subscriber, would be as below Note you always use [listname] regardless 
> of what the listname is, as this expands to the name!
>
>   CustomCondition::rate_limit([listname],"",60,[header->Message-ID]) 
> smtp,dkim,smime,md5 -> reject(tt2=rate_limit)
>   CustomCondition::rate_limit([listname],[sender], 6,[header->Message-ID]) 
> smtp,dkim,smime,md5 -> reject(tt2=rate_limit)
>
> Stick these into the send scenario for the list you're testing on and all 
> should work.
>
> I don't know if it would work on 6.1.7, since it depends on the 
> CustomCondition feature being available, and I don't know if that was 
> present in 6.1.7
>
> Steve
>
> Steve Shipway
> University of Auckland ITS
> UNIX Systems Design Lead
> address@concealed
> Ph: +64 9 373 7599 ext 86487
>
>
> ________________________________________
> From: e.c. [address@concealed]
> Sent: Sunday, 29 June 2014 1:33 a.m.
> To: Steve Shipway
> Cc: David Verdin; address@concealed
> Subject: Re: [sympa-users] Strict, per list, send limit...
>
> This looks like exactly what I want, or at least one of the things I
> want if the effects of these changes can be limited to one list. Is
> that true? Even if not, I would like to test this on 6.1.17. If I
> install sympa on a recent debian box can
> I test this by myself (using a few different email addresses)? For a
> limit of 6 per day per member and 60 per day per list is this what I
> want (if listname is grex):
>
>   CustomCondition::rate_limit([grex],[""],60,[header->Message-ID])
> smtp,dkim,smime,md5 -> reject(tt2=rate_limit)
>
> and
>
>   CustomCondition::rate_limit([grex],[sender], 6,[header->Message-ID])
> smtp,dkim,smime,md5 -> reject(tt2=rate_limit)
>
> or do I lose the brackets?
>
> Also, where do I set $WINDOW to 86400?
>
> Thanks,
>
> Ed
>
>
>
> On Thu, Jun 26, 2014 at 6:39 PM, Steve Shipway <address@concealed> 
> wrote:
>> Here is my solution to provide message rate limiting on a per-list or
>> per-subscriber-per-list basis.
>>
>>
>>
>> It works by adding a custom condition for send scenari that allows you to
>> reject a message if a given subscriber has sent more than a given number of
>> messages that hour.  Thanks to the suggestion from Giorgio, you can work
>> around the issue of caching, though it may be more memory-efficient to
>> simply disable the custom check caching by changing the verify_custom
>> subroutine in the  lib/Scenario.pm file like this:
>>
>>
>>
>> ---line 1273, Scenario.pm----
>>
>>         return undef unless defined $res;
>>
>>
>>
>> # Comment out these three lines
>>
>> #        $persistent_cache{'named_filter'}{$condition}{$filter}{'value'} =
>> ($res == 1 ? 1 : 0);
>>
>> #        $persistent_cache{'named_filter'}{$condition}{$filter}{'update'} =
>> time;
>>
>> #        return
>> $persistent_cache{'named_filter'}{$condition}{$filter}{'value'};
>>
>> # Add this one
>>
>>         return $res;
>>
>> }
>>
>>
>>
>> The custom condition creates a new table rate_limit in the Sympa database
>> which is used to hold subscriber/list posting counts.  This should not need
>> to be pruned except with public lists which could attract spam.
>>
>>
>>
>> You use a custom scenario line similar to this to make it work:
>>
>>
>>
>>   CustomCondition::rate_limit([listname],[sender],10,[header->Message-ID])
>> smtp,dkim,smime,md5 -> reject(tt2=rate_limit)
>>
>>
>>
>> The first two parameters are list name and sender; they are used to form 
>> the
>> key for counting.  To make a per-list count, use “” instead of [sender].  
>> If
>> you have multiple robots then replace [listname] with [listbame]@[robot].
>> The third parameter is the maximum messages per clock hour (reset count at
>> top of hour).  The fourth parameter is Giorgio’s trick to effectively
>> disable the Custom Condition caching for this function; you do not need 
>> this
>> if you modified Scenario.pm as suggested above.
>>
>>
>>
>> It will log information about the calling parameters, the count value found
>> in the database, and the updated value at debug2 level in the logs.
>>
>>
>>
>> HOW TO INSTALL
>>
>> 1.       Copy rate_limit.pm into the $SYMPAETC/custom_conditions directory
>> (create directory if you have to).  This makes the new function available 
>> to
>> scenari.
>>
>> 2.       Copy rate_limit.tt2 into the $SYMPAETC/mail_tt2 directory (create
>> directory if you have to).  This defines a custom notification message for
>> messages rejected due to rate limiting.  (If this mod is merged into the
>> main branch, I would expect this to be merged into 
>> authorization_reject.tt2)
>>
>> 3.       Create a Custom Scenario, such as $SYMPAETC/scenari/send.ratelimit
>> which has the custom condition.  An example is provided.  Note how we try
>> and exclude all non-posters before calling the Custom Condition in order to
>> minimise table size and to prevent per-list counts from being incorrect.
>>
>> 4.       Configure your list to use this custom scenario, and all should be
>> fine.
>>
>>
>>
>> Caveats:
>>
>> 1.       Only tested with Sympa 6.1.19
>>
>> 2.       Only tested with MySQL, likely doesn’t work with other databases
>>
>> 3.       The rate_limit table may grow large if scenario used on lists
>> permitting public posting due to spam attacks.
>>
>> 4.       The condition limits messages per hour, resetting the count at top
>> of hour (IE, at 12:00, 1:00, etc).  If you want per-day counts, modify the
>> $WINDOW setting in the plugin to be 86400.  Note that if you make this
>> change to daily, it will reset the count at midnight UCT regardless of your
>> local timezone.
>>
>> 5.       Do not call the test twice with the same parameters in the same
>> scenario.  Each time it is called it increments the count.  You can,
>> however, call it once with listname/sender to enforce a per-sender limit 
>> and
>> once with listname only to enforce a per-list limit.
>>
>>
>>
>> ***Please provide feedback.***
>>
>>
>>
>> If the Sympa team are interested in providing this functionality within
>> Sympa, I’d probably expect it to be done either as a new internal function
>> available to send scenari, or else as an option within list configuration
>> that is tested before the send scenario (around the same time as the 
>> message
>> size tests are run).  As it also requires a new database table to be added
>> it might be a bit more complex to merge in smoothly.  Fortunately, Sympa 
>> has
>> excellent hooks and API so it was not too hard to write a plugin.
>>
>>
>>
>> Steve
>>
>>
>>
>> Steve Shipway
>>
>> address@concealed
>>
>> University of Auckland, New Zealand