Developers of Sympa

[Olivier Salaun <address@concealed>]

Hi,

In November 2001, Harald Wilhelmi provided us a patch for managing Sympa's 
included
lists in the database (MySQL or another) instead of using DB Files. This had 
the following
interests :
        + making Sympa processes smaller when managing big «include» type of 
lists.     
        + allowing mixed lists (included + subscribed users)
        + allowing reception options for «included» users, as well
        as bounces management.

We've integrated this patch and developped these features into Sympa's current
development version. It is currently running on our own server and it is now 
available for download, so you can test it :
        http://www.sympa.org/distribution/beta/

Attached is a description of this new 'include2' feature.

Thanks.

--
Olivier Salaün
Comité Réseau des UniversitésNew 'include2' user_data_source ; should replace 'include'.
Users cache is stored in the main database instead of using a 
DB_File. Updates of the cache are performed by the task_manager

* Advantages

        + lower memory usage :
        Sympa processes used to use lots of memory when managing
        big (>200 000) 'include' types of lists. The list of 
        subscribers is now stored in the database.

        + updating the cache :
        Cache update is now performed by the task manager. It used to
        be performed by sympa.pl and wwsympa.fcgi.

        + bounces management : 
        Because included users now have an entry in subscriber_table,
        their bounces are managed like any other subscribed users

        + user reception options :
        Included users can set their reception mode

* Subscriber_table changes

The subscriber_table Db table has 3 new fields :

        * included_subscriber
        This boolean value tells if the user has been included into
        the list

        * subscribed_subscriber
        This boolean value tells if the user has subscribed to the
        list

        * include_sources_subscriber
        Comma-separated list of include source IDs. It is not 
        exploited yet, but could be used to :
                - set different TTL for each include data_sources
                - show include sources in the REVIEW pages
                - tell the user he/she has been included as part
                of X,Y external datasources

* Task manager 

New 'sync_include' list task : 
Only one model (ttl) is available. TM checks if the list is
in 'include2' mode and runs the sync_include. The syn_include()
does an in-memory comparison of already cache users list and
included users list ; subscriber_table is then updated. 
List::delete_user() and List::add_user() have been optimized
to make sync_include() faster.
A new task is created based on 'ttl' list parameter.

New 'purge_user-table' global task :
Each time a user unsubscribes the user_table had to be cleaned ;
this was time-consuming. This is now performed periodically
by the 'purge_user-table' task. 


* Changes of commands behavior

Reception options 
Included users can now set their reception mode

Subscription
Included users can moreover subscribe to the list (if allowed by scenario).
It allows them to remain in the list though they might no more be included.

Review (web only)
Foreach user, tell if he/she is included or subscribed


* Locking while updating cache

An exclusive lock is performed by sync_include() to prevent other processes
from accessing an uncomplete list of users. A shared lock is set in 
List::get_first_user(), then released for the last List::get_next_user().
File locking is performed on ~sympa/expl/<list>/include.lock file

* Stats file 

A new value has been added to lists 'stats' file : last_sync
I is the date (epoch format) of the last include cache update.

* List configuration edition

When the list configuration has been changed (from the web), then :
        * if we changed a datasource-related parameter
        => the current 'sync_include' task is removed from the task
        spool, to make cache update happen as soon as possible.

        * if user_data_source changed from 'database' to 'include2'
        => existing subscribers have their 'subscribed_subscriber' DB
        field set to '1'

        * if user_data_source changed from 'include2' to 'database'
        => existing included users are purged from subscriber_table

* TODO

        + sending messages when users are included/decluded (should be an 
option)
        + migrating from 'include' to 'include2'
        + showing user datasources within REVIEW page
        + adding "sort by datasource" in REVIEW page