Developers of Sympa

[Olivier Salaün <address@concealed>]

Hi,

Eventually we've applied Charles Paul's patch in Sympa's development 
branch. Here is the patch : 
http://sourcesup.cru.fr/cgi/viewvc.cgi/trunk/src/List.pm?r1=4737&r2=4744

We've performed some tests and noticed significant performances 
improvements with the patch.

address@concealed a écrit :
> > Can you post your mods to this list ?
> > BTW, the first get_list call is to cache list configs in memory.
> > sympa used to store scenarios with every list, which is partly why it's using 
> > so much memory.
> >     
Seriy is right, the latest versions of Sympa that store scenarios data 
only once (in the memory) should help to have slimmer Sympa processes.
> It was a pleasure to see your response.  I have posted the diff to the sympa SourceSup at https://sourcesup.cru.fr/tracker/index.php?func=detail&aid=3219&group_id=23&atid=170 . Please tell me what you think of these changes.  
>
>
> As for the first call to List::get_lists inside of wwsympa - I know see how this will populate the %list_of_lists used by save and load. 
>
> I will experiment with using this %list_of_lists hash in combination with Cached::Memcached::Tie to see if it helps wwsympa performance.  It is my strong suspicion it will not, as the main culprit for fcgi performance seems to be the @all_lists structure returned by get_lists...  
>
>
> My current thinking for optimizing wwsympa.fcgi involve either: 
>
>   1. Implementing calls to get_lists which request specific lists to return (as below with the $userlifts argument).  Is this done best by making queries to a database table kept in sync with list configs on the filesystem -- or by merely creating and maintaining seperate %list_of_pending, and etc..?   
>   
There are probably some calls to List::get_lists() that will still 
require to load all list objects. However it is probably worth analyzing 
all of these calls and find out where optimizations can be done.
The other option you mention, storing part of the configuration files in 
the database, is also worth exploring. I can think of advantages and 
drawbacks for doing this :

   * advantages : allows to get the list of lists very fast, the
     response time being independent of the number of lists. However it
     is only worth storing those data from a list config file that are
     useful when querying all lists, ie listname, subject, topics,...
     Other list config data could remain in the list config file. On
     the other hand if were able to store list configuration in the DB,
     if would be a first step toward Sympa ML services provided by a
     cluster of servers.
   * drawbacks : maintaining the list configuration data (or part of
     it) in the database would oblige sympa processes to cope with
     synchronization of these data, because we still need to support
     list config flat files. If a list configuration is edited via the
     web interface or any other sympa process, then we can trigger an
     update of the DB. However, list server admins may either edit the
     list config files with emacs or via a home-made script. We need to
     provide a mean to trigger synchronization of the DB in such
     situations.

>   2. Developing a reasonable and efficient method for wwsympa to access the %list_of_list result cache.  Maybe by means of wwsympa accessing to a tie'd memcache share, rather than a conventional 200mb array.  Right now, list_of_lists is used only inside of List.pm -- it would be nice for wwsympa to have access to that SAME memory space.  Ideas?  
>
>   
That would be great indeed to share the same memory space among Sympa 
processes, however the technology used needs to be quite mature.
If you have experiences to go in this direction, please share it with us.
> The current method of loading the values from %list_of_lists into an array is, to put it bluntly, puting the servers through much pain.  
>
> Thanks again for your response Serge.  I totally appreciate your attention to 
> this matter!
>