Developers of Sympa

[Olivier Salaün - CRU <address@concealed>]

Hi Sergiy,

You're right, the figures we've got show it should not be memory leaks problem.

It is also undoubtly related to the number of lists managed and therefore it is probably related to :

• either data structures we use for the list (configuration files, autorization scenarios,..)
• or function calls related to each list that would allocate memory (DBI context while performing SQL queries for example)
  

We will try to correlate the number of scenario rules with the amount of memory used. If that's the problem, we'll change the way we manage scenario rules in memory : currently these data are stored in each list object data structure ; we would consider sharing these data among all lists.

If your hypothesis regarding sympa.pl process is right, then sending a 'WHICH' command to sympa.pl should make the process as big as wwsympa.fcgi and task_manager.pl.


Thank you for feeding these reflexions on Sympa optimizations.

Sergiy Zhuk wrote:

On Thu, 6 Oct 2005, [ISO-8859-1] Olivier Sala?n - CRU wrote:
  

   4. the wwsympa.fcgi process is somehow fat (around 200 Mb for 2000 lists)
    

Same with task_manager.pl which is somehow even bigger.
With about 8,500 lists, each fastcgi process takes about 512Mb of RAM, while task manager is at 530Mb or so.
  

(4) we didn't find out why wwsympa.fcgi processes might be that big. But maybe it's acceptable if wwsympa.fcgi has good performences because a few of them would be enough then.
    

AFAIR the only difference between task manager, cgi and the rest of processes is that task manager and cgis touch all the lists, while the rest of processes only cache data for the ones they are delivering mail to or processing bounces etc.
That means we're gonna have the same problem with them if all of the lists are active (receiving mail that is).
Say sympa.pl process is about 400Mb on my system now and it's growing...
I think something is wrong with how sympa is handling memory allocation, since it looks like it takes an incredible amount of RAM per list, e.g. the average config file with all extra spaces and formatting still takes about 1kb and the whole sympa database is about 25Mb only.
I don't understand how that could lead to 512Mb usage. What makes it even more interesting is that all that memory is gone right on start-up, when we're scanning all configs and such.
It doesn't change much after that, meaning we can't blame it on memory leaks.
  

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature