Developers of Sympa

[Sergiy Zhuk <address@concealed>]

hi

On Fri, 6 Sep 2002, Aumont wrote:

> developpement server. We will have a lok at it.On thinl that could
> be a matter for 1,600 listes is the size of expl directory.
> May be it could be usefull to trace all readdir operation in sympa.

I've looked at it some more and found out that this is not a directory read
matter, but a matter of caching and working with configs in plain text files
rather than storing them in a database.
get_lists() works fast enough, I tried replacing readdir with a read from 
a plain text file, and performance was the same.
The performance bottleneck is the storage and the caching we have to deal
with instead of using database select with needed attributes.

Take a look at do_search_list() in wwsympa.fcgi.

foreach my $l ( &List::get_lists($robot) ) {
    my $list = new List ($l);

So, we're updating the cache including stats, which we don't really need for
list search and some other ops, for every list even if we know that we only 
need lists with certain subject or topic.
I think doing 'select' instead of updating cache (every config option) for
every possible list would be much faster.
There's also an issue with parsing and writing config files properly, which
could be solved if we kept them in the database as separate fields.
Also, cached data takes up a lot of memory.
I currently have about 1,800 lists and sympa processes take about 100Mb or
RAM each.
Simply getting rid of cache won't help, since it's gonna be slower if we
don't change the way we work with data, but optimizing the algorithms to use
select with a bunch of options instead of going thru the whole bunch of lists,
will definitely help.
There's another disadvantage of using cache in its present form for list
data - we can't effectively share it among sympa processes, so instead of 
having one cache and an interface to that cache, we have a separate cache 
for every sympa process.
I guess mmap() would be faster even than using mysql select, so if we at
least used a common cache, updated by a separate process, everything would
be faster.
I didn't check if there's a perl interface we could use to get advantage of
mmap() though...
What do you think about storing configs in the database ?
We could keep a plain text copy in the list's directory, but would update it
only once in a while and also keep backup configs there.
I think there's virtually no need to edit configs manually, since one can
either use email or web interface to do it.

thanks

-- 
rgds,
serge