Developers of Sympa

[Charles Paul <address@concealed>]

Forwarding this email from sympa-users to sympa-dev.  Roy, you may
wish to consider joining sympa-dev to discuss scalability issues.  I
helped write the Riseup patches.

My instinct is to throw the configuration hash into a modern document
store and not have to think about it too much again.  The memcached
route, iirc, was dependent on an efficient scheme to map perl hashes
to a key/value store like memcached.  Do you have an url to the
memcached proposal?

Thanks!

"""

Hello:
I am new to this list and have some questions about possible patcheswe
could apply to improve Sympa scalability. The items of mostconcern are
responsiveness of the Web interface and also the SOAPinterface. The
latter has only become a requirement in the pastmonth. One of our
developers has written a portal application and hehopes to call the
SOAP interface to do things such as create a newlist or list
subscribers of an existing list.
What we have noticed is that, at least on the initial loading of
theserver suite, the programs must crawl the disk for
configurationinformation of all of the lists. We currently have about
6000 lists,so this can take anywhere from about 3 minutes for a local
diskconfiguration to about 15 minutes if we use NFS. I think that
FCGItuning may alleviate some of these problems.
I have also looked into applying some patches which may help to
relyless on crawling the disk for configuration data. First, we have a
2node environment. One node is at version 5.3.4 and runs the
Sympasuite to process mail and the other node is at version 5.4.7 and
runswwsympa.fgci for the Web interface. The data is shared between the
2via NFS and we also use a remote MySQL database.
We used to suffer from slow Web performance, especially when
listingall of the lists, but we improved that by using binary
configurationfiles and by cutting down on the size of the session
tables. Now thatwe have to concern ourself with the SOAP interface as
well, I wonderif we can make further improvements on performance for
operations likelisting the lists and and listing the subscribers of
lists.
I have looked at the available patches and see that there is one
whichuses the Perl "tie" feature to improve caching or that there
isanother larger patch from riseup.net which stores more
configurationinformation in a database. I also note a proposal to use
"memcached"as a cache of list configuration. It seems like the
riseup.net patchis most promising, but I suppose that the others might
be acceptableif the loading of list configuration data is only slow on
startup. Imainly wonder which of these approaches seems the most
consistent withfuture versions of Sympa. Also, if anyone has some
pointers on FCGItuning or other comments, that would be welcome.
Thanks,
-- Roy Bixler < address@concealed>IT Services / Collaborative Systems"""