The mailing list for listmasters using Sympa

[Christian Dahlhausen <address@concealed>]

Roy, 

we run about 16000 lists here at UVa on Sympa 5.4.3 and 
needless to say the web interface is pretty slow. As you mentioned this 
is because the wwsympa cgi crawls through every list config multiple 
times. I did have a test machine (6.1.4) where I added some of the 
riseup.net patches and got to good results. Of course that hasn't made 
it to production so I don't really know how good that works but it 
seemed promising. I am in the process of upgrading to the latest 6.1.7 
version right now and will try again to implement the riseup patch in 
the new server and run some tests before it might make production. I am 
aware that deviating from the standard code might bite me in the next 
iteration of an upgrade but our users are moaning too much about the 
sluggishness of the web interface. 
Christian

On Tue, Jan 3, 2012 at 1:49 PM, Roy Bixler <address@concealed> wrote:

Hello:

I posted earlier over the holidays regarding possible patches to improve Sympa's scalability in cases like ours where we have several thousand lists. I am basically re-posting since I only got one responce and I think that may be because my timing was a bit off and many people many have been away.

My original post, explaining our situation in more detail, is below. To summarise, we are experiencing some poor performance on the Web interface using Sympa 5.4.7 and the binary config files option and are wondering about the best way to improve it most consistent with future Sympa development. I believe it is because the wwsympa.fcgi must crawl the many thousand config files for configuration information.

Some of the options which looked promising are the use of Perl's tie function described here:

[#4787] Memory Optimization? An idea, tie @lists in List::get_lists to a helper package class

https://sourcesup.cru.fr/tracker/?func=detail&atid=170&aid=4787&group_id=23

the use of "memcached" (http://www.danga.com/memcached/) and putting Sympa configuration information in the database. The latter 2 items are described in the Sympa future development page (https://www.sympa.org/dev/project_direction). For the latter, the riseup.net patch available for Sympa v.6.0.6 (https://labs.riseup.net/code/projects/sympa/repository/revisions/master/changes/6.0.6/patches/01_list_db_caching) looks promising. My overall question is which of these options fits best with the current direction in Sympa development?

Also, on the Sympa project direction page under the "web interface with many lists" section, I note that the following lines are crossed out:

sharing memory storage of each scenario (currently duplicated for each list)

optimize and limits number of SQL request

do_log subroutine calls might be expensive

List::new calls might be expensive (example: testing the date of config files)

Does this mean that these have already been explored and upgrading to the latest version (6.1.7 now, I suppose) would help us?

(By the way, it seems that the sympa-dev list, where I originally posted this has been closed now. If there is a better list where I should pose these questions, I'm open to suggestions.)

Thanks and Happy New Year!

On Thu, Dec 22, 2011 at 06:15:13AM -0600, Charles Paul wrote:

> Forwarding this email from sympa-users to sympa-dev. Roy, you may

> wish to consider joining sympa-dev to discuss scalability issues. I

> helped write the Riseup patches.

Thank you for that.

It looks like Google mangled my post a little, so I'll add spaces back

to improve readability.

> """

>

> Hello:

> I am new to this list and have some questions about possible patches we

> could apply to improve Sympa scalability. The items of most concern are

> responsiveness of the Web interface and also the SOAP interface. The

> latter has only become a requirement in the past month. One of our

> developers has written a portal application and he hopes to call the

> SOAP interface to do things such as create a new list or list

> subscribers of an existing list.

> What we have noticed is that, at least on the initial loading of

> the server suite, the programs must crawl the disk for

> configuration information of all of the lists. We currently have about

> 6000 lists, so this can take anywhere from about 3 minutes for a local

> disk configuration to about 15 minutes if we use NFS. I think that

> FCGI tuning may alleviate some of these problems.

> I have also looked into applying some patches which may help to

> rely less on crawling the disk for configuration data. First, we have a

> 2 node environment. One node is at version 5.3.4 and runs the

> Sympa suite to process mail and the other node is at version 5.4.7 and

> runs wwsympa.fgci for the Web interface. The data is shared between the

> 2 via NFS and we also use a remote MySQL database.

> We used to suffer from slow Web performance, especially when

> listing all of the lists, but we improved that by using binary

> configuration files and by cutting down on the size of the session

> tables. Now that we have to concern ourself with the SOAP interface as

> well, I wonder if we can make further improvements on performance for

> operations like listing the lists and and listing the subscribers of

> lists.

> I have looked at the available patches and see that there is one

> which uses the Perl "tie" feature to improve caching or that there

> is another larger patch from riseup.net which stores more

> configuration information in a database. I also note a proposal to use

> "memcached" as a cache of list configuration. It seems like the

> riseup.net patch is most promising, but I suppose that the others might

> be acceptable if the loading of list configuration data is only slow on

> startup. I mainly wonder which of these approaches seems the most

> consistent with future versions of Sympa. Also, if anyone has some

> pointers on FCGI tuning or other comments, that would be welcome.

> Thanks,

> --?Roy Bixler?< address@concealed>IT Services / Collaborative Systems"""

>

--

Roy Bixler <address@concealed>

IT Services / Collaborative Systems

-- 
-----------------------------------------------------------

Christian Dahlhausen, Network Systems Engineer
University of Virginia - ITS Infrastructure Applications
PO Box 400324, 2015 Ivy Road, Charlottesville, VA 22904