The mailing list for listmasters using Sympa

[Steve Shipway <address@concealed>]

Steve Rich wrote:

>We currently have 5.4 running on one VM with 2 cores and 4GB of RAM.

> This machine hosts 8,224 lists  with 830,895 total subscribers (386,845 unique).

> At this time, I cannot speak to the activity of those lists and subscribers.  We're

>working to get relevant message delivery rates at this time but don't have them yet.

> 

>Has anyone on this list had success with Sympa 6.1.x at this scale?  If so, what does

>your Sympa architecture look like?  What unique challenges did the upgrade present

>and how did you overcome them.

 

We are running 22,842 lists currently, over 14 robots in one list server.  It has 4 cores and 12GB ram, though this is a bit excessive and it could likely work fine with 8GB.  The high memory use is due to mysql tuning (see later).  We handle an average of only about 12,000 messages per day through this server.  In comparison I believe the Riseup sympa server (the largest I’ve heard of so far) has possibly fewer lists, but a much higher mail throughput.

 

The two things that can affect at this scale are the number of lists, and the message throughput.   A large number of lists can heavily affect the performace of the web interface, particularly with searched; you need to have at least 6.1.14 to get the ‘Riseup patch’ which makes things work better; there is also a supplementary mysql-specific patch that we’ve added to further address performance of web searches by list name (this is not in the main release because it uses the RLIKE function, which is only in MySQL)

 

In both cases, the database is very important; we specifically tuned the MySQL instance to grant a lot of memory to caches and buffers, which made a significant difference to performance of the web frontend.  Having the database (and queues) on fast disk will certainly help if you anticipate a high message volume, as well.

 

When we upgraded to Sympa, we moved several other mailing list servers (mailman, aliases, and other systems) into the one Sympa instance as separate Robots.  This was made easier by having the default domain for the list server as pretty much unused, and then subsequently adding all the migrated list domains as additional robots.  Problems kicked in with the large number of lists, until we applied the previously mentioned Riseup patch, and then things further improved with the database tuning.  There were some migration issues caused by the migration tool having bugs or shortcomings but nothing significant.  We spent a fair amount of time planning the migration and upgrade, plus a lot of time training users (list owners and admins, as well as listmasters) and it paid off in less problems and a greater use of the new features available.

 

As far as performance goes, now that we’ve made the necessary MySQL tuning and patch to Sympa, everything is fine, and with this much memory we’ve a lot of spare capacity.  Our CPU use peaks at around 25% with a 95^th percentile of 18%; the active memory measure averages around 20% with an occasional peak at 50% and 95^th percentile of 26%.  We’re using normal virtualised SAN disk for the queue and database; this could be speeded up in the future if necessary using faster underlying disks, or SSDs, though currently there’s no need for it as our message throughput is not high enough.

 

Feel free to contact me off-list if you’ve any specific questions.  I’d also suggest trying to get some tips from the Riseup.net admins if you anticipate a high volume of messages.

 

Steve

 

 

Steve Shipway

address@concealed

 

Attachment: smime.p7s
Description: S/MIME cryptographic signature