The mailing list for listmasters using Sympa

[Matt Taggart <address@concealed>]

Hi Jérôme,

On 6/24/21 8:53 AM, David Carter wrote:
> On 24/06/2021 14:17, Jérôme Pouilloux wrote:
> > Dear,
> >
> > We have an instance of sympa (6.2.60)  with 15 000 lists created. Some 
> > queries like a search in the list of the queries take a very long time 
> > and may end with a 503 error.

We run a server with ~20k lists and had to do lots of tuning initially.

> > Do you have any idea to increase performance of the sympa UI ?
>
> I found that: "db_list_cache on" made a substantial difference to search 
> operations. That isn't on by default.
>
> We also hide /wws/lists as that takes a lot of effort to render.

We do both of those things.

Like Racke said, I think your system needs more resources. Do you have a 
way to graph usage? munin or grafana or something? that was how we 
figured out a lot. Probably giving it more will be enough.

Over the years we have done a bunch of things to improve scaling. When 
we first started some of these things were absolutely necessary to 
prevent that era's hardware from being completely overloaded with our 
workload. These days these things might not be required with current era 
hardware, but they might be later options if you need them, so here are 
some notes that might be helpful:

We moved a lot of the smtp effort to systems we call "outmx" hosts.

The server running sympa receives all list mail, but when sending things 
out they are relayed to a bunch of VMs to do the actual sending.

This both offloads some work from the main sympa machine, but also was 
needed in our case to A) deliver things in a timely manner by 
parallelizing the workload B) reduce the damage if a sending IP were to 
get blocked C) allows us to do some prioritization.
We have two sets of them, one for our largest lists and one for 
everything else, to help prevent large lists slowing down small lists. 
We also use the "priority" setting to have sympa itself control what 
order things go out.

The outmx hosts themselves each don't use that much CPU/RAM but we have 
a bunch of them and if the sympa host had to do all that it would add up.

Our sympa host averages about 1.5 CPU cores all the time, with 
occasional spikes up to 5-6 cores. The system has 64G of RAM, but most 
of the time 35G of that is being used for buffers/cache.

We also run clamav/spamassassin on the host, so not all of that is 
sympa, but I suspect your 2 CPU/6g RAM won't be enough.

Even with all that, certain operations on large lists still timeout in 
the web UI, in particular closing and reopening lists with tens of 
thousands of subscribers. When I run into one of those I sometimes 
manually adjust the apache timeouts, or I use a sympa command line to do 
what I want.

I hope this helps, feel free to post more questions, likely people here 
have run into the same.

--
Matt Taggart
address@concealed