The mailing list for listmasters using Sympa

["Steve Sobol - NTF" <address@concealed>]

Correction: archived is receiving SIGTERMs, not SIGKILLs.

-----Original Message-----
From: address@concealed <address@concealed> On Behalf Of
Steve Sobol - NTF
Sent: Monday, April 29, 2024 11:22 AM
To: Sympa EN mailing list <address@concealed>
Cc: address@concealed
Subject: [en@sympa] Can't import old messages from Mailman into Sympa
archive (out of memory? Not sure)

Weird problem on Debian 11 - Sympa was installed from packages from the
official Debian repo:

I have a mailing list hosted on our Sympa server. From mid-2022 until about
a month ago, it was hosted on a server running mlmmj. Moving archives from
mlmmj is pretty simple, as the messages are already stored in the format
Sympa requires (plain text, full headers, one file per message).

The archives that were generated when the list was hosted on mlmmj are
already moved over, and are visible in the web UI.

But prior to mid-2022, we have 17 or 18 years of archives from the time the
list was hosted on a Mailman server.

Getting the archives into the right format is not a problem. Mailman stores
entire months in a single file. It's just a question of copying each message
into its own text file, and un-munging the envelope and From: headers
(changing "user at domain.name" to "address@concealed"). I have scripts that
do that, and put the messages into a folder named YYYY-MM. (Actually, the
messages go into YYYY-MM/arctxt).

I manually copy YYYY-MM to /address@concealed.

From there, all I should need to do to complete the task is run "sympa
rebuildarc address@concealed".

And this worked, for archives from 2003, 2004, 2005, 2006 and 2007. At that
point, I had archives from those years, plus 2022-2024.

Then something weird happened. With log level set to 2, I could see Sympa
processing the emails for the years I'd already imported, but then
archived.pl died, apparently killed by something sending a SIGKILL.

There weren't any useful clues in sympa.log or syslog, but thinking that
maybe systemd might be killing archived.pl due to low memory, I tried
increasing the RAM allocated to the VM from 2 to 4 GB.

That got me a little further - I now have the first 11 months of 2008 in the
archive. But now I'm running into the same problem. And I still have to
import archives from December of 2008, plus archives from every year between
2009 and 2021 (inclusive). I can't keep throwing more RAM at the VM.

Are there any good solutions to this problem? Can I tell Sympa to only
rebuild the archive for a certain year?

Thanks
--Steve