Developers of Sympa

[IKEDA Soji <address@concealed>]

Hi,

First, I postpone planned change on bulk spool to sympa-6.2b.2,
because, I didn't suppose this issue is controvasial and didn't kept
the time for discussion.  Now there isn't enough time to continue
change.

Besides,

On Wed, 10 Sep 2014 10:30:41 +0200
David Verdin <address@concealed> wrote:

<<snip>>

> >> - 6.2: I want it out as soon as possible. Otherwise, Sympa may die.
> >> that's as simple as this. If I can't say by the beginning of October
> >> that we finally issued a beta, my management will cut deep in our Sympa
> >> ressources. So no new features, no big changes. I'll agree with Soji's
> >> proposal about bulk spool if he makes a strong argument about it.
> > You said 6.2 would be released "in two weeks".  Just a kidding!
> > It is impossible even if you were Jacques Anquetil!
> Ah Jacques anquetil... We never had such a good cyclist in France - at 
> least without drugs.
> I was speaking about 6.2 beta, obviously, not 6.2 stable.
> 
> I'll answer about your scenarios later, however I have a few questions 
> popping about moving bulk spool to filesystem:

See attached text file for details (my private memo).  I'll write
only summary here.

> - do you intend to move both tables (bulkspool and bulkmailer) or only 
> bulkspool?

(a) One file by each packet.
(b) One file by each message, and other file(s) by each packet.
(c) One file by each message, and another directory with the same
  name holding files by each packet.

Current implementation is similar to (b).  I prefer to (a) because
of easiness to implement.

Problem of (a) is the space.  Others won't increase space (rather
decrease in particular case).

> - how will you handle the metadata attached to mails?
> - how will you handle message priority and packets priority, as well as 
> delivery time and such?

Basically, I based on my proposal at 30 June:
  https://listes.renater.fr/sympa/arc/sympa-developpers/2014-06/msg00048.html

Metadata and attribute can replace all DB fields in current
bulk tables.


> - if you move bulkmailer, don't you fear to have difficulties with the 
> readdir necessary to parse the bulk? Sometimes, you can have tens of 
> thousands of packets awaiting distribution. That implies a long readdir.

In principle, DB accesses in current code (using LIMIT clause or
equivalent) is similar.

On the other hand, directory entries may be cached and reloading can
be reduced.  See pseudo-code in attachment.

                                *

> Maybe you could keep tables just for metadata and keep only receipients 
> and messages on the filesystem.

No, metadata (by meaning of my proposal) is encoded in file name
in the spool, if spool is based on filesystem.


> Well, just so we're clear about advantages / disadvantages of moving spools.

Please note:

Currently I don't consider of, say, short-term advantage (such as
useful viewspool feature).

My personal goal for the time being is refactoring on spools/
pipelines.

In other words, the advantage I mean is to handle spools and
workflows with uniform interfaces.  Once such interfaces were
implemented, it is much easier to replace storage with other
backend.

Conversely, disadvantage I mean is to keep disunited interfaces
by accumulating makeshift changes.  I hardly agree to the argument
such as "we can do it by adding one more field to xxx_table".


Regards,

--- Soji

> Cheers,
> 
> David

<<snip>>

-- 
株式会社 コンバージョン  セキュリティ&OSSソリューション部   池田荘児
〒231-0004　神奈川県横浜市中区元浜町3-21-2 ヘリオス関内ビル7F
e-mail address@concealed  TEL 045-640-3550
http://www.conversion.co.jp/

Attachment: bulk_changes.txt.us-ascii
Description: Binary data