Developers of Sympa

[IKEDA Soji <address@concealed>]

All,

I made change on bulk spool to switch from database to filesystem
(see r11556).

> > - do you intend to move both tables (bulkspool and bulkmailer) or only 
> > bulkspool?
> 
> (a) One file by each packet.
> (b) One file by each message, and other file(s) by each packet.
> (c) One file by each message, and another directory with the same
>   name holding files by each packet.
> 
> Current implementation is similar to (b).  I prefer to (a) because
> of easiness to implement.
> 
> Problem of (a) is the space.  Others won't increase space (rather
> decrease in particular case).

I chose (a).  If this method has big problem, I'll change it to (c).
Comments are appreciated.

Regards,

--- Soji


On Thu, 11 Sep 2014 12:17:51 +0900
IKEDA Soji <address@concealed> wrote:

> Hi,
> 
> First, I postpone planned change on bulk spool to sympa-6.2b.2,
> because, I didn't suppose this issue is controvasial and didn't kept
> the time for discussion.  Now there isn't enough time to continue
> change.
> 
> Besides,
> 
> On Wed, 10 Sep 2014 10:30:41 +0200
> David Verdin <address@concealed> wrote:
> 
> <<snip>>
> 
> > >> - 6.2: I want it out as soon as possible. Otherwise, Sympa may die.
> > >> that's as simple as this. If I can't say by the beginning of October
> > >> that we finally issued a beta, my management will cut deep in our Sympa
> > >> ressources. So no new features, no big changes. I'll agree with Soji's
> > >> proposal about bulk spool if he makes a strong argument about it.
> > > You said 6.2 would be released "in two weeks".  Just a kidding!
> > > It is impossible even if you were Jacques Anquetil!
> > Ah Jacques anquetil... We never had such a good cyclist in France - at 
> > least without drugs.
> > I was speaking about 6.2 beta, obviously, not 6.2 stable.
> > 
> > I'll answer about your scenarios later, however I have a few questions 
> > popping about moving bulk spool to filesystem:
> 
> See attached text file for details (my private memo).  I'll write
> only summary here.
> 
> > - do you intend to move both tables (bulkspool and bulkmailer) or only 
> > bulkspool?
> 
> (a) One file by each packet.
> (b) One file by each message, and other file(s) by each packet.
> (c) One file by each message, and another directory with the same
>   name holding files by each packet.
> 
> Current implementation is similar to (b).  I prefer to (a) because
> of easiness to implement.
> 
> Problem of (a) is the space.  Others won't increase space (rather
> decrease in particular case).
> 
> > - how will you handle the metadata attached to mails?
> > - how will you handle message priority and packets priority, as well as 
> > delivery time and such?
> 
> Basically, I based on my proposal at 30 June:
>   
> https://listes.renater.fr/sympa/arc/sympa-developpers/2014-06/msg00048.html
> 
> Metadata and attribute can replace all DB fields in current
> bulk tables.
> 
> 
> > - if you move bulkmailer, don't you fear to have difficulties with the 
> > readdir necessary to parse the bulk? Sometimes, you can have tens of 
> > thousands of packets awaiting distribution. That implies a long readdir.
> 
> In principle, DB accesses in current code (using LIMIT clause or
> equivalent) is similar.
> 
> On the other hand, directory entries may be cached and reloading can
> be reduced.  See pseudo-code in attachment.
> 
>                                 *
> 
> > Maybe you could keep tables just for metadata and keep only receipients 
> > and messages on the filesystem.
> 
> No, metadata (by meaning of my proposal) is encoded in file name
> in the spool, if spool is based on filesystem.
> 
> 
> > Well, just so we're clear about advantages / disadvantages of moving 
> > spools.
> 
> Please note:
> 
> Currently I don't consider of, say, short-term advantage (such as
> useful viewspool feature).
> 
> My personal goal for the time being is refactoring on spools/
> pipelines.
> 
> In other words, the advantage I mean is to handle spools and
> workflows with uniform interfaces.  Once such interfaces were
> implemented, it is much easier to replace storage with other
> backend.
> 
> Conversely, disadvantage I mean is to keep disunited interfaces
> by accumulating makeshift changes.  I hardly agree to the argument
> such as "we can do it by adding one more field to xxx_table".
> 
> 
> Regards,
> 
> --- Soji
> 
> > Cheers,
> > 
> > David
> 
> <<snip>>
> 
> -- 
> 株式会社 コンバージョン  セキュリティ&OSSソリューション部   池田荘児
> 〒231-0004　神奈川県横浜市中区元浜町3-21-2 ヘリオス関内ビル7F
> e-mail address@concealed  TEL 045-640-3550
> http://www.conversion.co.jp/


-- 
-- 
株式会社 コンバージョン  セキュリティ&OSSソリューション部   池田荘児
〒231-0004　神奈川県横浜市中区元浜町3-21-2 ヘリオス関内ビル7F
e-mail address@concealed  TEL 045-640-3550
http://www.conversion.co.jp/