Developers of Sympa

[IKEDA Soji <address@concealed>]

David and all,

On Fri, 08 Nov 2013 10:25:58 +0100
David Verdin <address@concealed> wrote:

> Dear Soji and all,
> 
> Le 08/11/13 06:36, IKEDA Soji a écrit :
> > David and all,
> >
> > On Thu, 07 Nov 2013 15:55:00 +0100
> > David Verdin <address@concealed> wrote:
> >
> >> Hi Soji,
> >>
> >> Le 07/11/13 01:07, IKEDA Soji a écrit :
> >>> On Wed, 06 Nov 2013 16:45:05 +0100
> >>> David Verdin <address@concealed> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Le 04/11/13 17:21, Guillaume Rousse a écrit :
> >>>>> Hello.
> >>>>>
> >>>>> Le 03/11/2013 03:50, IKEDA Soji a écrit :
> >>>>>> Hi,
> >>>>>>
> >>>>>> Guillaume's new directory layout seems to have a bit more room for
> >>>>>> improvement.  For example subdirectories under src/ may be moved to
> >>>>>> top level.
> >>>> Are you talking about the trunk or the 6.2 branch?
> >>>>>> Also expecting registration to CPAN, I drafted a modified layout.
> >>>>> The desirability of such registration has never been discussed sofar.
> >>>>> Altough I don't have any actual objection, I don't see much interest:
> >>>>> - sympa does not belong to the primary target of CPAN (it is an
> >>>>> application, not a reusable perl library)
> >>>>> - sympa doesn't use ExtUtils::MakeMaker or Module::Build build system,
> >>>>> meaning cpan clients won't be able to install it
> >>>>>
> >>>>> And even if desirable, there is no constraint on internal archive
> >>>>> structure.
> >>>> To sum up: we need to have a correct modularization and get rid of
> >>>> circular references.
> >>>> If this leads to porting some modules to CPAN, it is potentially
> >>>> interesting. But this is not our current priority.
> >>> "_Also_ expecting registration to CPAN", I wrote.  Expecting it
> >>> just results the name of a few directories.
> >> I think we agree on this. I just wanted to catch the occasion to precise
> >> my exact thought about CPAN. I'm all for moving modules to CPAN if it is
> >> udeful to the community.
> >>> My proposal _also_ moved some files and directories.  Do you say
> >>> they are unnecessary (or misled) changes?  Please look at table in
> >>> the quote below.
> >> I may have andswered to quickly.
> >> I like your repartition which looks like the one Guillaume did in the
> >> sympa-cleanup branch, except that it is not contained in a single
> >> src/Sympa directory.
> >>
> >> Let's have a look a the repartition Guillaume did for sympa-cleanup
> >> (simplified version):
> >>
> >> sympa-cleanup/
> >> |-- several files to be read: AUTHORS, COPYING, ChangeLog, INSTALL,
> >> NEWS, README, README.charset
> >> |-- some scripts: important_changes.pl, sympa.spec.in, tap-driver.sh
> >> |-- *doc/*: doc
> >> |-- *ext/*: external modules
> >> |-- *mail_tt2/*: mail templates
> >> |-- *t**/*: tests
> >> |-- *web_tt2/*: web templates
> >> |-- *po/*: user interface localization
> >> |-- *po-wwsympa/*: web help localization
> >> |-- *src**/*: source code
> >> |   |-- *bin/*: ponctual usage scripts
> >> |   |   |-- arc2webarc.pl.in
> >> |   |   |-- arcrepair.pl
> >> |   |   |-- create_db.Oracle
> >> |   |   |-- create_db.Pg
> >> |   |   |-- create_db.SQLite
> >> |   |   |-- create_db.Sybase
> >> |   |   |-- create_db.mysql
> >> |   |   |-- crypt_passwd.pl.in
> >> |   |   |-- find_missing_messages.pl
> >> |   |   |-- init_comment.pl.in
> >> |   |   |-- ldap_alias_manager.pl.in
> >> |   |   |-- logrotate
> >> |   |   |-- mod2html.pl.in
> >> |   |   |-- mysql_alias_manager.pl.in
> >> |   |   |-- p12topem.pl.in
> >> |   |   |-- sympa.generic
> >> |   |   |-- sympa.in
> >> |   |   |-- testldap.pl.in
> >> |   |   |-- testlogs.pl.in
> >> |   |   `-- tpl2tt2.pl.in
> >> |   |-- *cgi/*: web daemon and associated files
> >> |   |   |-- INSTALL
> >> |   |   |-- Makefile.am
> >> |   |   |-- README
> >> |   |   |-- mime.types
> >> |   |   |-- wwsympa-wrapper.fcgi.c
> >> |   |   `-- wwsympa.fcgi.in
> >> |   |-- *etc/*: configuration defaults and examples
> >> |   |   |-- create_list_templates/
> >> |   |   |-- families/
> >> |   |   |-- global_task_models/
> >> |   |   |-- list_task_models/
> >> |   |   |-- scenari/
> >> |   |   |-- auth.conf
> >> |   |   |-- ca-bundle.crt
> >> |   |   |-- charset.conf
> >> |   |   |-- crawlers_detection.conf
> >> |   |   |-- create_list.conf
> >> |   |   |-- edit_list.conf
> >> |   |   |-- ldap_alias_entry.tt2
> >> |   |   |-- ldap_alias_manager.conf
> >> |   |   |-- list_aliases.tt2.in
> >> |   |   |-- mhonarc-ressources.tt2
> >> |   |   |-- nrcpt_by_domain.conf
> >> |   |   `-- topics.conf
> >> |   |-- *lib/*: all modules (.pm files) in the Sympa namespace
> >> |   |   |-- Makefile.am
> >> |   |   `-- Sympa
> >> |   |-- *sbin/*: all excutable files needed for daily Sympa functionning
> >> (daemons and queue programs) maybe sympa_wizard.pl.in should move, the,
> >> as it is used at install time only?
> >> |   |   |-- alias_manager.pl.in
> >> |   |   |-- aliaswrapper.c
> >> |   |   |-- archived.pl.in
> >> |   |   |-- bounced.pl.in
> >> |   |   |-- bouncequeue.c
> >> |   |   |-- bulk.pl.in
> >> |   |   |-- familyqueue.c
> >> |   |   |-- queue.c
> >> |   |   |-- sympa.pl.in
> >> |   |   |-- sympa_wizard.pl.in
> >> |   |   |-- task_manager.pl.in
> >> |   |   `-- virtualwrapper.c
> >> |   |-- *soap/*: the SOAP server and associated files
> >> |   |   |-- Makefile.am
> >> |   |   |-- sampleClient.php
> >> |   |   |-- sympa.wsdl
> >> |   |   |-- sympa_soap_client.pl.in
> >> |   |   |-- sympa_soap_server-wrapper.fcgi.c
> >> |   |   `-- sympa_soap_server.fcgi.in
> >> |   `-- *www**/*: static files needed for the web
> >> |       |-- Makefile.am
> >> |       |-- icons
> >> |       `-- js
> >>
> >> I comment below your own proposal and compare it with the organization
> >> Guillaume proposes:
> >> *Warning: **I dont take into account the localization of the src
> >> directory in the Sympa namespace that exists in the sympa-cleanup
> >> branch, but not in the current trunk.*
> >>> -------------------------------- ------------------------
> >>> Current layout                    Proposed
> >>> --------------------------------  ------------------------
> >>> doc/                              doc/
> >> Identical
> >>> ext/                              ext/ --- Be separate modules in the
> >>> future.
> >> Identical
> >>> mail_tt2/                         default/mail_tt2/
> >> Interesting. You propose to have a "default/" directory containing all
> >> that will be later located in the "default" directory. I like it because
> >> it will probably simplify understanding for people willing to modify
> >> this content: They will probably start from what is installed, so if the
> >> organization is the same as in the install, it is probably simpler to
> >> locate which file to modify in the sources.
> >> However, in the context of sources, the term "default" is a bit
> >> confusing. Id' rather call it "default_src". We keep the "default"
> >> naming pattern and add "src" to precise it is the sources for the future
> >> "default" directory.
> >>
> >> In addition, I like the dichotomy Guillaume added. We should therefore
> >> have the following tree:
> >>
> >> default_src
> >> |--templates
> >> |   |-- mail
> >> |   |-- web
> >> |-- create_list_templates/
> >> |-- families/
> >> |-- global_task_models/
> >> |-- list_task_models/
> >> |-- scenari/
> >> |-- auth.conf
> >> |-- ca-bundle.crt
> >> |-- charset.conf
> >> |-- crawlers_detection.conf
> >> |-- create_list.conf
> >> |-- edit_list.conf
> >> |-- ldap_alias_entry.tt2
> >> |-- ldap_alias_manager.conf
> >> |-- list_aliases.tt2.in
> >> |-- mhonarc-ressources.tt2
> >> |-- nrcpt_by_domain.conf
> >> `-- topics.conf
> > I don't have objection to /mail/ and /web/ subdirectories.
> >
> > However, we have to keep in mind that current mail_tt2/ and web_tt2/
> > subdirectories shall be migrated into templates/mail/ and
> > templates/web/ subdirectories, not only under $defaultdir but also
> > under:
> >    $confdir, $confdir/VirtualDomain, $confdir/families, $expldir,
> >    $expldir/VirtualDomain.
> Actually this will be done by listmadsters when needed. If you don't 
> customize anything, you don't have to make those moves.

Won't we concern adding migration process to Upgrade.pm?
It's an idea: New Sympa may abandon customizations made with older
releases, or may try to derive them.

We would decide which we would do.

> > Besides, I'll write about default_src/ and src/ at below.
> >
> >
> >>> po-wwsympa/                       po-wwsympa/
> >>> po/                               po/
> >> I'd like to follow Guillaume's suggestion:
> >> po/
> >> |--user_interface (new name for what was "po/")
> >> |--web_help (new name for what was "po-wwsympa/")
> > If the principle is that "sources should be located at similar places
> > to installed binaries", such naming does not have inevitability.
> The idea here is just to have a single point of access for all 
> internationalization catalogues in the sources. It doesn't implies 
> anything regarding the place where they will be installed.
> >
> > For example, source tree of GNU Mailman does:
> >      messages/LOCALE/LC_MESSAGES/mailman.po
> > This will be installed as:
> >      PREFIX/messages/LOCALE/LC_MESSAGES/mailman.mo

OK.  If it is, paths would be:

    po/sympa/LOCALE.po
    ...
    po/web_help/LOCALE.po
    ...

    * LOCALE represents each gettext locale name: fr, ja, pt_BR, ...

Two subdirectories under po/ would be the names of gettext domains.

> >>> src/bin/sympa.generic             sympa.generic
> >>> src/bin/sympa.in                  sympa.in
> >>> src/bin/            [OTHERS]      bin/  or  script/ [std]
> >> "script" looks better as the programs what Guillaume put in
> >> "src/Sympa/bin" are more scripts to execute once or only once in a while.
> >>> src/cgi/mime.types                default/mime.types
> >> This is consistent as mime.types will be in "default" after install.
> >>> src/cgi/            [OTHERS]      libexec/
> >> I prefer "cgi". After all, this directory contains only the perl program
> >> responsible to run the Sympa web interface.
> > I misled.  Sympa (and configure script) distinguishes between
> > $libexecdir and $cgidir.  I agree to placing sources of FCGI scripts
> > and wrappers in cgi/.
> >
> > Additionally, I wish to install other wrappers including setUID
> > executables into single place, $libexecdir (this is not so strong
> > opinion).
> I'm sorry, I don't understand you. Which wrappers are you refering to? 
> Do you mean "future wrappers"?

I thought about aliaswrapper setUID executable which is currently
installd into $sbindir.  Since such files can be target of audit
systems (e.g. *BSD nightly checks changes of setuid'ed files),
if they are put in particular place, administrators can easily
exclude them from target.  But it is insignificant matter.

> >>> src/etc/                          default/
> >> OK, for the reasons mentionned above;
> >>> src/lib/                          lib/ [std]
> >>> src/sbin/                         sbin/  or bin/ [std]
> >> I disagree here. the reason is that I like the idea of having the whole
> >> perl code in a separated "src/Sympa/" directory. All the perl is in the
> >> same place. I think that for new developpers, it is easier to find the
> >> code (in addition to Guillaume's remarks regarding grep usage, which are
> >> more our concerns).
> >> Consequently, I'd like to keep it this way.
> > I'll write about it at below.
> >
> >>> src/soap/sympa.wsdl               default/sympa.wsdl
> >> This is consistent as sympa.wsdl will be in "default" after install.
> >>> src/soap/sympa_soap_client.pl.in  bin/  or /srcipt/ [std]
> >> script/ is pereferable here; As it is only a test script to check that
> >> the SOAP server works.
> >>> src/soap/sampleClient.php         ? --- May be moved to contrib.
> >> Well, this is kind of an awkward script. Useful, but certainly not
> >> essential.
> >> We could have a general "sample" directory in which to put this script
> >> and the other examples such as the family example.
> >>> src/soap/           [OTHERS]      libexec/
> >> Interesting proposal (though I don't agree with the name, I prefer
> >> "cgi"): After all, the SOAP server is a cgi script. It should therefore
> >> go to the same directory as wwsympa.fcgi.
> >> If we don't want to mix SOAP and web, we can make sub directories :
> >> cgi/web and cgi/soap
> > I agree to cgi/.  I feel subdirectories may not be needed.
> No needed in the sense of "vital", but I think it woul add clarity.

So, it would be:

    cgi/wwsympa/
    cgi/sympasoap/

Such a layout appears in the source packages including multiple programs.

However, in our case, directories essentially contain only two files by each
(*.fcgi.in and *.c).  I suppose clarification is less important.

> >>> src/www/                          www/
> >> Agreed. It is neither code nor future default directory. It should have
> >> its own directory.
> >>> t/                                t/ [std]
> >> Agreed. You both know that better than me.
> >>> web_tt2/                          default/web_tt2/
> >> Wee comment above about mail_tt2
> > Now, I'll write about src/ directory: What means the "source"?
> >
> > - If it means "source of executable", lib/, default/ (etc/) and so on
> >    _should not_ be placed under src/.
> >
> > - If it means "source of compiled binary", lib/, default/ (etc/) and
> >    so on _should not_ be placed under src/, while po/ (they are compiled!)
> >    and so on _should_ be placed under src/.
> >
> > - If it means "source of file to be installed at last", default_src/,
> >    www/, ... and so on _should_ be placed under src/.
> >
> > - etc.
> >
> > In the "least" case, only a few directories will be placed under src/.
> >
> > In the "most" case, almost all directories will be placed under src/.
> >
> > After all, I suppose that src/ directory is not necessary.
> Well, why not distinguish between language, then?
> We put in "src/" all that's perl and C code, and the rest in other 
> directories whose name is based on languages or functions:
> - templates for tt2 templates
> - js for javascript
> - po for i18n,
> - etc.
> That's quite a rough separation but at least it's clear.

I feel same.  Synmpa is composed using several "languages":
Perl, C, gettext catalog, SQL, TT2, HTML, CSS, JavaScript,
configurations by Sympa itself, and maybe MHonArc resource file.

Some of them will be used simultaneously (Perl and C, CSS and JS).
Separation by directories may be in accordance with such grouping.

Regards,

--- Soji

> Note an error in the tree I sent: "families" should not appear anymore 
> in the "default_src" directory.

> Best regards,
> 
> David
> >
> >
> >
> >>> --------------------------------  ------------------------
> >>>
> >>> Note: "[std]" indicates (approximately) standard source layout with
> >>> ExtUtils::MakeMaker.
> >> OK. Shall we use it some day, we would take that into account. It is not
> >> a pressing matter.
> >>> Resulting layout will become:
> >>>
> >>> --------------------------------
> >>> bin/  or  script/ [std]
> >>> doc/
> >>> ext/
> >>> default/
> >>> lib/ [std]
> >>> libexec/
> >>> po-wwsympa/
> >>> po/
> >>> sbin/  or  bin/ [std]
> >>> t/ [std]
> >>> www/
> >>> --------------------------------
> >> Here is my advice:
> >>
> >>   1. keep a "src/Sympa" directory to store our perl code
> >>   2. keep your proposals of renaming / reorganization that look good 
> >> indeed.
> >>
> >> We would therefore have the following mix of your proposal and what
> >> guillaume had done in the sympa-cleanup branch:
> >> (in green, directories that were moved / renamed, in red, files that
> >> were moved, according to Soji's proposals, Guillaume answers and my own
> >> advice)
> >> (please note the new "samples" directory, in blue)
> >>
> >> |-- several files to be read: AUTHORS, COPYING, ChangeLog, INSTALL,
> >> NEWS, README, README.charset
> >> |-- some scripts: important_changes.pl, sympa.spec.in, tap-driver.sh
> >> |-- *doc/*: doc
> >> |-- *ext/*: external modules
> >> |-- *t**/*: tests
> >> |-- *po/*: localization
> >> |-- *user_interface**/*: uesr interface localization
> >>       `-- *web_help**/*: web help localization
> >> |-- *default_src/*: configuration and templates defaults
> >> |   |-- create_list_templates/
> >> |   |-- families/
> >> |   |-- global_task_models/
> >> |   |-- list_task_models/
> >> |   |-- scenari/
> >> |   |-- *templates/*: mail templates
> >> |   | |-- *mail/*: mail templates
> >> |   |   `-- *web/*: web templates
> >> |   |-- auth.conf
> >> |   |-- ca-bundle.crt
> >> |   |-- charset.conf
> >> |   |-- crawlers_detection.conf
> >> |   |-- create_list.conf
> >> |   |-- edit_list.conf
> >> |   |-- ldap_alias_entry.tt2
> >> |   |-- ldap_alias_manager.conf
> >> |   |-- list_aliases.tt2.in
> >> |   |-- mime.types
> >> |   |-- mhonarc-ressources.tt2
> >> |   |-- nrcpt_by_domain.conf
> >> |   |-- sympa.wsdl
> >> |   `-- topics.conf
> >> |-- *www**/*: static files needed for the web
> >> |       |-- Makefile.am
> >> |       |-- icons
> >> |       `-- js
> >> |-- *samples**/*: static files needed for the web
> >> |   |-- *families*/
> >> |   `-- sampleClient.php
> >> |-- *src**/*: source code
> >> |   |-- *script**/*: ponctual usage scripts
> >> |   |   |-- arc2webarc.pl.in
> >> |   |   |-- arcrepair.pl
> >> |   |   |-- create_db.Oracle
> >> |   |   |-- create_db.Pg
> >> |   |   |-- create_db.SQLite
> >> |   |   |-- create_db.Sybase
> >> |   |   |-- create_db.mysql
> >> |   |   |-- crypt_passwd.pl.in
> >> |   |   |-- find_missing_messages.pl
> >> |   |   |-- init_comment.pl.in
> >> |   |   |-- ldap_alias_manager.pl.in
> >> |   |   |-- logrotate
> >> |   |   |-- mod2html.pl.in
> >> |   |   |-- mysql_alias_manager.pl.in
> >> |   |   |-- p12topem.pl.in
> >> |   |   |-- sympa.generic
> >> |   |   |-- sympa.in
> >> |   |   |-- sympa_soap_client.pl.in
> >> |   |   |-- testldap.pl.in
> >> |   |   |-- testlogs.pl.in
> >> |   |   `-- tpl2tt2.pl.in
> >> |   |-- *cgi/*: daemons that run under a web server
> >> | |   |-- *web**/*: web daemon and associated files
> >> | |   |   |-- INSTALL
> >> | |   |   |-- Makefile.am
> >> | |   |   |-- README
> >> | |   |   |-- wwsympa-wrapper.fcgi.c
> >> | |   |   `-- wwsympa.fcgi.in
> >> | |   |-- *soap/*: the SOAP server and associated files
> >> | |   |   |-- Makefile.am
> >> | |   |   |-- sympa_soap_server-wrapper.fcgi.c
> >> | |   |   `-- sympa_soap_server.fcgi.in
> >> |   |-- *lib/*: all modules (.pm files) in the Sympa namespace
> >> |   |   |-- Makefile.am
> >> |   |   `-- Sympa
> >> |   |-- *sbin/*: all excutable files needed for daily Sympa functionning
> >> (daemons and queue programs) maybe sympa_wizard.pl.in should move, the,
> >> as it is used at install time only?
> >> |   |   |-- alias_manager.pl.in
> >> |   |   |-- aliaswrapper.c
> >> |   |   |-- archived.pl.in
> >> |   |   |-- bounced.pl.in
> >> |   |   |-- bouncequeue.c
> >> |   |   |-- bulk.pl.in
> >> |   |   |-- familyqueue.c
> >> |   |   |-- queue.c
> >> |   |   |-- sympa.pl.in
> >> |   |   |-- sympa_wizard.pl.in
> >> |   |   |-- task_manager.pl.in
> >> |   |   `-- virtualwrapper.c
> >>
> >>>> What did you think about what Guillaume had done in the sympa-cleanup
> >>>> branch? It looked good: every perl code in the src/Sympa directory, 
> >>>> then
> >>>> more or less the same organization as the one you proposed?
> >>> My proposal covers where he had not changed.
> >> Well, my proposal tries to cover both you proposals and my own advice.
> >> It looks like a reasonable proposal to me.
> >> Do you agree?
> >>
> >> Best regards,
> >>
> >> David
> > Best regards,
> >
> >
> > --- Soji
> >
> >
> >
> > -- 
> > A bug in Sympa? Quick! To the bug tracker! 
> > <https://sourcesup.renater.fr/tracker/?group_id=23>
> > RENATER logo
> > *David Verdin*
> > Études et projets applicatifs
> >
> > Tél : +33 2 23 23 69 71
> > Fax : +33 2 23 23 71 21
> >
> > www.renater.fr <http;//www.renater.fr>
> >     RENATER
> > 263 Avenue du Gal Leclerc
> > 35042 Rennes Cedex
> >
> >
> >


-- 
株式会社 コンバージョン  セキュリティ&OSSソリューション部   池田荘児
〒231-0004　神奈川県横浜市中区元浜町3-21-2 ヘリオス関内ビル7F
e-mail address@concealed  TEL 045-640-3550
http://www.conversion.co.jp/