Developers of Sympa

[David Verdin <address@concealed>]

Dear all,

With the upcoming install of Sympa 6.2 on our production server (next week), the subsequent beta phase and the load of mails we received from contributors / users / funders, I feel it is time to start putting together our thoughts about Sympa for the next couple of years.

Development plans

Here are our plans of development. These are RENATER project, but you may have your own that can certainly find their place in the roadmap:

Features

• Sympa in SaaS mode. We want to allow full virtual hosting of a Sympa service. That means a lot of evolutions for scalability concerns and new robot deployment. It will also require to make the largest part of Sympa functionnalities accessible through the web and / or web services (REST, SOAP). This is our next big step for Sympa 6.3.
  
• RFC 6729 implementation. This RFC allows to add "Received by" headers with richer information, such as "kept for moderation". This will greatly improve tracability of message in mailing lists.
• RFC 5983 implementation. This RFC deals with i18n addresses and mailing lists. I think Sympa should implement it. I wonder whether Soji would have any advice about this implementation, which look tricky...
• Potentially: changing our configuration file formalism to switch to something more common, such as YAML. We could unify our configuration files and base them on CPAN modules instead of our current home made parsers.
  

This is what we really want to implement in the 6.3. Feel free to add the features you think should be added to this version. For example, we spoke about CSS 3 a while ago. It woul be great. Once we agree on development plans, we'll add a roadmap to the Sympa web site, with a section for Sympa 6.3 and another for releases after.

Code design

Shortly: As pointed by Guillaume and lot of other fine people, we need to improve our code design.

We are thinking that it could be useful to move some parts of the code to CPAN, for better maintainability.
Which parts, and how to achieve this is what I would like to discuss here.

A little recap of what has been done yet and what is planned already:

• Refactoring: Guillaume did a great work on the current 6.2 to desintricate modules. He alos shortenened them and reorganized them into a comprehensive architecture.
  
• As far as I remember, Guillaume started working on regression tests. I think it would be great to continue this work which could be a good step towards a stabler code.
• Soji rewrote the Site / Robot / List systems by using inheritance. It allows a new cache system which will decrease the number of List object and Robot objects parsing.
• I rewrote the DB access by abstracting the DB engine used. It looks like
  
• I may have missed some more recent works

What we need now is a common way of doing things in Sympa:

• What coding design?
• What coding practice? Should we move to git?
  

Code design: We have been in communication with Mak Overmeer over the last two months or so. He gave us a lot of idea about possible code evolution. we had to calm hilm down because he wanted to implement all of them tight now and we wanted to discuss them with you first.
Basically, he wants to make the full sympa code a big set of plugins.
Here is some pseudo code that would allow to use an external source category called VOOT to built the memebers list.

  package Sympa::PluginManager;

  sub savePluginInstance($$)
  {   my ($self, $list, $plugin) = @_;
      $self->db->update("UPDATE plugin SET listname=?,class=?,descr=?,session=?"
        , $list->name, ref $plugin, $plugin->description, $plugin->serialize);
  }

  sub listPluginInstances($)
  {   my ($self, $list) = @_;
      # returns array of hashes
      $self->db->select("SELECT * FROM plugin WHERE listname=?", $list->name);
  }

  sub plugginsFor($)
  {   my ($self, $list) = @_;
      map $self->revivePluginInstance($_), $self->listPluginInstances($list);
  }

  sub revivePluginInstance($)
  {   my ($self, $info) = @_;
      trace "reviving $info->{listname}: $info->{descr}";
      my $class = $info->{class};
      require "$class" or die $@;
      $class->deserialize($info->{session});
  }


  package Sympa::Plugin;

  sub new(%)                     # new() only in the base-class
  {   my ($class, %args) = @_;
      (bless {}, $class)->init(\%args);
  }

  sub init($)                    # init() required in the base-class, maybe up
  {   my ($self, $args) = @_;
      $self;
  }


  package Sympa::ListMembers;
  use base 'Sympa::Plugin';

  sub init($)                    # can be left-out, if nothing to do
  {   my ($self, $args) = @_;
      $self->SUPER::init($args);
      ... init on this level ...
      $self;
  }

  sub listMembers() {croak}      # all extensions need to implement this


  package Sympa::ListMembers::VOOT;
  use base 'Sympa::ListMembers';
  use Storable qw/freeze thaw/;

  sub init($)                    # can be left-out, if nothing to do
  {   my ($self, $args) = @_;
      $self->SUPER::init($args);
      ... init on this level ...
      $self;
  }

  sub serialize()
  {   my $self = shift;
      my %data = "...collect" the important data to save...;
      freeze \%data;
  }

  sub deserialize($)
  {   my ($self, $data) = @_;
      # Often, the saved data is the same as the parameters to new(), byt
      # sometimes some extra tricks are needed.
      $self->new(thaw $data);
  }

  sub listMembers()
  {   ... use VOOT etc to collect a list of names ...
      @members;
  }


  package Sympa::List;

  sub new(%)...
  sub init($)
  {    my ($self, $args) = @_;
       $self->{name}    = $args->{name}    or die;
       $self->{plugins} = $args->{plugins} || [];
       $self;
  }
  sub name() { shift->{name} }

  sub plugins(;$)
  {   my ($self, $type) = @_;
      my $p = $self->{plugins};
      $type ? (grep $_->isa($type), @$p) : @$p;
  }

  # Ah, this is the real thing!
  sub members()
  {   my $self = shift;
      map $_->listMembers, $self->plugins('Sympa::ListMembers');
  }


  package Sympa;     # the main program is very simple now!

  my $mgr  = Sympa::PluginManager->new($db);

  my $list = Sympa::List->new
    ( name    => $list_i_need
    , plugins => $mgr->pluginsFor($list_id_need)
    );

  print join "\n", $list->members;

This is a suggestion. What do you think about it? Do you think such a design would be a good turn for Symp to take? The idea of having Sympa a set of plugins where anything could be activated / decactivated independently looks promising.

I propose you put below:

1- Your code design suggestion
2- Your code documentation preferred structure (I think we agreed to use pod documentation now, didn't we?)
3- Your code presentation preferences (for example, I prefer tabs to spaces :-P ).

We'll compromise aroud this and constitute a set of coding best practices to apply from now on.

I think we should split the time devoted to development between large refactoring, as Guillaume is doing right now, and feature additions, as Saas development. We could dedicate the first half of the year to refactoring, and the other half to new features - and minor refactorings. This woul probably ease branches merges and give us a common base to work on new features, as well as experimenting code design by using it in feature development.

About refactoring, I think that the most important is to have very orthogonal modules, with the less intrications possible between modules; Converting such modules to CPAN / plugins thereafter would be a bonus.

Upcoming events

French training

There will be a training about Sympa in the first week of June. We didn't make the public announcment yet because there are still some logistical concerns to deal with. We considered making a little hackaton after that, are you still in?
As the training would be from 3rd to 6th of June, we could save the 7th of June for a one day hackaton during which we could achieve / finalize some basic refundations of the Sympa code architecture. This work could be continued over the summer, leaving us time to add new features over the last half of the year.

Sympa workshop and possible follow-ups

TERENA offered to sponsor a Sympa workshop, which would be more or less something like a training, but for our European colleagues. They could handle subscription to this event and fund it. This would probably be scheduled in October 2013.
We thought it could be nice to organize the Sympa days jointly. We could, for example, make a one day workchop, followed by a two days meeting called "Sympa days" in which people would share experience about the product. We could even make it followed by a two days hackaton. What do you think Marc? Do you think it could be possible to organize such a week in Strasbourg?

JRES 2013

For Soji: this is the French meeting for education and research conference that happens every two years.
I'll propose a short paper about the Sympa project, its organizations, irs members, contributors, and such. I'll also propose an event in which we could meet with Sympa admins and potential developpers. I hope at least one of those two will be accepted. The idea here is to introduce the current team of developpers (which is you, and we would be good if we could ) and the overall organization of the project, in the hope to bring still new blood to the project and the community. It is important to let know that the project keeps opening to the community, event though it is still hosted and guaranteed by RENATER.

OK...

That was quite a long mail, heh?
I sure hope you want to answer it... ;-)

Best regards,

David

--
A bug in Sympa? Quick! To the bug tracker!

David Verdin Services Applicatifs aux Utilisateurs Tel. +33 2 23 23 69 71 GIP RENATER

Attachment: smime.p7s
Description: Signature cryptographique S/MIME