Developers of Sympa

[David Verdin <address@concealed>]

Hi all,

Le 24/01/13 04:46, IKEDA Soji a écrit :

Hi Marc and all,

On Wed, 23 Jan 2013 15:26:35 +0100
Marc Chantreux <address@concealed> wrote:

… and still f****ing slow! 

hello everybody.

here we are, stuck with a very up to date sympa 6.1 which is too slow.
we really have some ideas to speed up the things but it means a lot of
code. I don't want to fork sympa 6.1 and i see that the renater instance
is still on 6.2.

What's slow exactly? I don't remember reading the details about which operation on your server takes so long.
I don't remember reading your different configuration settings, server characteristics and overall lod: number of lists, users, etc.
There are probably things to be done to improve performances, beyond changing the code - which certainly ALSO a solution.

6.2 branch incorporated db_list_cache branch and some improvements
by me.

List.pm:

o List::get_lists() takes charge of querying lists.
  It is based on the idea Riseup patch had been realized.

  Additionally,

  - Querying and sorting are available on extended number of list
    parameters.

  - Perl regexp or SQL RLIKE (extension by MySQL) is no longer used
    for search on list subjects.  Substring matches are performed
    by fc() and index().

    # Fold-cased search keys are stored into database field.
    # On Perl earlier than 5.12.6, Unicode::CaseFold external module
    # is used to support fc().

  - Now get_which() is a thin wrapper of get_lists() above.  SQL
    queries on subscriber/admin_table and list_table were unified
    to single query with JOIN condition.

  - In addition to binary file (LIST_DIR/config.bin), list config
    may be cached into database BLOB field.
    It can reduce the time to scan under list_data directory.

    # Set "config_list_cache" robot parameter to be "database".

o Introducing accessor methods, abstraction of List parameters.

  - By my profiling, Scenario objects were heavily reloaded during
    Lists objects are loaded.
    Now Scenario objects will be loaded when they are really
    needed, if they are accesssed through accessors.

  - Default parameter values will no longer saved into LIST_DIR/config
    file, LIST_DIR/config.bin nor list_table cache.

wwsympa.fcgi:

o Several search operation on lists was made faster thanks to
  List::get_lists() above.

o Limiting number of "Your lists" side menu.  Pages can be loaded
  faster.
  # This will be effectual in the case that someone (e.g. the system
  # administrator) is an owner of almost all lists of the site.

Issues NOT yet solved:

o Searching users.
  It has not supported substring match.

o Slow "List of lists" page
  Currently WWSympa trys to load all lists.  This behavior is a big
  performance hit on the site with hundreds or many of lists.

We should probably cache something here, but I don't see how. We don't display a lot of things in the list of lists, so we could probably cache them and save ourselves the pain to load configs. But the problem is the visibility. We have two choices:

1- either we cache the visibility result for each user and each list in database. That's a lot of entries to store in DB  - and to keep up to date.
2- either we change the way visibility is defined. Switching to a larger granularity, such as yes/no/subscribers only could probably improve performances on the list of lists. The thing here, is to NOT load the full lists.

o Codes to handle robot config parameters and config files would be
  refactored and improved, however, I shall not be able to finish
  it in this winter (if I did).

That could be work for the 6.3. I'm working on a candidate roadmap for Sympa, that I really would like to discuss with all of you.
We, on the RENATER side, have features that we really want to implement, and also features that would be great to add but are not necessary at once. All this should be discussed between us. You probably have your own priorities.

o Bi-directional layout support by CSS.
  My private experiment using CSSJanus showed promising result,
  however, slight changes on code will be nessessary.

Mark Overmeer, with whom I'm in contact for a development of his own in Sympa, suggested CSS 3.
He also suggested a lot of desing improvements that I will discuss with you.

o ..., etc.

We need to find a solution fast but we have to decide what's our next
step before monday: maintaining a fork or test the 6.2 upgrade. what
would be your move? any good idea to share? 

I'm all right to swich 6.2 as a release candidate.

I will have to disagree with this proposal.
Even if most of the work is done, I still have too much crashes on my developemnt server to release a beta.
I think I still have, with an optimistic view, at least a week worth of testing / bug fixing to have a decent release candidate.

My planned work on 6.2 branch has almost over.  Unsolved issues above
would be postponed.

You and guillaume have really done a great work on Sympa these last months!
Once the 6.2 is in beta, we will have plenty of time to decide what should be next for Sympa.

Cheers!

David

regards

Regards,

-- Soji

-- 
Marc Chantreux
Université de Strasbourg, Direction Informatique
14 Rue René Descartes,
67084  STRASBOURG CEDEX
☎: 03.68.85.57.40
http://unistra.fr
"Don't believe everything you read on the Internet"
    -- Abraham Lincoln

--
A bug in Sympa? Quick! To the bug tracker!

David Verdin Services Applicatifs aux Utilisateurs Tel. +33 2 23 23 69 71 GIP RENATER

Attachment: smime.p7s
Description: Signature cryptographique S/MIME