The mailing list for listmasters using Sympa

["Stefan Hornburg (Racke)" <address@concealed>]

On 5/4/20 11:33 AM, Olivier Salaün wrote:
> Hello,
> 
> I wish to summarize here the results of our investigation to understand the 
> origin of very bad performances of wwsympa
> in our production server (Sympa 6.2.48).
> 
> We are running a ML service with : 8 robots, 5.200 lists (total), 2.500 
> family lists, 960.000 subscribers
> (subscriber_table), 36.000 admins (admin_table).
> 
> You may know that on Sympa's web interface the critical operations are the 
> ones that require wwsympa.fcgi to load all
> the mailing lists. This includes the */lists* and */lists_categories* URLs. 
> On our main robot (1.500 ML) /lists
> processing takes: *3 minutes* if the wwsympa.fcgi process is not running ; 
> *16 seconds* otherwise. These response times
> make Sympa's web interface almost unusable. That's why we decided to 
> investigate...
> 
> To find out what's going on we used a Perl Profiler 
> (https://metacpan.org/pod/Devel::NYTProf). Once enabled it keeps
> track of all subroutine calls and draws a picture of the time spent in each 
> subroutine, how many times it was called and
> other fine grained details.
> 
> The profiler results helped us identify where wwsympa.fcgi spends too much 
> time. We then tried to disable parts of the
> code and observed when it had significant positive impact on the 
> performances. Below I detail the 5 tracks we followed.
> 
> *Patch #1*: prevent database calls from get_current_admins()
> 
> Response time gain: *-62%*
> 
> We noticed that there were 50.846 calls to 
> Sympa::Database::do_prepared_query(), most of them from 
> get_current_admins().
> It occurs in the main loop of wwsympa::do_lists() but also every time a 
> scenario including 'is_editor' or 'is_owner'
> condition is evaluated. Obviously a level of cache should be introduced here
> 
> => found a related Pull Request: 
> https://github.com/sympa-community/sympa/pull/925
> 
> *Patch #2*: disable authorization scenario evaluation
> 
> Response time gain:*-34%*
> 
> During scenario evaluation the code does too many regular expression 
> evaluation (490.000). Having a look at
> Sympa::Scenario::_verify(), it could be optimized. One option would be to 
> use https://metacpan.org/pod/Regexp::Grammars
> to parse the scenarios files. It would also make the code more readable. 
> But to prevent issues with existing scenarios,
> this requires more complete unit tests.
> 
> => I will propose a new approach for scenario grammar parsing
> 
> *Patch #3*: disable syslog calls
> 
> Response time gain: *-9%*
> 
> The profiler reveals huge amount of time spent in Sympa::Log::syslog() 
> (151.862 calls). Many low-level subroutines in
> the code provide syslog() logs for debugging purpose. Even when Sympa is 
> configured with "log_level info", these
> syslog() calls have a significant cost because of the subroutine call 
> itself. A simple bypass would be to replace
> syslog('debug',...) calls with syslog('debug',...) if 
> ($Sympa::Log::levels{'debug'}). This could be done by configure/make.
> 
> => not a big performance gain, but a quick win though
> 
> *Patch #4*: prevent param_constraint.conf loading (for lists belonging to 
> families)
> 
> Response time gain:
> 
>   * for a fresh wwsympa.fcgi process: *-83%*
>   * for an existing wwsympa.fcgi process:*-18%*
> 
> With existing code, param_constraint.conf files are loaded every time a 
> Sympa::List object is loaded. It's expensive to
> run check_param_constraint() and it only make sense when (re)instantiating 
> list families.
> 
> => I submitted this Pull Request: 
> https://github.com/sympa-community/sympa/pull/933
> 
> *Patch #5*: disabling file locking
> 
> Response time gain: *-25%*
> 
> The profiler shows that Sympa code spends too much time calling 
> File::NFSLock functions. I just try disabling file
> locking to evaluate impact, but it would make sense to consider using 
> alternative file locking implementations.
> 
> *Conclusion*
> 
> Applying all 5 patches wwsympa performances are much better:
> 
>   * for a fresh wwsympa.fcgi process: *3min => 8,50 seconds*
>   * for an existing wwsympa.fcgi process:*16 seconds => 5,80 seconds*
> 
> Some changes I tried are quick wins, others either provide less benefits or 
> require more refactoring work.
> 

Hello Olivier,

thanks a lot for your report. This is certainly a sore point in the usability 
of the web interface, so improvements
there would be much appreciated.

Comments:

#1 This patch looks good to me, so it would be good to run tests on it and 
merge it when we are satisfied with
   the result.

#5 I think locking is not necessary for compiling the list information.

Regards
        Racke

> 
> -- 
> Olivier Salaün
> DSI / pôle SI / équipe SNUM
> Tel : 02 23 23 74 54
> 


-- 
Ecommerce and Linux consulting + Perl and web application programming.
Debian and Sympa administration. Provisioning with Ansible.

Attachment: signature.asc
Description: OpenPGP digital signature