Developers of Sympa

[Sergiy Zhuk <address@concealed>]

hi

The default algorithm is not very effective in case you need to add an
article to an archive with a large number of articles.
archived.pl is reading and sorting the arctxt dir on every article to find
the next article number, so if you have several thousand messages there,
it's taking a lot of time.
mhonarc takes a while to process the message too, but at least we can
eliminate the delay sympa is imposing by indexing the arctxt once and
keeping the last article number in a file.
The attached patch does that.
The change is transparent to current installations, since if we can't find
the index file, we create it on the fly.
It's also easy to reset the last number by removing the index file.
In addition, I've removed sort() in readdir() since if you have a lot of
messages for the same list, you won't be able to process anything else
before you process all of them, which is kinda bad for user experience.
I'm gonna re-do sympa queue processing in near future.

thanks

-- 
rgds,
serge--- archived.pl.orig    Fri Oct 17 22:16:26 2003
+++ archived.pl Fri Oct 17 21:42:25 2003
@@ -140,7 +140,7 @@
        fatal_err("Can't open dir %s: %m", $queue); ## No return.
    }
 
-   my @files =  (sort grep(!/^\.{1,2}$/, readdir DIR ));
+   my @files =  (grep(!/^\.{1,2}$/, readdir DIR ));
    closedir DIR;
 
    ## this sleep is important to be raisonably sure that sympa is not 
currently
@@ -331,10 +298,8 @@
 sub mail2arc {
 
     my ($file, $listname, $hostname, $yyyy, $mm, $dd, $hh, $min, $ss) = @_;
-
-    do_log ('debug2',"mail2arc ($file, $listname, $hostname, $yyyy, $mm, 
$dd, $hh, $min, $ss)");
-
     my $arcpath = $wwsconf->{'arc_path'};
+    my $newfile;
     
 
     my $list = new List($listname);
@@ -390,11 +355,22 @@
     }
     
     ## copy the file in the arctxt and in "mhonarc -add"
-    opendir (DIR, "$arcpath/$listname\@$hostname/$yyyy-$mm/arctxt");
-    my @files = (sort { $a <=> $b;}  readdir(DIR)) ;
-    $files[$#files]+=1;
-    my $newfile = $files[$#files];
-#    my $newfile = $files[$#files]+=1;
+     if( -f "$arcpath/$listname\@$hostname/$yyyy-$mm/index" )
+     {
+       open(IDX,"<$arcpath/$listname\@$hostname/$yyyy-$mm/index") || 
fatal_err("couldn't read index for $listname");
+       $newfile = <IDX>;
+       chomp($newfile);
+       $newfile++;
+       close IDX;
+     }
+     else
+     {
+       do_log('debug',"indexing $listname archive");
+       opendir (DIR, "$arcpath/$listname\@$hostname/$yyyy-$mm/arctxt");
+       my @files = (sort { $a <=> $b;}  readdir(DIR)) ;
+       $files[$#files]+=1;
+       $newfile = $files[$#files];
+     }
     
     my $mhonarc_ressources = &get_ressources ($listname . '@' . $hostname) ; 
     
@@ -415,8 +390,19 @@
     
     close ORIG;  
     close DEST;
+    &save_idx("$arcpath/$listname\@$hostname/$yyyy-$mm/index",$newfile);
 }
 
+sub save_idx
+{
+my ($index,$lst) = @_;
+
+   open(INDEXF,">$index") || fatal_err("couldn't overwrite index $index");
+   print INDEXF "$lst\n";
+   close INDEXF;
+#   do_log('debug',"last arc entry for $index is $lst");
+}
+   
 sub get_ressources {
     my $adrlist = shift;
     my ($mhonarc_ressources, $list);