Skip to content
  • Thomas Bellman's avatar
    1bd55820
    Logserver: Keep more log files open. · 1bd55820
    Thomas Bellman authored
    By default, rsyslog keeps at most 10 files open for each log action
    when using dynamic file templates like we do.  This can make rsyslogd
    close and open files constantly, which may increase filesystem load
    and decrease performance.  Some sampling on NSC's current log server
    (using strace -C) shows that it is opening and closing log files some
    10-15 times per second, and is on average writing less than 15 log
    messages to each file before it is closed.  This amount is not a big
    concern, but it feels enough that we should concider mitigating it.
    
    We here increase that cache of open files to 256 for the "main" log
    action.  As NSC currently has about 200 active syslog clients, that
    should make rsyslogd basically never have to close and reopen a log
    file.
    
    We have also tested smaller cache size.  At 100 open files, rsyslogd
    still does about 1.0-1.5 close and open calls per second.  Since a
    few hundred open files is not a big deal today, we can easily afford
    having the cache size set to 256 and almost never have to close a
    log file.
    
    If we start taking logs from worker nodes in our compute clusters, we
    will probably need to revisit this, and possibly change how we store
    logs, as we would then have some 3000-4000 clients or more, each with
    its own log file.
    1bd55820
    Logserver: Keep more log files open.
    Thomas Bellman authored
    By default, rsyslog keeps at most 10 files open for each log action
    when using dynamic file templates like we do.  This can make rsyslogd
    close and open files constantly, which may increase filesystem load
    and decrease performance.  Some sampling on NSC's current log server
    (using strace -C) shows that it is opening and closing log files some
    10-15 times per second, and is on average writing less than 15 log
    messages to each file before it is closed.  This amount is not a big
    concern, but it feels enough that we should concider mitigating it.
    
    We here increase that cache of open files to 256 for the "main" log
    action.  As NSC currently has about 200 active syslog clients, that
    should make rsyslogd basically never have to close and reopen a log
    file.
    
    We have also tested smaller cache size.  At 100 open files, rsyslogd
    still does about 1.0-1.5 close and open calls per second.  Since a
    few hundred open files is not a big deal today, we can easily afford
    having the cache size set to 256 and almost never have to close a
    log file.
    
    If we start taking logs from worker nodes in our compute clusters, we
    will probably need to revisit this, and possibly change how we store
    logs, as we would then have some 3000-4000 clients or more, each with
    its own log file.
Loading