Logserver: Keep more log files open.

By default, rsyslog keeps at most 10 files open for each log action
when using dynamic file templates like we do.  This can make rsyslogd
close and open files constantly, which may increase filesystem load
and decrease performance.  Some sampling on NSC's current log server
(using strace -C) shows that it is opening and closing log files some
10-15 times per second, and is on average writing less than 15 log
messages to each file before it is closed.  This amount is not a big
concern, but it feels enough that we should concider mitigating it.

We here increase that cache of open files to 256 for the "main" log
action.  As NSC currently has about 200 active syslog clients, that
should make rsyslogd basically never have to close and reopen a log
file.

We have also tested smaller cache size.  At 100 open files, rsyslogd
still does about 1.0-1.5 close and open calls per second.  Since a
few hundred open files is not a big deal today, we can easily afford
having the cache size set to 256 and almost never have to close a
log file.

If we start taking logs from worker nodes in our compute clusters, we
will probably need to revisit this, and possibly change how we store
logs, as we would then have some 3000-4000 clients or more, each with
its own log file.

templates/rsyslog-el6.conf-logserver.erb

-$MaxOpenFiles 6144	# Significantly larger than $InputTCPMaxSessions
+# Significantly larger than $InputTCPMaxSessions; remember also to
+# account for $DynaFileCacheSize in omfile actions.
+$MaxOpenFiles 6144
 
 $ModLoad imklog
 $ModLoad imudp
@@ -74,6 +76,7 @@ $FileCreateMode 0644
     local6.*                    -?AuditLogfile;LogFmtJoinedLog
     local6.*                    ~
 
+    $DynaFileCacheSize 256	# Sync with $MaxOpenFiles
     *.*				?LogFile;LogFmtSpaceTime
 
     # We log everything to one file as well, for easy 'tail -f':ing

templates/rsyslog-el7.conf-logserver.erb

@@ -3,7 +3,10 @@
 
 $umask 0022
 $WorkDirectory /var/lib/rsyslog
-$MaxOpenFiles 6144	# Significantly larger than imtcp maxSessions
+
+# Significantly larger than imtcp maxSessions; remember also to
+# account for dynaFileCacheSize in omfile actions.
+$MaxOpenFiles 6144
 
 
 
@@ -218,7 +221,9 @@ if $syslogfacility-text == "local6" and $programname == "audispd" then
 }
 
 # Everything else stored in the normal /syslogs/... paths
-action(type="omfile"  dynaFile="LogFile")
+action(type="omfile"  dynaFile="LogFile"
+       dynaFileCacheSize="256"		# Sync with $MaxOpenFiles
+)
 
 # We log authpriv information to one file as well, for easy 'tail -f':ing
 # This file is logrotated every day, with only one generation saved.