Do client queueing to handle server outages.

Add support to syslog::client to queue up messages in the client if
the server can't be reached.  This should allow us to not lose log
messages in case of shorter outages (hours, but not days) of the log
server, whether planned or unscheduled.

The defult setting is to have a queue of at most 100'000 messages,
stored in memory on the client rsyslog daemon.  For many systems at
NSC, that would cover several hours of server outage.  Systems that
generate large amounts of logs, may want to tune that, or enable
disk-assisted queueing, where rsyslog stores "overflow" on disk.

In order for the client to detect that the log server is gone, it
needs to use TCP or RELP, not UDP.  We thus change the default of
the protocol parameter of syslog::client to "tcp" as well.

Preliminary support for disk-assisted queueing is included, but we
have had some troubles getting this mode to work well in our testing
(rsyslogd hung when shut down, for example), so consider that support
to be experimental for the time being!

Currently, we generate old-style directives in the configuration
snippet.  This works both in rsyslog version 5 (RHEL-6) and rsyslog
version 8 (RHEL-7 and RHEL-8), so we don't need multiple code paths
to handle them both.  However, when we can retire version 5 support
some time in the future, we should transition to RainerScript for
this.

manifests/client.pp

@@ -28,14 +28,36 @@
 #		formatted.  Defaults to 'RSYSLOG_ForwardFormat', which
 #		includes microsecond resolution and timezone information
 #		in the timestamps (but requires rsyslog as the server).
+#
+#  - queueing	What kind of queueing of log messages to perform when
+#		messages can't be sent to the logserver.  Queueing is
+#		only useful for protocols tcp and relp; udp can't detect
+#		that the server is gone.
+#		The queueing parameter can be set to one of:
+#		 'none'		Messages that can't be sent are discarded
+#				immediately.
+#		 'inmemory'	Unsent messages are kept in process memory
+#				only.  If the rsyslog daemon dies, they are
+#				lost.
+#		 'diskassisted'	Unsent messages are kept in process memory,
+#				but some effort is made to save them to disk
+#				upon shutdown.
+#				NOTE: THIS MODE IS CURRENTLY EXPERIMENTAL!
+#
+#  - queuesize	Maximum number of log messages to keep in the in-memory
+#		queue.  When a disk-assisted queue gets close to this
+#		limit, it will flush entries to disk; the disk queue is
+#		not size-limited.
 
 class syslog::client(
 	$logserver='syslog.nsc.liu.se',
 	$ipfamily='ipv4',	# 'ipv4', 'ipv6' or ['ipv4','ipv6']
-	$protocol='udp',	# udp, tcp or relp
+	$protocol='tcp',	# udp, tcp or relp
 	$port=false,		# Default: 514 for TCP/UDP, 2514 for RELP
 	$selector='*.*',
 	$format='RSYSLOG_ForwardFormat',
+	$queueing='inmemory',
+	$queuesize=100000,	# A few tens of megabytes
       )
 {
     include syslog::service
@@ -44,28 +66,61 @@ class syslog::client(
 
     $syslogaddrs = resolve_ipnets($logserver, $ipfamily, 'stable')
 
+    # NOTE: Old-style config directives used, to support both version 5
+    # and version 8.  When version 5 support is no longer needed, we
+    # should switch to RainerScript.
+
+    $load_output_module = $protocol ? {
+	'relp'	=> [ "\$ModLoad omrelp\n", ],
+	'tcp'	=> [ ],
+	'udp'	=> [ ],
+	default	=> fail("${title}: Bad protocol, ``${protocol}''"),
+    }
+    $queueparams = $queueing ? {
+	'none'		=> [ ],
+	'inmemory'	=> [
+		"\$ActionQueueType LinkedList\n",
+		"\$ActionResumeRetryCount -1\n",
+		"\$ActionQueueSize ${queuesize}\n",
+		"\$ActionQueueTimeoutShutdown 2000\n",
+		# Avoid warnings from rsyslogd; these should be defaults
+		sprintf("\$ActionQueueDiscardMark %d\n", $queuesize * 0.98),
+		sprintf("\$ActionQueueHighWaterMark %d\n", $queuesize * 0.90),
+	],
+	'diskassisted'	=> [
+		"\$ActionQueueType LinkedList\n",
+		"\$ActionResumeRetryCount -1\n",
+		"\$ActionQueueSize ${queuesize}\n",
+		"\$ActionQueueTimeoutShutdown 2000\n",
+		"\$ActionQueueFileName logserver-${logserver}\n",
+		"\$ActionQueueSaveOnShutdown on\n",
+		# Avoid warnings from rsyslogd; these should be defaults
+		sprintf("\$ActionQueueDiscardMark %d\n", $queuesize * 0.98),
+		sprintf("\$ActionQueueHighWaterMark %d\n", $queuesize * 0.90),
+	],
+	default	=> fail(
+		"Syslog::Client: Bad parameter queueing, ${queueing}")
+    }
     $protoprefix = $protocol ? {
 	'udp'	=> '@',
 	'tcp'	=> '@@',
 	'relp'	=> ':omrelp:',
-	default	=> fail("${title}: Bad protocol, ``${protocol}''"),
     }
     $xport = $port ? {
 	false	=> $protocol ? { 'relp' => ':2514', default => '' },
 	default => ":${port}",
     }
+    $cfglines = [
+	$load_output_module,
+	$queueparams,
+	sprintf("%s\t\t %s[%s]%s;%s\n",
+		$selector, $protoprefix, $syslogaddrs[0], $xport, $format),
+    ]
 
-    $cfg_line = sprintf(
-	"%s\t\t%s[%s]%s;%s\n",
-	$selector, $protoprefix, $syslogaddrs[0], $xport, $format)
-
-    if ($protocol == 'relp') {
-	include syslog::relp_package
-    }
     file {
 	$cfg_d_file:
 	    ensure => file, owner => 'root', group => 'root', mode => '0444',
-	    content => $cfg_line,
+	    content => inline_template('<%= [@cfglines].flatten.join() %>'),
 	    require => Package['rsyslog'],
 	    notify => Service['rsyslog'];
     }