Stanza retries forever when log entry is too big for Google Cloud Logging

[jsirianni]

Describe the bug

I see this occasionally while using the vcenter plugin. The error repeats over and over in the log.

{"level":"warn","timestamp":"2021-02-26T18:24:54.777-0500","message":"Failed flushing chunk. Waiting before retry","error":"rpc error: code = InvalidArgument desc = Log entry with size 256.4K exceeds maximum size of 256.0K","wait_time":21.213547168}

If I stop stanza, delete stanza.db, and restart stanza, the error does not show up for a while. Indicating that only a few log entries are too large. Google Cloud Logging console shows plenty of log activity.

To Reproduce
Steps to reproduce the behavior:

1. Use syslog input
2. Use Google Cloud output
3. Send a very large message to stanza ( I am working on an easy way to do this )

Expected behavior

Stanza should handle messages that are too large for Google Output, either by breaking them up or dropping them and logging a warning or error.

[jsirianni]

Possibly unrelated, and requires its own issue. I created a large single line file, and sent it to stanza's tcp listening port. I seem to have crashed stanza in a way that prevents me from stopping and starting. I rebooted the VM only to find it hanging during shutdown:

I filed an issue related to systemd #258.