This PR started out as a fix for an issue reported by @faxm0dem on gitter, with config and patterndb here: https://gist.github.com/faxm0dem/3dfc0873842d4c05ebcd8cb7f8eda8ab

The issue turned out to be an issue how we process the "unmatched" state for a log path. With time, another case came up in #4339 and it turns out we have a limited-incomplete definition on how we define that a log path did not match.

Since this is a complicated issue, let me try to write this up, hopefully it will be useful for my future self or anyone trying to understand the compilation process.

So let's take one step back.

Why do we need the notion of a message being "unmatched" for a log path?

syslog-ng supports both independent and dependent log paths. An independent one is simple: let's take the stream of messages and route them to multiple outputs. These outputs are independent, as one output does not depend on the result of the other.

For example, there are two independent outgoing paths from the whatever source:

log {
    source(whatever);
    log {
        filter(app1);
        destination(app1_logfile);
   };
   log {
       filter(app2);
       destinatrion(app2_logfile);
   };
};

Sometimes however we only want to take a route if a previous did not match, like this:

log {
    source(whatever);
    log {
        filter(app1);
        destination(app1_logfile);
        flags(final);
   };
   log {
       destinatrion(catchall_logfile);
   };
};

Notice the "final" flag in the first log {} statement. This means that if the first log statement "matches", the 2nd is not processed at all. This form of routing is sometimes easier to write, but it also means that syslog-ng needs the notion of whether a message matched a specific log path.

Sometimes, we have hundreds of these log {} rules in a configuration, syslog-ng trying to match the right one automatically.

So what does it mean that a message matches?

In simple cases, a message is assumed to always match a path, unless a filter or parser drops it. From the example above:

    log {
        filter(app1);
        destination(app1_logfile);
        flags(final);
   };

if filter(app1) matches the message, this log path would match. If it does not, then the entire path is considered unmatching.

What about complex cases?

syslog-ng's language is flexible enough to route messages around based on complex criteria. So a simple filter might actually be a complex set of parsing/filtering/processing rules, hidden behind a single line of reference. Just look at our cisco parser (https://github.com/syslog-ng/syslog-ng/blob/master/scl/cisco/plugin.conf).

What about destinations?

So knowing what is "matching" is super useful. But what is considered a match, if we have "multiple" exit points from a log path:

Let's look at this example:

log {
    source(whatever);
    destination(dest1);
    filter(filter_app);
    destination(dest2);
};

As you can see, dest1 and dest2 are both destinations of this log path. dest1 would get a copy of all messages while dest2 would only get those that match the filter filter_app. What message is "matching" this log path?

Those that reach the first destination? Or those that also reach the second one too?

While this configuration may not make sense in most cases, sometimes it is pretty useful. If you want to debug your configuration for example, then using a destination right in the middle-point would make sense. You could get a sample of what the messages look like that traverse this part of the configuration.

I decided that a log path is considered matching if it fully matches all filtering elements along a log path, with the exception of filtering done by destinations themselves. If a destination were to drop messages as a part of its operation, the log path that references this destination would still be matching. This more or less matches existing behaviour except for implementation bugs. These bugs are now fixed and the definition above is the guiding rule.

This means that this rule would not match if filter_app does not match:

log {
    source(whatever);
    destination(dest1);
    filter(filter_app);
    destination(dest2);
};

While this is doing the same, this would actually be matching:

log {
    source(whatever);
    destination(dest1);
    destination {
        log {
            filter(filter_app);
            destination(dest2);
        };
    };
};

The reason this would match is that the filter at this time is part of a "destination" reference, and filtering embedded in destinations do not affect the parent log path. This is useful when a destination wants to be selective about what is delivered and what is not. It may have relatively complex rules that affect delivery of messages.

What else?

Another closely related issue was conditional evaluation, which is also fixed here.

The problem was roughly this:

log {
    if {
       parser { something-that-parses-a-message-and-drops-it-if-invalid-format() };
    } else {
       parser { unmatched-messages-are-processed-here(); };
    };
    ...
    ...
    filter { we-drop-message-here(); };
};

Without this PR, the else branch of the if statements could be taken, even if the parser in the if branch itself was successful/matching. The reason that happened was that the "unmatched" state from the subsequent filter was propagated back to the if (incorrectly!), which caused the else to be taken (as well).

What is in this PR

So here's a list of changes in this PR:

• it fixes the bug in conditional evaluation (if/else/elif) by using a new ENL_CONDITIONAL expr node and dropping an older, partial solution of the same problem (LC_DROP_UNMATCHED)
• it defines the midpoint destination problem and makes sure destinations don't propagate their matched state to the parent logpath.
• it adds more details into the "info" member of log paths
• it adds light based tests to exercise both the conditional evaluation and the midpoint destination cases