Fail loudly when LayoutWrappingEncoder has no layout (#1046)

[seonwooj0810]

Fixes #1046

Root cause

A LayoutWrappingEncoder whose layout ends up null — which happens when an <if>/<then>/<else> block is silently ignored (see the reports in #1046) — used to start() "successfully" (started = true) and then throw

NullPointerException: Cannot invoke "ch.qos.logback.core.Layout.doLayout(Object)" because "this.layout" is null

from encode() on every logging event. The event is dropped, so the appender runs while logging nothing and the only signal is a status warning. As several users noted on the issue, silent log loss is about the worst failure mode for a logging framework.

Notably headerBytes() and footerBytes() in the same class already guard if (layout == null) return null; — encode() was the one path that did not, so the class was internally inconsistent.

Change

• encode() returns null when layout == null, consistent with headerBytes()/footerBytes() (and OutputStreamAppender.writeBytes already treats a null/empty byte array as a no-op, so no event spam and no NPE).
• start() emits an ERROR status when layout == null, so the misconfiguration surfaces loudly at startup rather than as silent log loss at runtime.

Test

Added LayoutWrappingEncoderTest in logback-core:

• nullLayoutReportsErrorOnStartAndDoesNotThrowOnEncode — asserts an ERROR status is recorded at start() and that encode() returns null without throwing (fails before this change: NPE + no error status).
• encodesWhenLayoutIsSet — confirms the normal path is unaffected (error-free, correct output).

Verification done: (1) searched open/closed PRs for LayoutWrappingEncoder and issue #1046 — no in-flight PR (my other open PRs #1050/#1051 are unrelated); (2) no self-claim in the thread; (3) code-focused (.java only); (4) grepped master — start() still sets started = true unconditionally and encode() dereferenced layout directly; (6) no triage gate (not a spring-projects repo). Built and ran the new test with mvn -pl logback-core test on JDK 21 (release target 17): 2 passed, 0 failed.