fix malloc/mem.xfree inconsistency in isSameFuncLiteral

[rainers]

OutBuffer uses C malloc, but memory is released with mem.xfree

This can cause memory leaks when mem.xfree is using the GC.

Moving the OutBuffer up the call tree keeps memory management encapsulated.

[kinke]

LGTM. I suggest changing buf from ref to out, so that the return value is well-defined - it could wrongly return true if the buffer wasn't initially empty, or clear out any existing content in that buffer if the serialization fails. And ideally the Returns: docs would be updated too.

[rainers]

I'm not a fan of out parameters because they overwrite struct instances without checking whether they contain anything to cleanup, e.g. passing an OutBuffer with allocated data would cause a leak.

So I kept the ref but only return true if anything was appended. Updated the documentation, too.

[kinke]

I see a bunch of buf.length == 0 and buf.setsize(0) in the visitor, so this is clearly not suited for a non-empty buffer. Edit: So an assertion is IMO enough if you wanna keep the ref.

[rainers]

Indeed, I was probably assuming "serialization" was just appending. All content is cleared on error AFAICT, though that seems rather brittle (some writes are unconditonal, e.g. the closing ')' of a call).
I changed it to reset the buffer initially.