smite-scenarios: warm up and freeze Eclair's JIT before the snapshot - #161
smite-scenarios: warm up and freeze Eclair's JIT before the snapshot#161erickcestari wants to merge 1 commit into
Conversation
Snapshot-restore fuzzing of Eclair started every iteration on a cold JVM, so the channel path ran interpreted. Add an Eclair-specific SnapshotSetup that, before the snapshot is taken: - drives thousands of open_channel exchanges on throwaway connections so HotSpot JIT-compiles the channel path (batched under Eclair's pending- channel rate limit), - waits for the compile queue to drain (polling jcmd Compiler.queue), - freezes the JIT with a catch-all Exclude directive so no compiler threads run during fuzzing. Keeps C1 only (-XX:TieredStopAtLevel=1): C2 measured ~28% slower here as its speculation deopts on varied fuzzing inputs. The runtime image switches to the full JDK for jcmd.
morehouse
left a comment
morehouse
left a comment
There was a problem hiding this comment.
I had a couple big issues when testing this locally:
- Eclair actually rejects concurrent
open_channelmessages, so ~90% of theopen_channelswe send during warmup get immediately rejected. - After snapshot Eclair is in a state where all its pending channel slots are occupied. Depending on timing of the first
SendOpenChannelafter that, Eclair may reject it due to "rate limit reached". I think we're warming up and snapshotting too fast, so Eclair doesn't have time to process the queue ofopen_channels and then remove pending channels after disconnect.
Probably we should pause after each open_channel message and wait for Eclair's accept_channel, so we can be sure that Eclair's processing is done. We then also probably want a sleep or ping-pong after each disconnect so that Eclair has time to tear the actors down and free up pending slots. This will be slower, but maybe we'll need fewer iterations if we actually wait to make sure Eclair fully processes everything before moving on.
I also have some ideas to improve further, but those are for follow-up work.
- Measure the individual effects of JIT freeze and warmup in isolation
- Try warming up more of the funding flow (e.g., through
funding_signed, mining blocks,channel_ready) - Try warming up gossip paths
| let directive = std::env::temp_dir().join("smite-jit-exclude-all.json"); | ||
| fs::write(&directive, "[ { match: [\"*.*\"], Exclude: true } ]\n")?; | ||
| match self.run_jcmd(&["Compiler.directives_add", &directive.to_string_lossy()]) { |
There was a problem hiding this comment.
I'd be curious how much difference the JIT freeze makes on its own -- is this the load-bearing change or is it the warmup?
| /// (<= [`WARMUP_OPENS_PER_CONNECTION`]) then drops, releasing the slots. | ||
| fn warmup<T: Target>( | ||
| target: &T, | ||
| chain_hash: [u8; 32], |
There was a problem hiding this comment.
Nit: we can probably drop the chain_hash parameter and just hard-code regtest
| /// Eclair rate-limits pending (half-open) channels per peer (default 99). Opens | ||
| /// stay pending until the connection drops, so batches stay well under the limit | ||
| /// to keep Eclair accepting (and JIT-compiling) each one. | ||
| const WARMUP_OPENS_PER_CONNECTION: usize = 40; |
There was a problem hiding this comment.
Nit: this const should go directly above warmup
| // Sync so Eclair processes (and JIT-compiles) the whole batch before conn | ||
| // drops at the end of this iteration, releasing its pending-channel slots. | ||
| ping_pong(&mut conn)?; |
There was a problem hiding this comment.
Eclair actually processes open_channel messages async, so the only way to be sure it has processed the whole batch is to wait for all the accept_channel messages. Eclair also rejects concurrent open_channels, so we actually need to send them one at a time and wait for each response before sending the next one. A few other benefits of doing this:
- More thorough warmup, potentially requiring much fewer iterations.
- We can detect if our
open_channelmessages ever start getting rejected (e.g., due to Eclair update) and raise the alarm.
| if let Ok(opts) = std::env::var("SMITE_ECLAIR_JAVA_OPTS") { | ||
| cmd.env("JAVA_OPTS", opts); | ||
| } |
There was a problem hiding this comment.
This ends up overriding the JAVA_OPTS set in init.sh, which includes our coverage agent and the C1 compiler flag. I think we probably want to append to JAVA_OPTS here instead.
| /// per iteration. The `channel_type` is `Anchors`, the only type Eclair (the | ||
| /// sole JVM target, and thus the only one that runs warmup) accepts. |
There was a problem hiding this comment.
Eclair accepts other channel types if they're negotiated (e.g., zero-fee-commits, scid-alias).
2 participants
Snapshot-restore fuzzing of Eclair started every iteration on a cold JVM, so the channel path ran interpreted. Add an Eclair-specific SnapshotSetup that, before the snapshot is taken:
Keeps C1 only (-XX:TieredStopAtLevel=1): C2 measured ~28% slower here as its speculation deopts on varied fuzzing inputs. The runtime image switches to the full JDK for jcmd.
Warming the JVM before taking the snapshot increased steady-state fuzzing throughput from 2.3 to 3.9 execs/sec for or the following input, a 1.70× (69.6%) improvement. (I've used #160 to benchmark it).
We can see the JVM logs optimizing the code with:
Logs:
Input
Before
After
Target: eclair
1. Summary Statistics
Median Coverage Over Time
Distribution Comparisons