Repository navigation
Snapshot memory exhaustion - #9712
Conversation
…in a rocksdb batch during reading a snapshot.
… a memory threshold is reached.
| "Rocksdb write rate of flushes and compactions.") | ||
| ("persistent-storage-mbytes-snapshot-batch", bpo::value<uint32_t>()->default_value(config::default_persistent_storage_mbytes_batch), | ||
| "Rocksdb batch size threshold before writing read in snapshot data to database.") | ||
|
|
There was a problem hiding this comment.
Do we really need a knob for something only used when reading a snapshot?
There was a problem hiding this comment.
The intent was that having this means that if someone is using rocksdb because they have limited resources, that they won't be forced to have as high of memory available the we default to, but that if someone has plenty of memory, that they can jack it way up and start faster.
@tbfleming @larryk85 @b1bart
There was a problem hiding this comment.
I don't see any harm in having the knob. And it would allow for operators with more robust machines to set the value higher and read the snapshots in as fast as possible (bringing a reserve node up to parity or a main node back after failure).
There was a problem hiding this comment.
How much faster is it when turning the knob up?
There was a problem hiding this comment.
I prefer to have a knob; fill-rocksdb had perf gains with high batch sizes (somewhere between 100-150 blocks of full-history deltas on mainnet). I no longer have the numbers, but the difference was large.
There was a problem hiding this comment.
I'll defer. I just believe we need to question every new knob as each and every one means additional testing and potential additional error reports when used incorrectly.
There was a problem hiding this comment.
My measurements with a mainnet snapshot:
50MB: 63s
200MB: 63s
1024MB: 61s
Change Description
Code previously was batching up all key-value pairs from db intrinsic tables during initialization from a snapshot. Changed to track how much memory was being stored and writing out the batch when it passes a newly added configuration threshold.
Change Type
Select ONE
Consensus Changes
API Changes
New configuration parameter:
"persistent-storage-mbytes-snapshot-batch"
Rocksdb threshold, in MB, to determine how much memory in Key-Value pairs to batch before writing data read from a snapshot. Depending on the resources available on your machine, increasing this number may result in faster loading of a snapshot.
Documentation Additions