Repository navigation
Description
Context
Backup phase 1 (#6072, merged in #6080) made the full backup up to 27x faster. That moved the bottleneck: restore is now the slow half of a recovery.
Measured on the same 1.25 GB database, same machine (BackupCompressionBenchmark):
| duration | |
|---|---|
| Backup, level 1, 8 threads | 0.68 s |
| Restore of that archive | 2.9 s |
Restore is now ~4x slower than the backup it undoes. For RTO - the number that actually matters when someone is restoring - the work in #6072 barely moved the needle, because it only addressed the half that was already the faster one after the change.
Where the time goes today
FullRestoreFormat.restoreDatabase() (integration/src/main/java/com/arcadedb/integration/restore/format/FullRestoreFormat.java:87) is a single sequential loop:
ZipEntry compressedFile = zipFile.getNextEntry();
while (compressedFile != null) {
databaseOrigSize += uncompressFile(zipFile, compressedFile, databaseDirectory);
compressedFile = zipFile.getNextEntry();
}and uncompressFile (:113) inflates into an 8 KB buffer, writing each chunk to a FileOutputStream as it goes. One thread does inflate and write, alternating, for the whole archive.
Measure before building. It is not yet established whether restore is inflate-bound or write-bound - 1.25 GB in 2.9 s is ~430 MB/s, which is plausible as either on NVMe. The first task in this issue is to find out, because the two point at different fixes:
- Write-bound -> the fix is overlapping inflate with I/O, and the 8 KB
BUFFER(:49) is suspiciously small; raising it may be most of the win on its own for a fraction of the effort. - Inflate-bound -> the fix is real parallelism, below.
Proposed direction
Per-entry parallelism. ZIP entries are independent and each becomes its own output file, so N entries can be inflated and written concurrently. This is the natural unit and needs no format change.
The constraint that shapes the design: per-entry parallelism requires random access to the archive (ZipFile), but the current path is a stream (ZipInputStream), and two of its input sources cannot be made random-access:
openInputFile()(:147) supportshttp:///https://URLs - a one-shot stream.decryptFile()(:171) wraps the input in aCipherInputStreamfor encrypted backups.
So a parallel restore cannot simply replace the existing path; it has to be a fast path for a plain local archive, with the current sequential stream retained for remote and encrypted inputs. That fallback is not optional and should be settled before any code is written.
Note that intra-entry parallelism (the mirror of what the backup does) is not available: the backup's chunk boundaries are not recorded anywhere in the archive, and recording them would be an archive-format change, which phase 1 deliberately avoided and which would break the "old backups restore" guarantee. A single entry is one deflate stream and must be inflated serially.
Scope
- Establish whether restore is inflate-bound or write-bound, with numbers.
- Raise the 8 KB copy buffer if that alone is material.
- Per-entry parallel restore for the local-file case, with a documented thread setting consistent with
arcadedb.backup.compressionThreads(-1= auto,0= sequential legacy path). - Keep the sequential path for HTTP and encrypted archives, and cover both in tests.
Out of scope
- Any archive-format change, including recording chunk boundaries to enable intra-entry parallelism.
- Incremental restore / chain folding (Phase 3 of
docs/optimize-backup.md).
Acceptance criteria
- Restore duration measured before and after at several database sizes, in the same shape as the Speed up full backup: configurable compression level and parallel compression (Phase 1 of backup optimization)Â #6072 benchmark table, and reported in the PR.
- A restored database is byte-identical to what the sequential path produces -
DatabaseComparatoragainst the source, for plain, encrypted and HTTP inputs. - Old archives (produced before Speed up full backup: configurable compression level and parallel compression (Phase 1 of backup optimization)Â #6072, single-threaded level 9) restore unchanged.
- Peak heap bounded and stated, as the backup path does.
Follow-up to #6072. Related: docs/optimize-backup.md.
Activity
perf(#6086): the restore is parallel per entry, and stops reading the…
perf(#6086): the restore is parallel per entry, and stops reading the…