Skip to content

Parallel restore: restore is now ~4x slower than the backup it undoes #6086

Closed

Description

@lvca

Context

Backup phase 1 (#6072, merged in #6080) made the full backup up to 27x faster. That moved the bottleneck: restore is now the slow half of a recovery.

Measured on the same 1.25 GB database, same machine (BackupCompressionBenchmark):

duration
Backup, level 1, 8 threads 0.68 s
Restore of that archive 2.9 s

Restore is now ~4x slower than the backup it undoes. For RTO - the number that actually matters when someone is restoring - the work in #6072 barely moved the needle, because it only addressed the half that was already the faster one after the change.

Where the time goes today

FullRestoreFormat.restoreDatabase() (integration/src/main/java/com/arcadedb/integration/restore/format/FullRestoreFormat.java:87) is a single sequential loop:

ZipEntry compressedFile = zipFile.getNextEntry();
while (compressedFile != null) {
  databaseOrigSize += uncompressFile(zipFile, compressedFile, databaseDirectory);
  compressedFile = zipFile.getNextEntry();
}

and uncompressFile (:113) inflates into an 8 KB buffer, writing each chunk to a FileOutputStream as it goes. One thread does inflate and write, alternating, for the whole archive.

Measure before building. It is not yet established whether restore is inflate-bound or write-bound - 1.25 GB in 2.9 s is ~430 MB/s, which is plausible as either on NVMe. The first task in this issue is to find out, because the two point at different fixes:

  • Write-bound -> the fix is overlapping inflate with I/O, and the 8 KB BUFFER (:49) is suspiciously small; raising it may be most of the win on its own for a fraction of the effort.
  • Inflate-bound -> the fix is real parallelism, below.

Proposed direction

Per-entry parallelism. ZIP entries are independent and each becomes its own output file, so N entries can be inflated and written concurrently. This is the natural unit and needs no format change.

The constraint that shapes the design: per-entry parallelism requires random access to the archive (ZipFile), but the current path is a stream (ZipInputStream), and two of its input sources cannot be made random-access:

  • openInputFile() (:147) supports http:// / https:// URLs - a one-shot stream.
  • decryptFile() (:171) wraps the input in a CipherInputStream for encrypted backups.

So a parallel restore cannot simply replace the existing path; it has to be a fast path for a plain local archive, with the current sequential stream retained for remote and encrypted inputs. That fallback is not optional and should be settled before any code is written.

Note that intra-entry parallelism (the mirror of what the backup does) is not available: the backup's chunk boundaries are not recorded anywhere in the archive, and recording them would be an archive-format change, which phase 1 deliberately avoided and which would break the "old backups restore" guarantee. A single entry is one deflate stream and must be inflated serially.

Scope

  • Establish whether restore is inflate-bound or write-bound, with numbers.
  • Raise the 8 KB copy buffer if that alone is material.
  • Per-entry parallel restore for the local-file case, with a documented thread setting consistent with arcadedb.backup.compressionThreads (-1 = auto, 0 = sequential legacy path).
  • Keep the sequential path for HTTP and encrypted archives, and cover both in tests.

Out of scope

  • Any archive-format change, including recording chunk boundaries to enable intra-entry parallelism.
  • Incremental restore / chain folding (Phase 3 of docs/optimize-backup.md).

Acceptance criteria

Follow-up to #6072. Related: docs/optimize-backup.md.

Activity

  1. added this to the 26.9.1 milestone on Aug 12, 2026
  2. self-assigned this
    on Aug 15, 2026
  3. added 2 commits that reference this issue on Aug 16, 2026
  4. removed
    in progressThe team is actively working on this issue
    on Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions