Skip to content

net.http: implement http.download_file_with_progress/2, saving each chunk, as it is received, without growing the memory usage - #21633

Merged
spytheman merged 4 commits into
vlang:masterfrom
spytheman:net_http_streaming
Jun 2, 2024
Merged

spytheman merged 4 commits into
vlang:masterfrom
spytheman:net_http_streaming

Conversation

@spytheman

Copy link
Copy Markdown
Contributor
  • net.http: implement http.download_file_with_progress/2, to download big content to a file, by saving each chunk as it is received, without growing the memory usage
  • bump net.http bufsize to 64KB
  • tools: implement --sha1 and --sha256 options to v download (off by default).
  • net.http: enhance http.TerminalStreamingDownloader with speed, elapsed, eta, and bytes already downloaded

@spytheman spytheman changed the title net http streaming net.http: implement http.download_file_with_progress/2, saving each chunk, as it is received, without growing the memory usage Jun 1, 2024
@spytheman

Copy link
Copy Markdown
Contributor Author

On this PR:
image

On master, for the same file ~15MB file, the memory usage is several times higher:
image

@StunxFS

StunxFS commented Jun 2, 2024

Copy link
Copy Markdown
Contributor

Great job! 🎉

@spytheman

Copy link
Copy Markdown
Contributor Author

Great job! 🎉

Thank you.

@spytheman
spytheman merged commit 8504bea into vlang:master Jun 2, 2024
@ylluminate

Copy link
Copy Markdown
Contributor

Very nice work - as always.

raw-bin pushed a commit to raw-bin/v that referenced this pull request Jul 2, 2024
…hunk, as it is received, without growing the memory usage (vlang#21633)
@spytheman
spytheman deleted the net_http_streaming branch July 25, 2024 04:12
mut req := prepare(config)!
d.on_start(mut req, path)!
response := req.do()!
d.on_finish(req, response)!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spytheman great work, thanks!

Do you think it would be wise to guarantee on_finish() gets always called if on_start() finished successfully by defering it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants