Add support for SOURCE_DATE_EPOCH

[tonistiigi]

addresses #1058

Allows reproducible timestamps for layer and image timestamps.

Implemented as a frontend-opt because in the future, the same option could be detected by frontend for custom behavior
and the same value should also apply timestamps for FileOps. Definition for SOURCE_DATE_EPOCH can be found in https://reproducible-builds.org/docs/source-date-epoch/

In local and tar exporter, setting the build-arg sets overwrites the timestamp for the output files.

There is also a secondary implementation inside the Dockerfile frontend. This allows getting reproducible image manifests even on old buildkit, by just defining #syntax to point to new Dockerfile image.

In Dockerfiles the build-arg can also be exposed to inner processes by defining ARG SOURCE_DATE_EPOCH inside the stage.

This is the first stop toward better reproducible builds support that can be followed up with:

• Overwrite timestamp for files created with FileOp, eg. COPY/ADD in Dockerfiles dockerfile: add (ADD|COPY) --timestamp=<RFC3339Nano> #2911 @AkihiroSuda
• Possibly allow setting timestamp from Dockerfile, eg. #epoch= . Frontend side of this is already implemented and the frontend can pass the epoch value to the exporter(it can not overwrite the value if it has already been set by the user).
• Add some special case that allows using setting epoch to git commit timestamp when building from git. Currently need to do --build-arg SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
• Maybe allow configuring differ or overwriting files created by ExecOp

[crazy-max]

Can you update docs in https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#built-in-build-args for SOURCE_DATE_EPOCH

[sudo-bmitch]

Can we pass this into the RUN steps without defining an ARG in the Dockerfile? I.e. the same way we do for http proxy today?

[tonistiigi]

Can we pass this into the RUN steps without defining an ARG in the Dockerfile? I.e. the same way we do for http proxy today?

I think defining ARG should be required. This changes the behavior of the RUN and output of the created image so should be decision user makes. Proxy args are different because the expectation is that when two users are behind a different proxy they still generate the same image and proxy only affects their local network configuration and not the build result itself.

[sudo-bmitch]

I think defining ARG should be required. This changes the behavior of the RUN and output of the created image so should be decision user makes.

Since the user is running the build command, what if the forced ARG was added to the steps even if not defined in the Dockerfile (so it would show up in the history)? The goal I'm working towards is making a Dockerfile reproducible without needing to modify the Dockerfile itself.

[tonistiigi]

It would, for example, mean that build-cache does not apply between builds that use epoch and ones that do not, even if the run steps make no use of the added argument. I don't think avoiding changes in Dockerfile is a goal (in fact, we are probably adding #epoch), especially if the behavior of the Dockerfile commands changes. Ultimately it is the Dockerfile author who knows if the process they use in RUN step understands the env variable or not.

[AkihiroSuda]

Probably better to clarify the syntax version (1.5, unreleased)

[tonistiigi]

Updated

[AkihiroSuda]

LGTM but a couple of nits.
Also needs rebase.

[crazy-max]

LGTM

[tonistiigi]

@AkihiroSuda ok to merge?

[crazy-max]

needs rebase, PTAL @AkihiroSuda

[AkihiroSuda]

LGTM after rebase

[thaJeztah]

Looks like this needs a rebase again

[AkihiroSuda]

OCI exporter needs to be fixed to reset "org.opencontainers.image.created" annotation

buildkit/exporter/oci/export.go

Line 131 in 092e977

desc.Annotations[ocispecs.AnnotationCreated] = time.Now().UTC().Format(time.RFC3339)

[spacedub]

Is this going to apply as well for whiteout files?

[AkihiroSuda]

And this timestamp is also non-reproducible

buildkit/exporter/containerimage/export.go

Lines 246 to 249 in 092e977

	img := images.Image{
	Target: *desc,
	CreatedAt: time.Now(),
	}

[AkihiroSuda]

Is this going to apply as well for whiteout files?

Needs:

• archive: add WithSourceDateEpoch() for whiteouts containerd/containerd#7478

[AkihiroSuda]

LGTM, thanks

My comments above can be addressed later