Public API for compilers

[kitsonk]

For tracking purposes, please don't work on this without discussing with Ry or myself.

Having a public API that is similar to how we perform TypeScript compilation is a good idea. It would allow JS->JS transpilation (e.g. those who need Babel custom plugins or Flow) or other languages (e.g. CoffeeScript).

Related to #1738 and some other work in rationalising the compiler APIs internally, but we should be able to support loading a compiler in a web worker and instructing the privileged side what resources should be sent to that runtime compiler.

[daniele-orlando]

Dart and Elm are other candidate languages.

[islishude]

I think Dartlang is better than TypeScript.

[kitsonk]

@islishude then you are probably looking at the wrong project.

[afinch7]

What you really want is a extensible module loading system(not resolution as already discussed this is prone to problems). I have quite a few ideas on this one, and even some existing code. I've been working on similar idea for while now, and I think the only good way to design this is with a two stage loading process:

1. First "load"(what this means is really dependent on the media type) modules to js or ts(possibly with embedded wasm) source, since these are the only formats that the typescript compiler understands(other than JSON).
2. Compile using something similar to the existing ts compiler.

The first step would be the only part that is user extensible.

[kitsonk]

The public compiler API would need to:

1. Allow a user to register a compiler. That registration would include extensions and media types that it should compile. For security and consistency reasons, TypeScript media types and extensions should be disallowed, but JSON and JavaScript could potentially be registered. This would be a special type of web worker. I am not certain if this should be a new op or if, we add non-standard customisations to the options of new Worker(url, options) to provide the appropriate information.
2. When the media type is encountered, Rust would post a message on the worker providing the module specifier and refer.
3. The userland compiler would be able to fetch resources using the fetch_module_meta_data op. We would enforce the read file/network security on this (unlike the built in compiler).
4. The userland compiler would post back via the web work API to Rust the compiled module code, source map and any diagnostics.
5. The userland compiler would need to be able to be unregistered. (I guess we would just use Worker.terminate())

[afinch7]

You specified separating compilers by media type, but I don't think that extension(media type) is really a reliable way to do this(how would you handle no extension). I guess you could decide to on a list of supported media types, but I feel that would be very limiting. You could load modules by manifest and have the manifest tell the loader system what sort of loader to use, but that already sounds way to much like package.json if you ask me. That might be fine if some thought was put into it, but I don't think thats really what the deno community wants.

In general the more complicated your expectations get the more difficult this will be to implement, and the more bugs we will encounter in the process.
It might be much more simple to decide that user compilers should be trusted code, and expect them to handle the retrieval of resources required to complete their tasks thus the nomenclature loader would be more fitting.
The expectations for said loaders could be as simple as:

1. Loaders will be given a module's fully qualified url(this could be generated using native url parsing I.E. new URL(specifier, referrer ? referrer : defaultUrl)).
2. Loaders will also be given a modules referrer information: origin url, source code, source map, media type, source loader(what loader was used to provide this resource?), etc.
3. Loaders would be expected to take the information from 1 and 2, and accurately as possible return a ts or js module that represents that modules url(or error out if not possible).
4. Loaders should be given the ability to error out on a request for any reason(and should be encouraged to error out as soon as possible).
5. Loader priority should be determined by their order in the list of configured loaders, and each loader must be given an attempt and error out before trying the next one.
6. Loaders should be designed to be platform agnostic, so they can be integrated in tooling like a typescript language services plugin. This would most likely be achieved by having the implementation passing the loaders platform specific implementations of a shared resource accessor api.

The new dynamic import could be used to load these "loaders" as modules with a defined structure.
I tried to describe this as best as possible, but I figured I might be able to better represent my ideas in typescript interfaces and I also included a simple example implementation: https://gist.github.com/afinch7/4356a4377ec20dc336456d4639777578.

This would absolve the implementation of a lot of potentially complicated responsibilities, and even allow loaders to make security decisions about the content they are attempting to load(like a browser would if you tried to make a cross origin request).

A simple approach like this could support just about any use case, and could be universal enough to enable parity between the deno compiler and a typescript language services plugin(Seamless accurate integration in the editor is rare for systems like this). It also wouldn't be limited to javascript transpilers or json processing you could very easily compile just about anything into a javascript or typescript module. Flatbuffers definitions could be compiled to a typescript module at runtime, or c++, c, or even rust source code could be compiled to wasm and embedded into a js/ts module at runtime. You could have a nearly completely language agnostic platform, and give the developer the information to actually use it effectively.

[afinch7]

This might require a preflight check of sorts to check for redirects #1742 or give deno control of fetching the module "entry point".

[kitsonk]

You specified separating compilers by media type, but I don't think that extension(media type) is really a reliable way to do this

This is exactly how we do it today. media type != extension.

(how would you handle no extension)

Without a media type, that is insecure. We wouldn't want to allow a file like that to be processed... Files without extensions require a media type.

It might be much more simple to decide that user compilers should be trusted code, and expect them to handle the retrieval of resources required to complete their tasks thus the nomenclature loader would be more fitting.

We don't trust our own compiler. It does not do module resolution. That is up to privileged/Rust, as is appropriate.

There is nothing preventing someone from implementing a loader and eval'ing code today with the right Deno permissions. A public API for a userland compiler needs to follow the pattern of the built in compiler.

[afinch7]

Those are valid concerns. You want something that doesn't require any user setup or config, and my approach would require user configuration to work thus it falls outside of the deno philosophy.

It will always be distributed as a single executable - and that executable will be sufficient software to run any deno program. Given a URL to a deno program, you should be able to execute it with nothing more than the 50 megabyte deno executable.

I think that pretty much settles what direction deno should go, but I still have my concerns with the idea of a untrusted compiler. I think my main concern here is how can you in any way trust the code a compiler emits if you don't have full trust in the compiler as a end user.

In general I think we are both on completely different pages right now, so I want to do what is needed to put us all on the same page with this one.

[rdeforest]

[snip]

I still have my concerns with the idea of a untrusted compiler. I think my main concern here is how can you in any way trust the code a compiler emits if you don't have full trust in the compiler as a end user.

[snip]

Just a lurker here, but I think what @kitsonk means by trusted/untrusted isn't what you think. You're right that one can't delegate code generation without the risk that the output will do something unwanted. Ken Thompson addressed this famously in his "Reflections on Trusting Trust" presentation in 1984.

The objective of treating the compiler as untrusted is to limit the damage it can do. Isolating the compiler prevents it from (for example) invoking /bin/bash in hopes of exploiting that unrelated program. The isolation is a reduction in attack surface as part of a defense-in-depth strategy.

The reason it's acceptable to risk nefarious or broken compiler output is because there is no architectural way to avoid the risk. The risk has to be addressed at a different layer, such as via module signing, code review, webs of trust, insurance mechanisms, etc.

I hope my comment is helpful.