Use umoci Go package rather than the command

[stgraber]

We are currently using the umoci command from our client package rather than importing the relevant Go packages from umoci and running things in-process.

By moving to the native Go packages from umoci, we should be able to not only remove an external dependency but also save ourselves a bit of time by being able to directly unpack the data retrieved by skopeo into the resulting tarball by using some of the VFS abstractions from umoci.

[stgraber]

@cyphar do you have a pointer on that VFS abstraction you once mentioned around umoci? Basically want to see if we could have umoci be tricked into directly writing into our tarball through a tarwriter.

[cyphar]

What parts of an image would you like to be put in the tarball?

If you want to extract all of the layers and flatten them into a tarball you could probably do that by creating a fake filesystem abstraction (that implements the FsEval interface) which writes to an in-memory version of the filesystem that then gets serialised, but I doubt that would be better than just creating a tmpfs and extracting there. Would be a lot more code, for one.

Also you need to provide an image store for umoci to access -- I'm not sure if the plan is to continue to use skopeo to create an OCI image layout directory and then operate on that, or do the entire thing in-memory. In principle you could fake that too (cas.Engine is an interface you could implement however you like -- I was planning to eventually add remote registry support as an alternative cas.Engine interface) but idk how difficult it would be to get that to work.

FWIW, the docs are at https://pkg.go.dev/github.com/opencontainers/umoci but I would probably need a bit more info to be able to point you in the right direction.

[stgraber]

Sounds like our current approach of doing a full unpack on disk and then generating a tarball from that will remain the easiest here :)

We basically use skopeo to fetch the layers, then umoci to flatten that as a rootfs which we then immediately turn into a tarball to import as an Incus image. It would have been cool to avoid part of that filesystem interaction by wiring the rootfs flattening directly into a Go tar writer, but the tar writer isn't the most flexible thing ever, so we'd need files to show up in their final form and basically in the correct order which may be too much to ask.

[cyphar]

Yeah, the problem is that emulating everything that goes into how files would be theoretically unpacked without an actual filesystem is quite complicated -- just using a tmpfs would lead to far less bugs. And outputting them in-order would require you to scan the archives to know which layer contains the latest version or some other equally horrific thing.