Feature: Implement MMap for In Memory IO

[pedrocarlo]

This PR implements MMap to support, in the future, persistence of an in-memory database to a file, as is tracked by this issue: #859. I wanted to thank @PThorpe92 a lot for letting me try implementing this feature.

Changes/Additions

• Swap BTreeMap<usize, MemPage> for MMapMut
• For Linux, utilize remap to attempt to resize the map in place, else move the mmap.
• Added a test suite to test for insertions, selects with ":memory" option selected.
• Added an environment variable, INITIAL_PAGES, that defines the initial number of pages that the MMap will be initialized with. The default is 16, that amounts to 16 * 4096 bytes = 64kb of memory allocated on startup.
• The resizing factor at the moment is set at 2. I picked this value as it is somewhat common place in hashmaps and other datastructures.
• Also, a CI enviroment value has been added in the testing github workflow to set the INITIAL_PAGES env var to 5, so that resizing happens more often in testing.

TODO's and Improvements

• As in mentioned in Support persisting in-memory database to disk #859, to support writing the in-memory database to disk Limbo would need to have a modification in its syntax and/or opcode.
• One suggestion for that would be to implement a PRAGMA statement that takes a file path or DB name to persistence the file on disk. This value would then be saved in the MemoryFile struct. This could happen by extending the File trait with a method e.g fn persist(db_path:&str) .
• Then, to improve the user experience, we could flush the contents of the MMap in WAL checkpoints and when the MemoryFile is dropped.
• TODO: implement tests that incorporate DELETE statements.

I would appreciate some feedback on my implementation and any other point I may have overlooked.

[penberg]

Looks sensible, @PThorpe92 did you have the opportunity to review this already?

[LtdJorge]

Hey @pedrocarlo for Linux we are importing rustix in io, which has mmap, munmap, mremap, etc. In case you want to use that instead of importing a new dependency.

The main difference is that rustix doesn't give you a clean safe API, it exposes unsafe functions, closer to the syscalls/libc but returning Result and such. If you want help, feel free to ping me.

[pedrocarlo]

Hey @LtdJorge! I agree. For the project, it would be better to use an already existing dependency instead of introducing a new one like memmap2. I started with memmap2 for the reasons you mentioned, but now that we have some functioning decent piece of code to test against, it would be a great idea to use rustix to do the same functionality. I think it would make sense to first merge this and then try to eliminate the dependency in favor of rustix. Also, would love to talk with you about this! I'm still inexperienced with some low-level systems and I really want to learn more about. Please tell me where I should ping you (discord, github, x, etc...).

[penberg]

Hey @pedrocarlo! I think dropping the new dependency is a pre-requisite for merging this, but otherwise looks sensible to me.

[pedrocarlo]

Hey @penberg! Unfortunately, I just noticed that rustix::mm module does not support windows or wasi. In my view, it would make sense to use rustix for unix and memmap2 for the rest. What do you think?

[PThorpe92]

@pedrocarlo I think it might make more sense to feature flag mmap for unix, then the others can use the original impl and we add no new dependencies

[pedrocarlo]

@PThorpe92 that seems the best path. Sad that persisting to database will only be available to unix for now. I will fix it either today or tomorrow.

[PThorpe92]

Don't want to hold this up for too much longer, but just a few minor comments. Overall looks good, glad we were able to keep the dependencies down 👍

[PThorpe92]

Sad that persisting to database will only be available to unix for now.

Indeed. Fortunately after #960 is merged it will be much easier to extend the IO layer :) It's possible that functionality will end up being baked into a vfs extension anyway, where we are much limited by the stricter package requirements of core

[pedrocarlo]

@PThorpe92 I think everything should be good to go now!

[PThorpe92]

Looks good to me 👍

[pedrocarlo]

@penberg is there anything more you think I should change here?

[pedrocarlo]

@penberg I just noticed that there is some conflicts in memory.rs as its trying to implement Send and Sync. How would this change impact the changes I am trying to do here?

EDIT:

@PThorpe92 gave me a hand on the rebase so everything should be fine now.