autodrive

Dumb, permissionless, but secure, real-time, multi-writer, distributed file system

Install

npm i autodrive

Usage

import Autodrive from "autodrive"
import b4a from "b4a"
import Corestore from "corestore"
import { once } from "bare-events"
import fs from "fs"

const driveA = new Autodrive(new Corestore("./storageA"))
await driveA.ready()
const driveB = new Autodrive(new Corestore("./storageB"), driveA.key)
await driveB.ready()

await driveA.put("/example.txt", b4a.from("Hello, World!"))
const data = await driveB.get("/example.txt")

const ws = driveB.createWriteStream("/example.txt")
const rs = fs.createReadStream("./file.txt")
rs.pipe(ws)
await once(ws, "finish")

for await (const chunk of driveA.createReadStream("/example.txt")) {
  console.log(b4a.toString(chunk))
}

await driveA.del("/example.txt")

API

const drive = new Autodrive(store, key, options)

Create new Autodrive. store must be an instance of Corestore. If you provide a key you get read access to the drive at key. Options are passed-through to underlying autobase.

await drive.ready()

Wait until internal state is loaded. Use it once before accessing synchronous properties like key.

await drive.close()

Fully close this drive and all internal hypercores used by it.

drive.id

The ID of the drive, this consists of base-z-32 encoded key of the drive.

drive.key

The binary hash of the drive's signers. Giving this someone grants full read access to the drive.

drive.discoveryKey

Hash of the drive.key. Use this to e.g. replicate drive over Hyperswarm without leaking read access.

drive.keyPair

For writable drives this is the key pair used to authenticate local writer. null otherwise.

drive.encrypted

Boolean indicating whether this drive uses encryption.

drive.encryptionKey

Buffer containing the key used to encrypt this drive. null otherwise.

drive.local

Local writer hypercore. If the drive is not writable this is null.

drive.db

The underlying file database, an instance of Hyperbee.

drive.store

The Corestore used to instantiate this drive.

drive.corestore

The Corestore used to instantiate this drive. Hyperdrive compat.

drive.version

The current version of the drive's file database.

drive.writable

Boolean indicating whether this drive has write access.

drive.readable

Should always be true.

drive.supportsMetadata

Whether this drive supports file metadata. Should always be true.

const blobs = await drive.getBlobs(forWriter?: Uint8Array)

Get the Hyperblobs instance for the given writer. If forWriter is omitted, returns Hyperblobs for the local writer.

await drive.addWriter(key: Uint8Array, { indexer?: boolean })

Add writer to this drive. key can be obtained by accessing drive.local.key on the writer to be added. Can only be called by existing writer.

await drive.removeWriter(key: Uint8Array)

Remove a writer from this drive. key should be drive.local.key of the writer to be removed. Can only be called by existing writer.

await drive.update()

Try to download the latest version of the files database.

const stream = drive.replicate(isInitiator: boolean)

See Corestore.replicate()

const done = drive.findingPeers()

Indicating that you're finding peers in the background, operations will be on hold until done() is called. Call done() when current iteration finishes, e.g. after swarm.flush().

const checkout = drive.checkout(version: number)

Checkout an earlier database version. This allows to go back in time to an earlier filesystem state.

const mirror = drive.mirror(dest: Drive, options?: MirrorDrive Options)

Mirror this drive into an instance of either another Autodrive, Hyperdrive, or Localdrive. Returns an instance of MirrorDrive.

const item = await drive.entry(path: string | Node, options?: { follow?: boolean, wait?: boolean, timeout?: number })

Get an entry in the files database.

• If follow is true and the entry is a symlink, this will resolve the destination.
• wait: false returns immediately, if the entry is not locally available, it will return null.
• timeout specifies how long to wait for the entry data to download. The default timeout of 0 means wait indefinitely.

const buffer = await drive.get(path: string | Node, options?: { follow?: boolean, wait?: boolean, timeout?: number })

Get the file contents. Takes either a path, or a Node object returned by e.g. drive.entry(). Options are the same as drive.entry().

await drive.put(path: string, data: Uint8Array, options?: { executable?: boolean, metadata?: JSON })

Write data to the filesystem, either creating or updating a file. For large data consider drive.createWriteStream().

await drive.del(path: string)

Delete a file from filesystem.

for await (const chunk of drive.createReadStream(path: string, options?: { start?: number, end?: number, length?: number, follow?: boolean, wait?: boolean, timeout?: number }))

Get a readable stream of file data at path. Options are same as drive.entry() with the addition of range options:

• start: byte offset at which to start reading
• end: byte offset at which to stop
• length: how many bytes to read

const ws = drive.createWriteStream(path: string, options?: { executable?: boolean, metadata?: JSON })

Efficiently write data at given path, returns a writable stream. Options are the same as drive.put().

await symlink(path: string, dest: string, options?: { executable?: boolean, metadata?: JSON })

Create a symlink to another location in the filesystem. Options are the same as drive.put().

await drive.exists(path: string, options?: { wait?: boolean, timeout?: number })

Check if the file at given path exists. Options are the same as drive.entry() without the follow.

for await (const item of drive.list(folder?: string, options?: { recursive?: boolean }))

List file entries at a given folder. If recursive: true also descends into the subdirectories.

for await (const filename of drive.readdir(folder?: string))

List filenames of entries at given folder.

const int = drive.compare(a: Node, b: Node)

Returns 0 if entries are the same, 1 if a is older, and -1 if b is older.

for await (const { left: Node, right: Node } of drive.diff(version: number, folder?: string, options?: { limit?: number }))

Efficiently create a stream of the shallow changes to folder between version and drive.version. Each entry is sorted by key. If an entry exists in drive.version of the folder but not in version, then left is set and right will be null, and vice versa.

const watcher = drive.watch(folder?: string)

Returns an iterator that listens on folder to yield changes, by default on /.

Usage example:

for await (const [current, previous] of watcher) {
  console.log(current.version)
  console.log(previous.version)
}

Those current and previous are Autodrive snapshots that are auto-closed before next value. Don't close those snapshots yourself because they're used internally, let them be auto-closed.

await watcher.ready()

Waits until the watcher is loaded and detecting changes.

await watcher.destroy()

Stops the watcher. You could also stop it by using break in the loop.

await drive.download(folder?: string, options?: { recursive?: boolean })

Downloads the blobs corresponding to all entries in the drive at paths prefixed with folder.

await drive.downloadDiff(version: number, folder?: string, options?: { limit?: number })

Downloads all the blobs in folder corresponding to entries in drive.checkout(version) that are not in drive.version. In other words, downloads all the blobs added to folder up to version of the drive.

const { blocks: number } = await drive.clear(path: string | Node, options?: { diff?: boolean })

Deletes the blob from storage to free up space, but the file structure reference is kept. Pass { diff: true } to get stats about cleared blocks. Otherwise returns void.

const { blocks: number } = await drive.clearAll(options?: { diff?: boolean })

Clear all blobs from storage, freeing up space. Options are the same as drive.clear()

await drive.purge()

Delete all data used by this drive from the local filesystem, effectively erasing the drive.

Types

Blob

The ID object identifying data store in blobs hypercores, paired with a source this allows to retrieve the actual binary data.

interface Blob {
  blockOffset: number
  blockLength: number
  byteOffset: number
  /** The size of this file in bytes */
  byteLength: number
}

Entry

The main file object Autodrive deals with.

interface Entry {
  /** The writer core key that produced this version of a file */
  source: Uint8Array
  /** Blob object pointing to the actual file data */
  blob?: Blob | null
  executable: boolean
  /** If this is a symlink, linkname will point to the destination */
  linkname?: string | null
  /** Any additional data you want to associate with the file, stored as JSON */
  metadata?: JsonValue
}

Node

Carry over from hyperbee, Node wraps entry with additional metadata.

interface Item {
  /** The database version when this entry was written */
  seq: number
  /** The path at which this entry is stored */
  key: string
  /** The Entry object */
  value: Entry
}