Multi-Backend Object Storage for Go

Store anything. Anywhere.

Multi-backend object storage with composable middleware, streaming engine, content-addressable storage, and virtual filesystem — across 6 storage drivers. Part of the Forge ecosystem.

$go get github.com/xraph/trove

Get Started GitHub

Trove

Drivers

Middleware

Streaming

object.put

stored

mw.compress

zstd

stream.upload

chunked

GCS

Azure

Local

SFTP

Memory

Drivers

6 backends. One API.

Each driver implements the same storage interface. Local filesystem uses atomic file operations, S3 uses multipart uploads, GCS uses resumable uploads — no unified abstraction compromising capabilities.

Local Filesystem

Atomic file operations with sidecar .meta.json metadata. Directory-as-bucket layout with configurable base paths.

local.go

drv := localdriver.New()
drv.Open(ctx, "file:///tmp/storage")
 
t, _ := trove.Open(drv)
 
// Store an object
t.Put(ctx, "uploads", "photo.jpg", reader)
 
// Retrieve it
obj, _ := t.Get(ctx, "uploads", "photo.jpg")
defer obj.Close()

S3 / MinIO / R2

S3-compatible storage with multipart uploads, presigned URLs, server-side copy, and range reads.

s3.go

drv := s3driver.New()
drv.Open(ctx, "s3://us-east-1/my-bucket",
    s3driver.WithCredentials(key, secret),
)
 
t, _ := trove.Open(drv)
t.Put(ctx, "my-bucket", "data.csv", reader)

Google Cloud Storage

GCS with resumable uploads, presigned URLs, and compose-based multipart. Native IAM integration.

gcs.go

drv := gcsdriver.New()
drv.Open(ctx, "gcs://my-project/my-bucket",
    gcsdriver.WithCredentialsFile("sa.json"),
)
 
t, _ := trove.Open(drv)
t.Put(ctx, "my-bucket", "report.pdf", reader)

Azure Blob

Block blob storage with SAS tokens and multipart

SFTP

Remote file storage over SSH

In-Memory

Ephemeral storage for testing and caching

One API

Every driver plugs into the same trove.Open(drv) interface. Switch from local filesystem to S3 without changing application code. Middleware, routing, and streaming compose across all 6 backends.

Local FilesystemS3 / MinIO / R2Google Cloud StorageAzure BlobSFTPIn-Memory

main.go

// One interface. Any backend. Zero changes.
local := localdriver.New()
local.Open(ctx, "file:///tmp/storage")
 
s3drv := s3driver.New()
s3drv.Open(ctx, "s3://us-east-1/my-bucket")
 
// Same trove.Open — same API for every backend
localStore, _ := trove.Open(local)
s3Store, _ := trove.Open(s3drv)

Capabilities

Everything you need for storage

Drivers, middleware, streaming, CAS, VFS, routing, and security — Trove is a complete object storage toolkit for Go applications.

Multi-Backend Storage

6 storage drivers with a unified interface. Local filesystem, S3, GCS, Azure Blob, SFTP, and in-memory — swap backends without changing application code.

storage.go

drv := localdriver.New()
drv.Open(ctx, "file:///tmp/storage")
 
t, _ := trove.Open(drv,
    trove.WithDefaultBucket("uploads"),
    trove.WithBackend("archive", s3drv),
    trove.WithRoute("*.log", "archive"),
)

Composable Middleware

Direction-aware, scope-aware middleware pipeline. Zstd compression, AES-256-GCM encryption, BLAKE3 deduplication, ClamAV scanning, and invisible watermarking.

middleware.go

t, _ := trove.Open(drv,
    trove.WithMiddleware(
        compress.New(),
        encrypt.New(encrypt.WithKeyProvider(kp)),
        dedup.New(),
    ),
)

Streaming Engine

Chunked transfers with configurable sizes, backpressure handling, pause/resume, and managed stream pools. Upload, download, and bidirectional streaming modes.

stream.go

s, _ := t.Stream(ctx, "media", "video.mp4",
    stream.Upload,
    stream.WithChunkSize(16 * 1024 * 1024),
    stream.OnProgress(func(p stream.Progress) {
        log.Printf("%.1f%%", p.Percent())
    }),
)
defer s.Close()

Content-Addressable Storage

Store by content hash with BLAKE3, SHA-256, or XXHash. Automatic deduplication via reference counting and garbage collection support.

cas.go

t, _ := trove.Open(drv, trove.WithCAS(cas.BLAKE3))
 
ref, _ := t.CAS().Store(ctx, reader)
// ref.Hash = "blake3:a1b2c3..."
 
obj, _ := t.CAS().Retrieve(ctx, ref.Hash)
defer obj.Close()

Virtual Filesystem

io/fs.FS-compatible interface over flat object storage. Hierarchical directory view, metadata management, and standard Go file operations.

vfs.go

fsys := t.VFS("uploads")
iofs := vfs.NewIOFS(ctx, fsys)
 
// Serve files over HTTP
http.Handle("/files",
    http.FileServer(http.FS(iofs)))

Multi-Backend Routing

Route objects to backends via glob patterns or custom functions. Send logs to archive storage, images to CDN-backed S3, temp files to memory.

routing.go

t, _ := trove.Open(drv,
    trove.WithBackend("archive", s3drv),
    trove.WithBackend("cdn", gcsDrv),
    trove.WithRoute("*.log", "archive"),
    trove.WithRoute("images/*", "cdn"),
)

Capability Interfaces

Opt-in driver capabilities: multipart uploads, presigned URLs, server-side copy, versioning, change notifications, and lifecycle rules.

capabilities.go

if mp, ok := drv.(driver.MultipartDriver); ok {
    upload, _ := mp.InitiateMultipart(ctx,
        bucket, key)
    mp.UploadPart(ctx, upload.ID, 1, part)
    mp.CompleteMultipart(ctx, upload.ID)
}

Forge Extension

First-class Forge integration with REST API, ORM models, database migrations, DI container, and ecosystem hooks for Chronicle, Dispatch, and Warden.

extension.go

app := forge.New(
    troveext.New(
        troveext.WithDriver("local", localDrv),
        troveext.WithDriver("s3", s3Drv),
        troveext.WithDefaultDriver("local"),
    ),
)

Security

AES-256-GCM encryption with pluggable KeyProvider for key rotation. ClamAV-powered virus scanning on write. Invisible watermarking for PNG, JPEG, and PDF.

security.go

t, _ := trove.Open(drv,
    trove.WithMiddleware(
        encrypt.New(
            encrypt.WithKeyProvider(vault.Provider()),
        ),
        scan.New(scan.WithClamAV(clamAddr)),
    ),
)

Middleware

Composable by design.

Direction-aware, scope-aware middleware pipeline. Stack compression, encryption, deduplication, virus scanning, and watermarking in any order.

Compression

Zstd compression with automatic skip lists for already-compressed formats like JPEG, PNG, and ZIP.

Encryption

AES-256-GCM with pluggable KeyProvider interface for key rotation and integration with Vault or KMS.

Deduplication

BLAKE3 content hashing to detect and eliminate duplicate objects. Reference counting prevents premature deletion.

Learn about middleware

compress

encrypt

dedup

Compressed

Encrypted

Deduplicated

pipeline.go

t, _ := trove.Open(drv,
    trove.WithMiddleware(
        compress.New(),
        encrypt.New(encrypt.WithKeyProvider(kp)),
        dedup.New(),
    ),
)
 
// Middleware runs automatically on Put/Get
t.Put(ctx, "uploads", "data.csv", reader)

upload:

download:

Local

transfer:

GCS

Active

Streaming

Queued

stream.go

s, _ := t.Stream(ctx, "media", "video.mp4",
    stream.Upload,
    stream.WithChunkSize(16 * 1024 * 1024),
    stream.WithBackpressure(stream.BackpressureBlock),
)
defer s.Close()
 
for s.Next() {
    chunk := s.Chunk()
    // process chunk...
}

Streaming

Stream everything.

Chunked transfers with backpressure handling, pause/resume, and managed stream pools. Upload, download, and transfer large objects efficiently.

Chunked Transfers

Configurable chunk sizes with buffered I/O. Default 8MB chunks with 32KB stream buffers for optimal throughput.

Backpressure

Block, drop, or adaptive backpressure modes to prevent memory exhaustion during high-throughput transfers.

Stream Pools

Managed concurrency with configurable pool sizes, bandwidth throttling, and lifecycle hooks for progress, chunk, and completion events.

Explore streaming

Object Pipeline

From upload to storage.

Trove orchestrates the entire object lifecycle — middleware processing, backend routing, driver execution, and stream management.

Middleware Pipeline

Data flows through composable middleware stages. Compression reduces size, encryption secures content, deduplication eliminates redundancy.

Smart Routing

Glob patterns and custom functions route objects to the right backend. Route logs to archive storage, images to CDN-backed S3.

Capability Detection

Drivers expose optional capabilities like multipart uploads, presigned URLs, and range reads. Trove detects and uses them automatically.

Learn about the architecture

Upload

object.put

Middlewarepipeline

Driverbackend

Storagestored

mw.compress

zstd

mw.encrypt

AES-256

obj.stored

Compressed

Processing

Stored

Error

Developer Experience

Same API. Any backend.

Write storage code that works with local filesystem and deploy to S3 without changing a single line. Trove's unified interface means your code is backend-agnostic.

Local Filesystem

local.go

1package main
2 
3import (
4  "context"
5  "fmt"
6  "strings"
7 
8  "github.com/xraph/trove"
9  "github.com/xraph/trove/drivers/localdriver"
10)
11 
12func main() {
13  ctx := context.Background()
14 
15  // Create and open the driver
16  drv := localdriver.New()
17  drv.Open(ctx, "file:///tmp/storage")
18 
19  // Pass connected driver to Trove
20  t, _ := trove.Open(drv,
21    trove.WithDefaultBucket("uploads"),
22  )
23  defer t.Close(ctx)
24 
25  // Store an object
26  t.Put(ctx, "uploads", "hello.txt",
27    strings.NewReader("Hello, Trove!"))
28 
29  // Retrieve it
30  obj, _ := t.Get(ctx, "uploads", "hello.txt")
31  defer obj.Close()
32  fmt.Printf("stored %d bytes\n", obj.Size)
33}

Amazon S3

s3.go

1package main
2 
3import (
4  "context"
5  "fmt"
6  "strings"
7 
8  "github.com/xraph/trove"
9  "github.com/xraph/trove/drivers/s3driver"
10)
11 
12func main() {
13  ctx := context.Background()
14 
15  // Create and open the driver
16  drv := s3driver.New()
17  drv.Open(ctx, "s3://us-east-1/my-bucket",
18    s3driver.WithCredentials(key, secret),
19  )
20 
21  // Same API, different backend
22  t, _ := trove.Open(drv,
23    trove.WithDefaultBucket("my-bucket"),
24  )
25  defer t.Close(ctx)
26 
27  // Store an object
28  t.Put(ctx, "my-bucket", "hello.txt",
29    strings.NewReader("Hello, Trove!"))
30 
31  // Retrieve it
32  obj, _ := t.Get(ctx, "my-bucket", "hello.txt")
33  defer obj.Close()
34  fmt.Printf("stored %d bytes\n", obj.Size)
35}

Build storage-powered Go apps

Multi-backend object storage with composable middleware, streaming engine, content-addressable storage, and VFS across 6 storage drivers. Add Trove to your Go service in minutes.

$go get github.com/xraph/trove

Get Started View Examples