Introduce `diskann-record` crate to support serialization + deserialization of DiskANN indexes by suhasjs · Pull Request #1188 · microsoft/DiskANN

suhasjs · 2026-06-18T17:52:24Z

This PR is part 1/2 of #1079 and only introduces the diskann-record crate. See #1079 for sample output when the save::Save and load::Load traits from this crate are implemented for a simple in-memory index.

Does this PR have a descriptive title that could go in our release notes?
Does this PR add any new dependencies?
Does this PR modify any existing APIs?
Is the change to the API backwards compatible?
Should this result in any changes to our documentation, either updating existing docs or adding new ones?

Reference Issues/PRs

Part 1/2 of #1079. Also see #737.

What does this implement/fix? Briefly explain your changes.

Introduces two new traits (diskann_record::save::Save and diskann_record::load::Load)and a new create diskann-record to support file-based serialization + deserialization.

Any other comments?

…oadable] traits [THIS PR] + impls for structs [TODO]

Copilot

⚠️ Not ready to approve

Sidecar artifact path handling allows unsafe/incorrect path shapes (including potential directory escape on save) and needs validation hardening before merging.

Pull request overview

Introduces a new diskann-record crate that defines a versioned JSON-manifest + sidecar-artifact framework for persisting DiskANN-related structures, including a Save/Load trait surface, wire-level value model, and basic round-trip tests. This is positioned as the foundational crate for the follow-up PR that will implement these traits for real index types.

Changes:

Added save + load modules with Save/Load and Saveable/Loadable traits, plus save_fields! / load_fields! macros.
Implemented wire types (Value, Record, Handle), schema Version, and a lossless Number container for manifest numeric values.
Integrated the new crate into the workspace (members + workspace dependency) and added initial unit tests validating round-trips and handle escape rejection.

File summaries

File	Description
diskann-record/src/lib.rs	Crate-level API/docs, reserved-key policy, 64-bit platform assertion, and end-to-end tests.
diskann-record/src/version.rs	Defines `Version` and its string serialization/deserialization form.
diskann-record/src/number.rs	Adds `Number` wire type and safe narrowing conversions.
diskann-record/src/save/mod.rs	Save-side traits, entry point, macros, and primitive `Saveable` impls.
diskann-record/src/save/context.rs	Save-side context and sidecar writer + manifest finalization logic.
diskann-record/src/save/error.rs	Save-side error wrapper.
diskann-record/src/save/value.rs	Wire-level `Value`/`Record`/`Handle` representations and serde behavior.
diskann-record/src/load/mod.rs	Load-side traits, entry point, macros, and primitive `Loadable` impls.
diskann-record/src/load/context.rs	Load-side context/object/array APIs and sidecar reader.
diskann-record/src/load/error.rs	Load-side error type and recoverable-vs-critical classification.
diskann-record/Cargo.toml	New crate manifest and dependencies.
Cargo.toml	Adds `diskann-record` to the workspace and workspace dependencies.
Cargo.lock	Records the new workspace package entry.

Copilot's findings

Files reviewed: 12/13 changed files
Comments generated: 4

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

codecov-commenter · 2026-06-18T18:27:05Z

Codecov Report

❌ Patch coverage is 79.30769% with 269 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.65%. Comparing base (3aa44ac) to head (6c51b91).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
diskann-record/src/load/context.rs	67.78%	48 Missing ⚠️
diskann-record/src/load/error.rs	53.01%	39 Missing ⚠️
diskann-record/src/value.rs	82.11%	39 Missing ⚠️
diskann-record/src/backend/disk.rs	83.83%	32 Missing ⚠️
diskann-record/src/lib.rs	87.02%	31 Missing ⚠️
diskann-record/src/save/context.rs	58.82%	21 Missing ⚠️
diskann-record/src/save/mod.rs	70.49%	18 Missing ⚠️
diskann-record/src/number.rs	71.18%	17 Missing ⚠️
diskann-record/src/backend/memory.rs	86.66%	14 Missing ⚠️
diskann-record/src/load/mod.rs	87.93%	7 Missing ⚠️
... and 1 more

❌ Your patch status has failed because the patch coverage (79.30%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1188      +/-   ##
==========================================
+ Coverage   89.46%   89.65%   +0.18%     
==========================================
  Files         487      500      +13     
  Lines       92170    94482    +2312     
==========================================
+ Hits        82460    84706    +2246     
- Misses       9710     9776      +66

Flag	Coverage Δ
miri	`89.65% <79.30%> (+0.18%)`	⬆️
unittests	`89.31% <79.30%> (+0.19%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-record/src/save/error.rs	`100.00% <100.00%> (ø)`
diskann-record/src/version.rs	`94.54% <94.54%> (ø)`
diskann-record/src/load/mod.rs	`87.93% <87.93%> (ø)`
diskann-record/src/backend/memory.rs	`86.66% <86.66%> (ø)`
diskann-record/src/number.rs	`71.18% <71.18%> (ø)`
diskann-record/src/save/mod.rs	`70.49% <70.49%> (ø)`
diskann-record/src/save/context.rs	`58.82% <58.82%> (ø)`
diskann-record/src/lib.rs	`87.02% <87.02%> (ø)`
diskann-record/src/backend/disk.rs	`83.83% <83.83%> (ø)`
diskann-record/src/load/error.rs	`53.01% <53.01%> (ø)`
... and 2 more

... and 27 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

hildebrandmw

Thanks Suhas - I have one big architectural comment about allowing pluggable backend contexts that I think will address many of the concerns about how heavy this is as a dependency and support for VFS that I think is probably worth doing. Happy to help out if needed.

… and load paths now use ; feature gated serde and disk impls

…ite a Vec<struct> where each struct might produce a file with the same key

suhasjs · 2026-06-22T19:11:00Z

@hildebrandmw Requesting review for the following changes:

Moved up value.rs to reflect the shared usage between load/save paths
Trait objects to enable VFS pluggability (with a DiskContext impl for disk-based serialization, feature gated)
Change SaveContext::write to take Option<&str> to treat the input key as a hint. Filename is now INTEGER-{key}, where INTEGER is just the number of artifacts written so far. Creating a random value would pull in rand, and I didn't want to add that dependency.
Improved test coverage

hildebrandmw

Thanks Suhas - one more round on the SaveContext/LoadContext traits. I think it would be helpful to implement a purely in-memory backend to see how it interactws with the reader/writers. Such a backend can work directly with Value and avoid pulling in serde entirely.

hildebrandmw · 2026-06-23T16:37:18Z

+/// [`Writer::finish`] flushes the buffer, closes the file, and returns a [`Handle`].
+#[derive(Debug)]
+pub struct Writer<'a> {
+    io: BufWriter<File>,


One thing you might find useful when working through the backend design is developing an in-memory only implementation of SaveContext and LoadContext ;)

Adding the in-memory only impls actually improved the organization of the code now. Previously, the disk-variants of SaveContext and LoadContext were part of the {load|save}/context.rs. After adding the in-memory, I realized I could abstract that into a Backend enum and moved the two backends to backend/{memory|disk}.rs.

This keeps the Context traits in load|save directories and puts the impls under backend.

hildebrandmw · 2026-06-23T16:40:57Z

+        // the same `key` (or omitting it) still yields a unique file name.
+        let name = match key {
+            Some(key) => format!("{:03}-{}", files.len(), key),
+            None => format!("{:03}", files.len()),


Maybe include a little more information in the file name? Like 001-record.bin?

I thought about adding the .bin suffix, but then decided against it because we already have a .bin format that is pervasive throughout the repo, and there's no guarantee that the the save impl for a struct will look anything like the .bin file we know.

hildebrandmw · 2026-06-23T16:51:48Z

+                name,
+            )));
+        }
+        let full = self.dir.join(&name);


We can consider using cap_std to systematically force files to be generated in a subdirectory instead of relying on path name manipulation. It also adds a bit of safety to the saving process.

Doesn't that also pull in a whole new crate? Right now, we only depend on anyhow. Do you think it's worth adding?

…ls into it now; added in-memory ONLY variant of SaveContext and LoadContext --> moved to backend/; added an enum Backend to choose between Disk*Context and InMemory*Context; moved Disk*Context to backend/

…::write()

…ed WriterInner impls for DiskWriter and MemoryWriter; renamed InMemoryContext -> MemoryContext (same for InMemorySaveContext)

…ates

hildebrandmw

Thanks Suhas, this is coming together. I love how light-weight it is getting when the disk backend and serde are excluded. I have a few higher-level comments. Mostly about testing and fortifying the unhappy paths in addition to the happy paths.

hildebrandmw · 2026-06-26T20:41:22Z

+///
+/// The generic [`save`](super::save) entry point is parameterized over this trait so
+/// that the base crate carries no hard dependency on any particular implementation.
+pub trait SaveContext {


What are your thoughts on intending this to be a public extension point from the get-go versus keeping it pub(crate) for now? The issue with it currently is that it's public, but write returns a Writer and Writer::new is just pub(super). So external users cannot implement it. Even if they could, they would still need to create a Handle, which I firmly believe should not have a public constructor. It might be a better idea in the short-term to keep this (and LoadContext) pub(crate) at least initially. It's better to go from private to public than the other way.

I want to keep it public and not pub(crate). I see the issue though. We'll need to also make WriteInner pub as well. For now, I'll make this pub(crate) and we can open it up later if there's a need for it.

hildebrandmw · 2026-06-26T20:43:13Z

+    Bool(bool),
+    Number(Number),
+    String(Cow<'a, str>),
+    Bytes(Cow<'a, [u8]>),


I do think Bytes has value, but maybe not in its current form (where it will be store in JSON as integers rather than as a byte-string). We could either remove it, or figure out how to get it represented in binary form in JSON. Thoughts?

hildebrandmw · 2026-06-26T20:48:21Z

+pub enum Number {
+    U64(u64),
+    I64(i64),
+    F64(f64),


Something subtle here is that this silently loses data when the float64 value is infinity or NaN, as serde_json does not support these representations. We can support this by checking at for these values and using strings "inf", "neg_ing", and "nan" (and then teaching the deserializer to process these values).

hildebrandmw · 2026-06-26T20:52:46Z

+        }
+        // Reserve the name so the count advances and concurrent writers cannot collide;
+        // the placeholder is overwritten with the real bytes by `Writer::finish`.
+        files.insert(name.clone(), Vec::new());


Both backends do this to some extent, but it's maybe a little worse here. Eagerly registering Vec::new() means we cannot differentiate between a writer that was created but dropped before properly finishing, and a valid writer that just didn't write everything.

It's better to make files Mutex<HashMap<String, Option<Vec<u8>>>>. When we first pass out a file, set the corresponding value to None and only over-write it to Vec when finishing. This lets us differentiate properly between successfully completed files. As a bonus, we can also check in SaveContext::finish that all created files finished correctly and error out if this is not the case.

For the disk backend, we can do the same thing.

hildebrandmw · 2026-06-26T20:53:19Z

+                        "artifact file name hint {:?} must be a relative file name with no path \
+                         separators",
+                        key,
+                    )));


This behavior diverges from the disk backend. Can we make name mangling infallible like the disk path?

hildebrandmw · 2026-06-26T20:56:37Z

+            return Err(save::Error::message(format!(
+                "file {} already exists",
+                full.display()
+            )));


If a previous save fails for some reason, it will leave around junk which will then make a second round fail. I think defensively not over-writing is the call, but do we want to support some amount of cleanup on failure?

hildebrandmw · 2026-06-26T20:59:20Z

+        if version == T::VERSION {
+            T::load(object)
+        } else {
+            T::load_legacy(object)


Many of the built-in impls are not tested for round trippability, and this load_legacy path is completely uncovered. Please add tests for the unhappy paths as well and verify that load_legacy works as expected.

harsha-simhadri · 2026-06-26T21:41:41Z

@@ -0,0 +1,14 @@
+# DiskANN Record
+
+This crate provides a small framework for persisting structured Rust values as a 


what are "structured rust values"

harsha-simhadri · 2026-06-26T21:42:23Z

+field-by-field plumbing for plain structs. Every record carries a `Version` so loaders
+can detect schema changes and either upgrade or fall back through a probing chain.
+
+The goal is to allow crates like `diskann` to checkpoint their state without depending on


Are manifest and binary artifacts compatible across providers or defined per provider?

harsha-simhadri · 2026-06-26T21:44:30Z

@@ -0,0 +1,453 @@
+/*
+ * Copyright (c) Microsoft Corporation.


could the primitives in number.rs, value.rs, and version.rs be used from some existing external crate. Is there a strong reason to define this here

harsha-simhadri

couple of comments inline. will read backed code once you have final updates

harsha-simhadri

couple of comments inline. will read backed code once you have final updates

split saveload PR into two -- core with all the Save[Saveable]/Load[L…

45ddbcf

…oadable] traits [THIS PR] + impls for structs [TODO]

suhasjs self-assigned this Jun 18, 2026

suhasjs requested review from a team and Copilot June 18, 2026 17:52

suhasjs added enhancement New feature or request rust Pull requests that update rust code labels Jun 18, 2026

suhasjs added this to DiskANN backlog Jun 18, 2026

suhasjs moved this to Done in DiskANN backlog Jun 18, 2026

Copilot started reviewing on behalf of suhasjs June 18, 2026 17:53 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread diskann-record/src/save/context.rs Outdated

Comment thread diskann-record/src/save/context.rs Outdated

Comment thread diskann-record/src/load/context.rs Outdated

Comment thread diskann-record/src/save/mod.rs

suhasjs added 2 commits June 18, 2026 11:07

making copilot review happy

7d29f1c

making copilot review happy

f064610

hildebrandmw reviewed Jun 19, 2026

View reviewed changes

Comment thread diskann-record/src/save/context.rs Outdated

Comment thread diskann-record/src/value.rs

suhasjs added 4 commits June 22, 2026 10:47

moved up one level since both load/save paths use them

5ca175c

removed ContextInner; added new SaveContext, LoadContext traits; save…

2cd7e6e

… and load paths now use ; feature gated serde and disk impls

added file-level tests to improve coverage

6cea251

Changed SaveContext::write to take Option<&str>. This allows us to wr…

14ceecc

…ite a Vec<struct> where each struct might produce a file with the same key

hildebrandmw reviewed Jun 23, 2026

View reviewed changes

suhasjs added 5 commits June 23, 2026 14:20

gave Reader a new(); added Display and from_str to Version, serde cal…

407ff32

…ls into it now; added in-memory ONLY variant of SaveContext and LoadContext --> moved to backend/; added an enum Backend to choose between Disk*Context and InMemory*Context; moved Disk*Context to backend/

added fallback to None if key does not pass validation in SaveContext…

07f0020

…::write()

added a quick short README

0c7da43

abstracted away Backend --> Writer now stores a dyn WriterInner + add…

3f82e6b

…ed WriterInner impls for DiskWriter and MemoryWriter; renamed InMemoryContext -> MemoryContext (same for InMemorySaveContext)

added ReaderInner trait to allow trait object inside Reader + doc upd…

6c51b91

…ates

hildebrandmw reviewed Jun 26, 2026

View reviewed changes

harsha-simhadri reviewed Jun 26, 2026

View reviewed changes

		@@ -0,0 +1,14 @@
		# DiskANN Record

		This crate provides a small framework for persisting structured Rust values as a

Uh oh!

Conversation

suhasjs commented Jun 18, 2026

Reference Issues/PRs

What does this implement/fix? Briefly explain your changes.

Any other comments?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

⚠️ Not ready to approve

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hildebrandmw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

suhasjs commented Jun 22, 2026

Uh oh!

hildebrandmw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hildebrandmw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harsha-simhadri left a comment

Choose a reason for hiding this comment

Uh oh!

harsha-simhadri left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

codecov-commenter commented Jun 18, 2026 •

edited

Loading