Skip to content

Enhancement: Implement Full Kubo Interoperability for Bitswap, UnixFS, and DHT #1347

@sumanjeet0012

Description

@sumanjeet0012

Issue Description:

Background:
The current py-libp2p implementation suffers from multiple isolation issues preventing it from seamlessly communicating with the broader IPFS network (specifically Kubo nodes).

Identified Problems:

  1. Incompatible CID Generation: MerkleDag uses CODEC_RAW for leaf chunks instead of dag-pb + UnixFS, producing CIDs entirely foreign to Kubo.
  2. Flat DAGs: The DAG structure creates flat, oversized root blocks for large files instead of a balanced layout, breaking Kubo's 174-link limits.
  3. Ephemeral Storage: MemoryBlockStore loses all fetched blocks on exit, making the node impractical for long-running environments.
  4. Kubo Disconnects: Fetching many blocks sequentially opens hundreds of streams, causing Kubo to issue GO_AWAY disconnects.
  5. DHT Record Rejection: DHT value records lack proper cryptographic signing, causing them to be rejected by Kubo peers.

Proposed Solution:
This epic will overhaul the Bitswap and Kademlia DHT sub-protocols to achieve the following:

  • UnixFS / Balanced Layout: Implement UnixFS Data wrapping for leaves and a balanced.Layout tree builder to generate byte-for-byte identical CIDs to Kubo.
  • Canonical Serialization: Enforce Links-before-Data ordering for DAG-PB protobufs.
  • Batch Message Streaming: Implement get_blocks_batch() to group wantlists and reuse streams.
  • Persistent Storage: Implement FilesystemBlockStore and a BlockService caching layer.
  • Payment Extension (1.3.0): Create a modular IBitswapExtension architecture, introducing BitswapPaymentClient, PaymentLedger, and BlockPricingEngine.
  • DHT Signing: Implement signed/verified Peer and Value records in the Kademlia DHT.
  • Type Safety & Streaming: Replace magic integers with typed Wantlist classes and introduce io.IOBase streaming for large file ingestion.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions