Issue Description:
Background:
The current py-libp2p implementation suffers from multiple isolation issues preventing it from seamlessly communicating with the broader IPFS network (specifically Kubo nodes).
Identified Problems:
- Incompatible CID Generation:
MerkleDag uses CODEC_RAW for leaf chunks instead of dag-pb + UnixFS, producing CIDs entirely foreign to Kubo.
- Flat DAGs: The DAG structure creates flat, oversized root blocks for large files instead of a balanced layout, breaking Kubo's 174-link limits.
- Ephemeral Storage:
MemoryBlockStore loses all fetched blocks on exit, making the node impractical for long-running environments.
- Kubo Disconnects: Fetching many blocks sequentially opens hundreds of streams, causing Kubo to issue
GO_AWAY disconnects.
- DHT Record Rejection: DHT value records lack proper cryptographic signing, causing them to be rejected by Kubo peers.
Proposed Solution:
This epic will overhaul the Bitswap and Kademlia DHT sub-protocols to achieve the following:
Issue Description:
Background:
The current
py-libp2pimplementation suffers from multiple isolation issues preventing it from seamlessly communicating with the broader IPFS network (specifically Kubo nodes).Identified Problems:
MerkleDagusesCODEC_RAWfor leaf chunks instead ofdag-pb+ UnixFS, producing CIDs entirely foreign to Kubo.MemoryBlockStoreloses all fetched blocks on exit, making the node impractical for long-running environments.GO_AWAYdisconnects.Proposed Solution:
This epic will overhaul the Bitswap and Kademlia DHT sub-protocols to achieve the following:
balanced.Layouttree builder to generate byte-for-byte identical CIDs to Kubo.get_blocks_batch()to group wantlists and reuse streams.FilesystemBlockStoreand aBlockServicecaching layer.IBitswapExtensionarchitecture, introducingBitswapPaymentClient,PaymentLedger, andBlockPricingEngine.Wantlistclasses and introduceio.IOBasestreaming for large file ingestion.