-
Notifications
You must be signed in to change notification settings - Fork 239
feat: implemented the perf protocol #1176
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 6 commits
Commits
Show all changes
48 commits
Select commit
Hold shift + click to select a range
fe89373
the first version of the perf protocol
ItshMoh ba19233
fixed the lint errors
ItshMoh a9d4848
changed the description
ItshMoh e0238b1
Merge branch 'main' into perfexamples
seetadev d977f50
fixed the yamux and noise header size errors
ItshMoh 98df9ab
Merge branch 'perfexamples' of github.com:ItshMoh/py-libp2p into perf…
ItshMoh 2bff259
wip: first interop perf experiment
acul71 5eded13
fix: Dockerfile as transport
acul71 99741f2
Merge branch 'main' into perfexamples
acul71 2c48b4c
fixed the lint issue
ItshMoh 195f711
moved the interfaces to abc.py file and restructure the code
ItshMoh efdb1cd
pushed the .rst files
ItshMoh 814bdda
Merge branch 'main' of github.com:ItshMoh/py-libp2p into perfexamples
ItshMoh 25e7a76
included the perf in examples.rst
ItshMoh 8d3a784
Merge branch 'main' into perfexamples
acul71 10d359d
pushed the newsfragments
ItshMoh 7fdb852
fixed thelength issue
ItshMoh cd80d31
Merge branch 'main' of github.com:ItshMoh/py-libp2p into perfexamples
ItshMoh e75e432
Merge branch 'perfexamples' of github.com:ItshMoh/py-libp2p into perf…
ItshMoh 80e2b96
fixed some issues in perf_test.py
ItshMoh a6b029a
fixed the 8bytes logic error
ItshMoh dad99d4
tests for the perf
ItshMoh 3903a99
Merge branch 'main' into perfexamples
seetadev 1f2f795
Merge branch 'main' of github.com:ItshMoh/py-libp2p into perfexamples
ItshMoh c0298da
changes the newsfragment name to the issue number
ItshMoh 6a7767f
Merge branch 'perfexamples' of github.com:ItshMoh/py-libp2p into perf…
ItshMoh a4b5c28
fixed the lint errors in init
ItshMoh ccb0c58
added the docstring about the high data to send back and resouce exha…
ItshMoh 4fefe15
Merge branch 'main' into perfexamples
seetadev d739255
fixed the perfoptions and header type issue
ItshMoh 8493989
fixed the lint issue by changing the type of chunk
ItshMoh d0cbd22
fixed teh lint errors by the ignore
ItshMoh c6c7e29
Merge branch 'main' into perfexamples
acul71 62c7542
Merge branch 'main' of github.com:ItshMoh/py-libp2p into perfexamples
ItshMoh 2e3fd23
Merge branch 'perfexamples' of github.com:ItshMoh/py-libp2p into perf…
ItshMoh 08372ca
used the .started variable in the handle message function
ItshMoh a2a922a
removed the configs that are not implemented in the set_stream_handle…
ItshMoh bba4e2a
Merge branch 'main' into perfexamples
seetadev 578ee35
Merge branch 'main' into perfexamples
seetadev 8fd2f3a
Merge branch 'main' into perfexamples
acul71 5a306b3
fixed the typo
ItshMoh d963285
Merge branch 'perfexamples' of github.com:ItshMoh/py-libp2p into perf…
ItshMoh 5aecd09
removed the config option detail from the newsfragment
ItshMoh 842714d
Merge branch 'main' into perfexamples
acul71 6bc8a23
Merge branch 'main' into perfexamples
acul71 a5acc03
fix: resolve perf interop addr rebuild and align block size
acul71 e8856b2
Merge branch 'main' into perfexamples
acul71 b43d971
doc: interop/ dir README.md for unified-test new repo
acul71 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,163 @@ | ||
| """ | ||
| Perf protocol example - Measure transfer performance between two libp2p nodes. | ||
|
|
||
| Usage: | ||
| # Terminal 1 - Run the server (listener) | ||
| python perf_example.py -p 8000 | ||
|
|
||
| # Terminal 2 - Run the client (measures performance to server) | ||
| python perf_example.py -p 8001 -d /ip4/127.0.0.1/tcp/8000/p2p/<PEER_ID> | ||
| """ | ||
|
|
||
| import argparse | ||
|
|
||
| import multiaddr | ||
| import trio | ||
|
|
||
| from libp2p import new_host | ||
| from libp2p.peer.peerinfo import info_from_p2p_addr | ||
| from libp2p.perf import PROTOCOL_NAME, PerfService | ||
|
|
||
| ONE_UNIT = 16 * 16 # 256 bytes | ||
| UPLOAD_BYTES = ONE_UNIT * 10 # 2560 bytes upload | ||
| DOWNLOAD_BYTES = ONE_UNIT * 10 # 2560 bytes download | ||
|
|
||
|
|
||
| async def run_server(host, perf_service) -> None: | ||
| """Run as a perf server - listens for incoming perf requests.""" | ||
| await perf_service.start() | ||
|
|
||
| print("\nPerf server ready, listening on:") | ||
| for addr in host.get_addrs(): | ||
| print(f" {addr}") | ||
|
|
||
| print(f"\nProtocol: {PROTOCOL_NAME}") | ||
| print("\nRun client with:") | ||
| print(f" python perf_example.py -d {host.get_addrs()[0]}") | ||
| print("\nWaiting for incoming perf requests...") | ||
|
|
||
| await trio.sleep_forever() | ||
|
|
||
|
|
||
| async def run_client( | ||
| host, perf_service, destination: str, upload_bytes: int, download_bytes: int | ||
| ) -> None: | ||
| """Run as a perf client - measures performance to a remote peer.""" | ||
| await perf_service.start() | ||
|
|
||
| maddr = multiaddr.Multiaddr(destination) | ||
| info = info_from_p2p_addr(maddr) | ||
|
|
||
| print(f"\nConnecting to {info.peer_id}...") | ||
| await host.connect(info) | ||
| print("Connected!") | ||
|
|
||
| print("\nMeasuring performance:") | ||
| print(f" Upload: {upload_bytes} bytes") | ||
| print(f" Download: {download_bytes} bytes") | ||
| print() | ||
|
|
||
| async for output in perf_service.measure_performance( | ||
| maddr, upload_bytes, download_bytes | ||
| ): | ||
| if output["type"] == "intermediary": | ||
| # Progress report | ||
| upload_bytes_out = output["upload_bytes"] | ||
| download_bytes_out = output["download_bytes"] | ||
| time_s = output["time_seconds"] | ||
|
|
||
| if upload_bytes_out > 0: | ||
| throughput = upload_bytes_out / time_s if time_s > 0 else 0 | ||
| print( | ||
| f" Uploading: {upload_bytes_out} bytes in {time_s:.2f}s ({throughput:.0f} bytes/s)" | ||
| ) | ||
| elif download_bytes_out > 0: | ||
| throughput = download_bytes_out / time_s if time_s > 0 else 0 | ||
| print( | ||
| f" Downloading: {download_bytes_out} bytes in {time_s:.2f}s ({throughput:.0f} bytes/s)" | ||
| ) | ||
|
|
||
| elif output["type"] == "final": | ||
| # Final summary | ||
| total_time = output["time_seconds"] | ||
| total_upload = output["upload_bytes"] | ||
| total_download = output["download_bytes"] | ||
| total_data = total_upload + total_download | ||
|
|
||
| print(f"\n{'=' * 50}") | ||
| print("Performance Results:") | ||
| print(f" Total time: {total_time:.3f} seconds") | ||
| print(f" Uploaded: {total_upload} bytes") | ||
| print(f" Downloaded: {total_download} bytes") | ||
| print(f" Total data: {total_data} bytes") | ||
| print(f" Throughput: {total_data / total_time:.0f} bytes/s") | ||
| print(f"{'=' * 50}") | ||
|
|
||
| await perf_service.stop() | ||
|
|
||
|
|
||
| async def run(port: int, destination: str, upload_mb: int, download_mb: int) -> None: | ||
| """Main run function.""" | ||
| from libp2p.utils.address_validation import find_free_port | ||
|
|
||
| if port <= 0: | ||
| port = find_free_port() | ||
|
|
||
| listen_addrs = [multiaddr.Multiaddr(f"/ip4/127.0.0.1/tcp/{port}")] | ||
| host = new_host(listen_addrs=listen_addrs) | ||
|
|
||
| # Create perf service | ||
| perf_service = PerfService(host) | ||
|
|
||
| async with host.run(listen_addrs=listen_addrs): | ||
| if destination: | ||
| # Client mode | ||
| await run_client( | ||
| host, | ||
| perf_service, | ||
| destination, | ||
| upload_mb * ONE_UNIT, | ||
| download_mb * ONE_UNIT, | ||
| ) | ||
| else: | ||
| # Server mode | ||
| await run_server(host, perf_service) | ||
|
|
||
|
|
||
| def main() -> None: | ||
| description = """ | ||
| Perf protocol example - Measure transfer performance between libp2p nodes. | ||
|
|
||
| To use: | ||
| 1. Start server: python perf_example.py -p 8000 | ||
| 2. Start client: python perf_example.py -d <MULTIADDR_FROM_SERVER> | ||
| """ | ||
|
|
||
| parser = argparse.ArgumentParser(description=description) | ||
| parser.add_argument("-p", "--port", default=0, type=int, help="listening port") | ||
| parser.add_argument("-d", "--destination", type=str, help="destination multiaddr") | ||
| parser.add_argument( | ||
| "-u", | ||
| "--upload", | ||
| default=10, | ||
| type=int, | ||
| help="upload size in units of 256 bytes (default: 10)", | ||
| ) | ||
| parser.add_argument( | ||
| "-D", | ||
| "--download", | ||
| default=10, | ||
| type=int, | ||
| help="download size in units of 256 bytes (default: 10)", | ||
| ) | ||
|
|
||
| args = parser.parse_args() | ||
|
|
||
| try: | ||
| trio.run(run, args.port, args.destination, args.upload, args.download) | ||
| except KeyboardInterrupt: | ||
| print("\nShutting down...") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| """ | ||
| Perf protocol for measuring transfer performance. | ||
|
|
||
| This module implements the libp2p perf protocol as specified in: | ||
| https://github.com/libp2p/specs/blob/master/perf/perf.md | ||
| """ | ||
|
|
||
| from .constants import ( | ||
| MAX_INBOUND_STREAMS, | ||
| MAX_OUTBOUND_STREAMS, | ||
| PROTOCOL_NAME, | ||
| RUN_ON_LIMITED_CONNECTION, | ||
| WRITE_BLOCK_SIZE, | ||
| ) | ||
| from .index import ( | ||
| IPerf, | ||
| PerfComponents, | ||
| PerfInit, | ||
| PerfOptions, | ||
| PerfOutput, | ||
| ) | ||
| from .perf_service import PerfService | ||
|
|
||
| __all__ = [ | ||
| # Constants | ||
| "PROTOCOL_NAME", | ||
| "WRITE_BLOCK_SIZE", | ||
| "MAX_INBOUND_STREAMS", | ||
| "MAX_OUTBOUND_STREAMS", | ||
| "RUN_ON_LIMITED_CONNECTION", | ||
| # Types | ||
| "PerfOutput", | ||
| "PerfInit", | ||
| "PerfOptions", | ||
| "PerfComponents", | ||
| # Interface | ||
| "IPerf", | ||
| # Implementation | ||
| "PerfService", | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # Protocol constants for the perf protocol | ||
| # https://github.com/libp2p/specs/blob/master/perf/perf.md | ||
|
|
||
| PROTOCOL_NAME = "/perf/1.0.0" | ||
| WRITE_BLOCK_SIZE = 65500 | ||
| MAX_INBOUND_STREAMS = 1 | ||
| MAX_OUTBOUND_STREAMS = 1 | ||
| RUN_ON_LIMITED_CONNECTION = False |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| """ | ||
| Perf protocol interfaces and types. | ||
| Spec: https://github.com/libp2p/specs/blob/master/perf/perf.md | ||
| """ | ||
|
|
||
| from abc import ABC, abstractmethod | ||
| from collections.abc import AsyncIterator | ||
| from typing import ( | ||
| TYPE_CHECKING, | ||
| Literal, | ||
| TypedDict, | ||
| ) | ||
|
|
||
| if TYPE_CHECKING: | ||
| from multiaddr import Multiaddr | ||
|
|
||
| from libp2p.abc import IHost | ||
|
|
||
|
|
||
| # ------------------------------- Types ------------------------------- | ||
|
|
||
|
|
||
| class PerfOutput(TypedDict): | ||
| """Output data from a performance measurement.""" | ||
|
|
||
| type: Literal["connection", "stream", "intermediary", "final"] | ||
| time_seconds: float | ||
| upload_bytes: int | ||
| download_bytes: int | ||
|
|
||
|
|
||
| class PerfInit(TypedDict, total=False): | ||
| """Initialization options for the perf service.""" | ||
|
|
||
| protocol_name: str | ||
| max_inbound_streams: int | ||
| max_outbound_streams: int | ||
| run_on_limited_connection: bool | ||
| write_block_size: int # Default: 65536 (64KB) | ||
|
|
||
|
|
||
| class PerfOptions(TypedDict, total=False): | ||
| """Options for a performance measurement run.""" | ||
|
|
||
| reuse_existing_connection: bool # Default: False | ||
|
|
||
|
|
||
| class PerfComponents(TypedDict): | ||
| """Components required by the perf service.""" | ||
|
|
||
| host: "IHost" | ||
|
|
||
|
|
||
| # ------------------------------- Interface ------------------------------- | ||
|
|
||
|
|
||
| class IPerf(ABC): | ||
| """Interface for the perf protocol service.""" | ||
|
|
||
| @abstractmethod | ||
| async def start(self) -> None: | ||
| """Start the perf service and register the protocol handler.""" | ||
| ... | ||
|
|
||
| @abstractmethod | ||
| async def stop(self) -> None: | ||
| """Stop the perf service and unregister the protocol handler.""" | ||
| ... | ||
|
|
||
| @abstractmethod | ||
| def is_started(self) -> bool: | ||
| """Check if the service is currently running.""" | ||
| ... | ||
|
|
||
| @abstractmethod | ||
| def measure_performance( | ||
| self, | ||
| multiaddr: "Multiaddr", | ||
| send_bytes: int, | ||
| recv_bytes: int, | ||
| options: PerfOptions | None = None, | ||
| ) -> AsyncIterator[PerfOutput]: | ||
| """ | ||
| Measure transfer performance to a remote peer. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| multiaddr : Multiaddr | ||
| The address of the remote peer to test against. | ||
| send_bytes : int | ||
| Number of bytes to upload to the remote peer. | ||
| recv_bytes : int | ||
| Number of bytes to request the remote peer to send back. | ||
| options : PerfOptions, optional | ||
| Options for the performance run. | ||
|
|
||
| Yields | ||
| ------ | ||
| PerfOutput | ||
| Progress reports during the transfer, with a final summary at the end. | ||
|
|
||
| """ | ||
| ... | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.