2024-07-22 Waku Weekly

Milestone - Store Service Upgrade §

• Store v3-beta - Message Hashes

  • achieved:
    • [research] Proof of concept of store testing with waku-simulator
  • next:
    • [research] Storev3 test plan and integrating tools for testing
  • blockers:
    • [research] Some complexities around merging/testing, currently handled by nwaku team

• DOS protection for req-res protocols and metrics

  • achieved:
    • [nwaku] BW metrics per shard: implemented per shard metric collection feat: Proper bandwidth metrics per shard
    • [nwaku] Added dashboard panels for relay per shard traffic
    • [nwaku] Added dashboard panels for non-relay protocols data traffic
    • [nwaku] Added dashboard panels for non-relay protocols request rates
    • [nwaku] Released feat: Enforce service specific rate limits
      • Added per peer filtering of high users
      • Load balancing compensation applied to token replenish
      • Filter service specific limits for ping and subscribe per peer

• PostgreSQL Maintenance

  • achieved:
    • [nwaku] Improved and merged Sonda - canary service to check liveness of store nodes
    • [nwaku] Merged store-v3 cursor fix chore(archive): archive and drivers refactor
  • next:
    • [nwaku] PostgreSQL query optimizations chore: query performance in PostgreSQL by avoiding ordering in queries
    • [nwaku] PostgreSQL enhance retention policy: bug: retention policy does not work when postgres db is over loaded
    • [nwaku] PostgreSQL enhance query performance chore: improve postgres query performance

Milestone - Direct Message Reliability §

• Review connection management strategy and back-off and fix long disconnection issues

  • achieved:
    • [chat] feat: flag to enable/disable missing message verification

• Tooling: filter and light push protocols

  • achieved:
    • [chat] added support for 2 lightpush peers to be used for better reliability feat: support for lightpush to use more than 1 peer
    • [chat] fix metadata protocol from disconnecting light-clients and only use clusterID and accept connections from lightclients fix: for light node do not check for matching shards but only clusterID
  • next:
    • [nwaku] Little enhancement for network connectivity, use bootstrap enr’s to connect to any kind of network and detect shardings
    • [nwaku] Auto select service peers and use them randomly for testing on both sides.
    • [nwaku] run lite-protocol-tester on shards.staging

• Reliability Protocol for Relay

  • achieved:
    • [nwaku] started local sqlite registry implementation feat: enhance reliability thanks to StoreV3
    • [chat] config to enable/disable storev3 confirmations for sent messages feat: flag to enable/disable sent message store query confirmations
  • next:
    • [nwaku] carry on with implementation feat: enhance reliability thanks to StoreV3

• Reliability Protocol for Resource-Restricted Clients

  • achieved:
    • [js-waku] improved renewal of peers for Filter feat(filter): peer/subscription renewal with recurring Filter pings
    • [js-waku] check for missing messages from Filter subscriptions feat: validate messages for individual filter nodes & perform renewals
    • [js-waku] peer renewal and keep alive improvements feat: fix peer renewal, change Filter keep alive
  • next:
    • [nwaku] Implementing: feat: Enhance lightpush protocol error handling

• Review MVDS usage and fail path

  • achieved:
    • [chat] status messages need for e2e reliability collected
    • [chat] fix_: delivered message should not send envelope sent signal

Milestone - End-to-end reliability protocol §

• End-to-end reliability protocol - PoC
  • achieved:
    • [research] implemented ACK handling, request missing messages, resend message
  • next:
    • [research] comprehensive tests for the poc

Milestone - Static Sharding - dedicated shards §

• Sharding peer management and discovery hardening
  • achieved:
    • [research] review of discv5 both spec and nim implementation
    • [research] compiled audit of discovery status quo and potential future strategies
    • [nwaku] Creating versions for DST team to test discv5 and message reliability
    • [nwaku] Proposing new solution to log received message info from nim-libp2p. Created custom version for DST team to use chore: creating branch to test Waku’s received messages
    • [nwaku] Started investigating and implementing improvements for connection management of bootstrap only nodes bug: Node accepts connections when configured for just discovery
    • [nwaku] Improved logging chore: improving logging under debugDiscv5 flag, chore: logging content topic of spam messages
    • [nwaku] Investigated at depth discv5’s implementation bug: discv5 not returning enough peers
  • next:
    • [research] review other discovery methods and state of implementation in nim
    • [research] review metrics collected and add more if needed
    • [research] implement identified discv5 improvement strategies
    • [nwaku] Get feedback on nim-libp2p’s logging solution
    • [nwaku] Refactor code now that named shading is removed, and deprecate pubsub-topic configuration
    • [nwaku] Continue discussing and analyzing discv5’s performance and possible improvements

Other Work §

Enhancements §

• achieved:
  • [chat] chore: rename shards.staging to status.staging
  • [chat] chore: rename shards.staging to status.staging
  • [chat] chore: bump go-libp2p
  • [chat] enable metadata protocol regardless of cluster-ID used
  • [js-waku] chore: enforce access modifiers
  • [js-waku] chore: bump @waku peer dependencies
  • [js-waku] improve network options handling chore: throw if more than one network config is passed
• next:
  • [nwaku] deploy release v0.31.0 Prepare release 0.31.0

Bugs §

• achieved:
  • [chat] investigation of slowness to retrieve messages in storenode message counter
  • [nwaku] Build nwaku image for Windows - feature: support Windows 11
• next:
  • [nwaku] Manage stale peers dynamically with better reliability fix: peer-exchange issue
  • [nwaku] bug: failed to retrieve peer info via peer exchange protocol