2024-07-29 Waku Weekly

Milestone - Store Service Upgrade §

• Store v3-beta - Message Hashes

  • achieved:
    • [research] completed first draft of storev3 benchmark test plan
  • next:
    • [research] implementing test tools, finalising test plan
  • blockers:
    • [research] not clear on HW setup on which benchmarking should be done

• Store v3 - store synchronisation

  • achieved:
    • [research] last online timestamp periodically saved to new sqlite db, store client query for last messages, tests
  • next:
    • [research] reviews, hand-off to nwaku team for merging

• DOS protection for req-res protocols and metrics

  • achieved:
    • [nwaku] Node Bandwidth Management Features
    • [nwaku] Enforce service specific rate limits
    • [nwaku] Separate add distinction between gross/net inbound traffic of shards. chore: Distinction between gross/net trafic in bandwidth per shard metric
  • next:
    • [nwaku] dogfooding

• PostgreSQL Maintenance

  • achieved:
    • [nwaku] Improved Sonda’s Grafana dashboard chore: show relative metrics instead of absolute in Sonda
    • [nwaku] Improved Sonda’s logging chore: including UTC time in Sonda logs
    • [nwaku] Analysed that messages_lookup is a good approach chore: improve postgres query performance
  • next:
    • [nwaku] Implement messages_lookup to enhance hashes-only queries chore: improve postgres query performance

Milestone - Direct Message Reliability §

• Review connection management strategy and back-off and fix long disconnection issues

  • achieved:
    • [chat] Rate limit message publishing
    • [chat] Disconnect all peers if ping to randomly choosen peers fails twice, chore: disconnect on subsequent ping failures
    • [chat] improve lightclient connectivity by publishing their shard info in metadata fix: lightclient enr shards and fix: check for lightclient only if req doesn’t contain shards
    • [chat] record connection failures for req/resp protocols

• Tooling: filter and light push protocols

  • achieved:
    • [chat] Lightpush and Filter bandwidth metrics, feat: store filter and lightpush stats
    • [nwaku] Little enhancement for network connectivity, use bootstrap enr’s to connect to any kind of network and detect shardings
    • [nwaku] Auto-select service peers and use them randomly for testing on both sides.
    • [nwaku] Run lite-protocol-tester on shards.staging

• Telemetry: direct message reliability

  • achieved:
    • [chat] Fix concurrent RW on map in telemetry server: fix: concurrent rw on map
    • [chat] Improve storednode message counter dashboard: storenode-message-counter

• Reliability Protocol for Relay

  • next:
    • [nwaku] ongoing implementation in nwaku: feat: Enhance lightpush protocol error handling

• Reliability Protocol for Resource-Restricted Clients

  • achieved:
    • [chat] lightclient error handling
    • [js-waku] health state of nodes feat: node and protocols health
    • [js-waku] improve continuous peer discovery feat(peer-exchange): support continuous peer information updates
  • next:
    • [js-waku] complete review for health state of node and continuous peer discovery

• User apps for large scale dogfooding

  • achieved:
    • [js-waku] dogfooding app is up and running https://lab.waku.org/dogfooding/
  • next:
    • [js-waku] need to fix HTTP headers on the serving site, build basic dashboard

Milestone - End-to-end reliability protocol §

• End-to-end reliability protocol - PoC
  • achieved:
    • [research] added comprehensive tests for the e2e reliability POC
  • next:
    • [research] fixes arising from tests, some cleanup and prepare for dogfooding

Milestone - Static Sharding - dedicated shards §

• Sharding peer management and discovery hardening
  • achieved:
    • [research] adding metrics, manual testing
    • [nwaku] Creating versions for DST team to test discv5 and message reliability
    • [nwaku] improved nim-libp2p PR with custom logging observer chore: creating branch to test Waku’s received messages
    • [nwaku] Opened issue and investigated the phenomenon of some nodes getting stuck and missing messages in DST simulations bug: node getting stuck and missing messages
  • next:
    • [nwaku] Continue with the implementation of the bootstrap nodes connection limiting
    • [nwaku] Refactor code now that named shading is removed, and deprecate pubsub-topic configuration

Other Work §

Enhancements §

• achieved:
  • [nwaku] Revised nwaku nodes’ logging and moved added PRs in nwaku and nim-libp2p to downgrade particularly spammy logs chore: reduce loglevel to some too frequent or unnecessary logs
  • [nwaku] Added a CI job verifying that new code is properly linted chore: adding lint job to the CI
  • [nwaku] Started going over failed interop tests bug: nwaku <> js-waku interop tests failing
  • [nwaku] Release candidate v0.31.0 looks nice from Status-QA PoV Prepare release 0.31.0
  • [nwaku] Have the interop tests fixed and a working CI
  • [nwaku] Little enhancement on tooling, now single file build/test-run available right from make
  • [js-waku] better linting for classes chore: enforce access modifiers
  • [js-waku] fix peer exchange tests chore(peer-exchange): use an event listener to gauge if the service is mounted
  • [js-waku] remove a bit of tech debt chore: remove content_topic specific API
• next:
  • [nwaku] Perform release v0.31.0
  • [js-waku] Release

Bugs §

• achieved:
  • [nwaku] bug: RLN_RELAY_MSG_LIMIT handling
  • [nwaku] feature: support Windows 11 made little progress on this it’s ongoing.
  • [nwaku] bug: peer exchange returns nodes that no longer exist
• next:
  • [nwaku] bug: failed to retrieve peer info via peer exchange protocol
  • [nwaku] building test-peer exchange app on top of the network to analyze peer-exchange behaviour again cache refresh rate.