2023-10-30 Waku weekly §

Waku Network Can Support 10K Users §

• Integration of static sharding in go-waku is continuing (see updates below).
• Testing of PostgreSQL enabled some performance improvement in the implementation that are being implemented.
• Internal instructions have been distributed to dogfood static sharding with the Waku team (Waku Discord private channel).
• risks:
  • Dependency on Vac/DST to conclude ~1k nodes simulations.
  • Implementation of static sharding in Status Communities and design decisions mostly driven by go-waku developer, with minimal input from Status dev (1, 2, 3). See status-go#4057 for remaining work. Mitigation by on-boarding Chat SDK lead on 6 Nov to drive effort.

Targeted dogfooding for Status Communities §

• achieved: unsuccesfully tried to avoid introducing a breaking change in status-go. We need to decide whether to go ahead and merge that PR
• blocker: discv5 filters out outdated ENR entries from DNS Discovery URL in shard fleet - https://github.com/waku-org/nwaku/issues/2162

Waku Network can Support 1 Million Users - 2023-11-30 §

• achieved:
  • See 10k milestone update for PostgreSQL status.
  • First version of the 10k-tool by DST is ready and is being tested with simulation running a small nim-libp2p/gossipsub binary.
• risks:
  • Dependency on Vac/DST to run 10k nodes simulations. Tracked under vac:dst:eng-10ktool.
  • Wakutorsis tool is being dropped, meaning new tooling needs to be developed for 10k nodes simulations. It is currently uncertain whether such tool can be developed.
  • Large scale simulations done by Vac/DST only covered nwaku relay. go-waku, status-go simulations are not planned short term (theoretical review of Status Communities messages is), nor are simulations including request-response protocols such as store and filter.

PostgreSQL in service node: Further optimisations §

[nwaku] PostgreSQL

• achieved:
  • Time processing enhancement when performing SELECT operations. There was an overhead caused by looping too many times over the returned rows, in order to convert the row types. By applying a “rowCallback” approach we can reduce by 30ms the time spent on the query under analysis.
• next:
  • The queries used in the comparison analysis still perform much better in SQLite (< ~5ms) than in Postgres (< ~15ms.) Therefore we need to push the investigation further to enhance that.

Waku Network Gen 0 - 2023-12-01 §

• achieved:
  • Further simulation done, with a continued focus on message propagation time and possible improvements.
  • Progress across all client son sharded peer management discovery
  • First PRs merged towards basic distributed store services
• risks:
  • Usage of RLN in js-waku and dependency on a (centralized?) Web3Provider remains unclear as one needs to know the merkle tree state (on chain) to generate proofs.
  • We are progressively moving a nwaku engineer to a solution engineer role we need to backfill the role.
  • js-waku team is juggling between dev ex and gen 0 with only 2 engineers (3rd one just joined) so delivery in this client is likely to lag behind other clients.

3.2: Basic DoS protection in production §

[nwaku] feat: add rln to waku simulator instance

• achieved: learnt about waku-simulator’s inner workings and got the background required to integrate RLN to it. Added service that generates traffic to the nodes via their REST APIs. Investigated and tested different ways of approaching the RLN integration.
• next: get RLN to work and add Grafana dashboards with RLN data

[js-waku-examples] feat: re-create rln-js

• achieved: addressed flaws in integration, completed rewriting;

[research] Tuning GossipSub’s D parameter in Waku

• achieved: nwaku simulations showing the impact in message propagation delay when reducing gossipsub’s D value. Main goal is to reduce bandwidth consumption in exchange of worsen propagation delay.
• next: asses if we want to move forward changing D.

[research] Message propagation times with waku-rln

• achieved: Final simulation results with 1000 nwaku nodes with rln enabled, with the goal of measuring message propagation delays under different conditions (amount of nodes and message size).
• next: NA

1.4: Sharded peer management and discovery §

[nwaku] feat: Service peer selection on specific shards

• achieved: REST APIs discovery handlers PR merged

[nwaku] feat: Peer management with shard as a dimension

• achieved: Waku Metadata shard subscriptions, Sharded relay peer management, draft sharded peer store pruning
• next: finalize sharded peer store pruning & run simulations

[go-waku] feat: Deprecate Named Sharding and Update Lightpush Client API

• achieved: Create PR for review for removing of Named Pubsubtopic.

[go-waku] feat: Service peer selection on specific shards

• achieved: draft PR #834 opened for on-demand peer discovery
• next: use on-demand peer discovery for service and relay peer selection

2.3: Basic distributed Store services §

[nwaku] feat: add new message_hash column to the archive protocol

• achieved: On SQLite’s schema transition (i.e. this PR) to messageHash feature complete PR posted (awaiting reviews), Gained insight into the connection and interplay between the store and archive components, and how they may be leveraged into making a sync protocol. Small stuff - bug fix on the jsWaku which was this PR dependent (that too was time-consuming since my first time interacting with JS code of waku), PR on vacuum on time-based retention policy, thought through the nitty gritty details of node based roles and incentives.
• next:
  • The sync protocol formulation totally based on the messages sync without any external factors into POV
  • Review PostgreSQL PRs by Ivan to gain more knowledge on the storage/archive feature.

2.1: Production testing of existing protocols §

[nwaku] PostgreSQL

• achieved:
  • Time processing enhancement when performing SELECT operations. There was an overhead caused by looping too many times over the returned rows, in order to convert the row types. By applying a “rowCallback” approach we can reduce by 30ms the time spent on the query under analysis.
• next:
  • The queries used in the comparison analysis still perform much better in SQLite (< ~5ms) than in Postgres (< ~15ms.) Therefore we need to push the investigation further to enhance that.

[waku-rust-bindings] feat: filterv2 support

• achieved: fix issues found during testing
• next: publish new version

Quality Assurance processes are in place - 2024-03-31 §

Support Many Platforms - 2024-04-30 §

Presentation Readiness §

[js-waku] feat: better support for development

• achieved: experimented with Next.js app;
• next: looking for ways to mitigate errors in console or catch others;

Ship RLN as part of non-native SDKs §

[go-waku] refactor: add user_data to c-bindings

• achieved: fixed issues found during tests with waku-rust-bindings

REST API service node §

[go-waku] feat: admin REST API

• achieved: Implemented Admin REST API and updated the spec.

Ecosystem Development §

• achieved:

  • new tictactoe example with @waku/react
  • Progress on Devconnect planning
  • Organising dev ex calls
  • Shipping resources for hackathon
  • Reviewed Graphcast proposal for using relay

• next:

  • ipfs/waku example for file transfer
  • Waku tutorial videos
  • @waku/react hacker pain-point feedback report
  • Metrics dashboard
  • ETHLisbon