2023-aug

Codex Monthly Report - August 2023 §

Client §

Milestone: Merkelizing block data §

• Initial design writeup completed: https://github.com/codex-storage/codex-research/blob/master/design/metadata-overhead.md
• Initial Merkle Tree implementation: https://github.com/codex-storage/nim-codex/pull/504
• Work on persisting/serializing Merkle Tree
• Storing and retrieving data using merkle trees: https://github.com/codex-storage/nim-codex/pull/541
• Coders for merkle trees: https://github.com/codex-storage/nim-codex/pull/519
• Refine merkle tree construction: https://github.com/codex-storage/nim-codex/pull/516

Milestone: Block discovery and retrieval §

• Initial work breakdown and milestones: https://docs.google.com/document/d/1hnYWLvFDgqIYN8Vf9Nf5MZw04L2Lxc9VxaCXmp9Jb3Y/edit
• Initial analysis of block discovery: https://rpubs.com/giuliano_mega/1067876
• Initial block discovery simulator: https://gmega.shinyapps.io/block-discovery-sim/
• Continued analysis and review of peer sampling papers
• Started work on simulations based on the analysis

Milestone: Distributed Client Testing §

• Extensive work on log collection/analysis and monitoring
• Testing and debugging Codex in continuous testing environment
• Major effort to stabilize the Codex client through continuous automated testing

Milestone: Async Disc Access & Threading support §

• Tests on sharing thread data with refc
• Addressed errorVariable and thread safety issues
• Prototype proxy IO threadpool in progress

Marketplace §

Milestone: L2 §

• Taiko L2 integration mostly completed, waiting on related fixes

Milestone: Reservations and slot management §

• Extensive work on slot reservation and queuing

Milestone: Marketplace Sales §

• Cleanup and refactoring of state machine
• Added support for loading node’s slots during Sale’s module start

Remote auditing §

Milestone: Implement Poseidon2 §

• First pass at an implementation by Balazs

Milestone: Refine proving system §

• Extensive thinking around storage proofs and proving systems

DAS (Data Availability Sampling) §

Milestone: DHT simulations §

• Implementing a DHT in Python for the DAS simulator
• Implemented logical error-rates and delays to interactions between DHT clients

Infra §

Milestone: Kubernetes Configuration and Management §

• Moved Dist-Tests cluster to OVH and defined naming conventions
• Configured Ingress Controller for Kibana/Grafana
• Created documentation for Kubernetes management
• Configured Dist/Continuous-Tests Pods logs shipping

Milestone: Continuous Testing and Labeling §

• Implemented and configured Dist-Tests labeling
• Set up logs shipping based on labels
• Improved Docker workflows and added ‘latest’ tag

Milestone: CI/CD and Synchronization §

• Set up synchronization by codex-storage
• Configured Codex Storage and Demo CI/CD environments

Milestone: Monitoring and Metrics §

• Installed Node exporter and Prometheus in Dist-Tests cluster
• Updated Grafana Dashboard
• Implemented automated metrics scraping

Notable Achievements §

• Significant progress in Merkelizing block data, improving the foundation for the proving system
• Advancements in block discovery and retrieval, including simulation work
• Major improvements in distributed client testing and stability
• Successful integration with Taiko L2, expanding the project’s capabilities
• Continued development of DHT simulations for Data Availability Sampling

Challenges and Next Steps §

• Continue refining the Merkle tree implementation and integration
• Further develop and optimize the block discovery and retrieval processes
• Enhance the continuous testing environment for improved client stability
• Complete the Taiko L2 integration and address any remaining issues
• Advance the DHT simulations and DAS implementation
• Continue improving infrastructure, monitoring, and CI/CD processes