The Sia Foundation Roadmap

This Sia roadmap provides mid to high level insight into core Sia development. It will be updated once a quarter at minimum, and will show an outline of what we’re currently working on, why we’re working on it, and what we have in mind after that’s done.

We’re focusing on updated versions of all the core components of the Sia network - renting, hosting, and hodling. These updates will bring significant improvements to scaling, performance, and modular internal architecture to meet the needs of individuals, enterprise users, and exchange partners. Furthermore, each component will be paired with a powerful new user interface, all with the overarching goal of making core Sia software more “legible” to end users. “Legible” is the antonym of “black-box:” in a legible system, the user can easily discern why any particular decision was made, and can override those decisions as they see fit. This goal follows naturally from the Foundation’s mission of empowering users.

Project Board

The new Sia renter, replacing the current siad renter module. Drawing on what we’ve learned from siad, skyd, and us, we designed renterd from the ground up to be modular and horizontally scalable. Although the average user likely won’t notice, a renterd deployment is actually a set of interconnected services. As such, it can distribute workloads in parallel across multiple machines, and can be configured to store its metadata in any SQL backend. This flexibilty makes renterd easier to integrate with other Sia ecosystem software, such as Sia Satellite and S5, and addresses the scalability barriers that have historically hampered enterprise solutions.

April 2024

June 2024

  • Add support for RHP4
  • Improve upload parallelism

The new Sia host, replacing the current siad host module. The host module has been chronically neglected for years, and suffers from poor upload performance, data integrity issues, and a general lack of user-friendliness. hostd is our greenfield reimagining of the Sia hosting experience, bringing a sorely-needed refresh to our host community. Aside from addressing performance bottlenecks, hostd also offers superior metrics and monitoring tools, which will allow users to make informed decisions about storage allocations, contract parameters, pricing, and quality of service.

April 2024

June 2024

  • Automate data integrity checks
  • Add support for RHP4

September 2024

  • Support Postgres as an alternative to SQLite
  • Support additional volume types to optimize performance for distributed file systems

The new Sia wallet, replacing the current siad wallet module. walletd aims to be the go-to option for exchanges, miners, or the average hodler who needs a secure place to store their SC. Accordingly, it supports both hot and cold setups, including multi-sig schemes and hardware wallet integration. And like renterd and hostd, walletd comes packaged with a sleek, yet powerful UI, which can be securely accessed from any device.

Note: you don’t need walletd to be a renter or a host; renterd and hostd include their own built-in hot wallets.

January 2024

  • Support SQLite - Done

May 2024

  • Add “Indexer” mode enabling easier integration for exchanges - In Progress
  • Beta Release - In Progress

The new Sia explorer, replacing and going far beyond the current siad explorer capabilities. explored will serve as both a standalone blockchain explorer with a web interface, and as a library providing powerful indexing and searching capabilities to third-party explorers (e.g. SiaStats) and “lite-client” systems like narwal. explored will be developed and launched alongside the Utreexo overhaul, making it one of the world’s first Utreexo-native block explorers.

June 2024

  • Beta release

“Utreexo” is an umbrella term that refers to an upcoming radical overhaul of Sia’s consensus code. The primary change is the transition from a large, unwieldy database of UTXOs to a compact cryptographic accumulator, bringing myriad benefits to performance, scalability, and functionality. This is a big deal: it requires changing the block and transaction formats, which have been untouched since Sia’s mainnet launch back in 2014. As such, we are taking this opportunity to clean up a few other warts in Sia’s consensus code, giving the project a solid foundation for many years to come. For a more technical dive on how Utreexo works, check out Luke’s blog post here and MIT Bitcoin talk here.

May 2024

  • Upgrade Zen testnet

RHP4, short for renter-host protocol 4, is the next iteration of the protocol hosts and renters use to communicate. Our goal for RHP4 is to increase the parallelism of data uploads, reduce protocol overhead, and improve download performance. A new protocol is required to enable some exciting new features to be available after the Utreexo hardfork, such as early contract termination, capacity reservation, and contract renewal fund rollover. RHP4 will also enable storage consumers to download and upload data directly in the browser without installing additional software.

February 2024

  • Finalize spec
  • Implement RHP4 client in coreutils

April 2024

  • Test RHP4 on Anagami

May 2024

  • Release experimental SiaJS SDK
  • Test RHP4 on Zen

fsd is our experimental implementation of an IPFS node with specific optimizations for storing and retrieving data from the Sia network. We plan to open source it in the near future once it has stabilized more and we’ve verified that pinned content is discoverable and accessible by other IPFS nodes.

June 2024

  • Release alpha
Document version date: Apr 22, 2024
Apr 24, 2024

Our jobs are kind of slow so I was looking at ways to speed them up. I couldn't really find any low hanging fruit but ended up extracting two steps (lint & analyze) I think should be run separately because they do not depend on the os or go-version. It saves us roughly a minute so nothing ground breaking.

I ended up removing the dev trigger for the CodeQL job. I'm fine reverting that change but I feel we run that a lot without any real upside (so far). I can't remember the time it failed and made me change my PR for the better, since it takes more than two minutes on every single run maybe it's fine to only run it when we merge dev.

Apr 24, 2024

Caught these errors on renterd shutdown, fixed by adding a context.DeadlineExceeded case in abortRetry.

2024-04-23T10:49:08Z INFO  bus  successfully saved 392 accounts
2024-04-23T10:49:42Z ERROR SQL  trace {"error": "context deadline exceeded", "elapsed": "34.701151966s", "rows": 0, "sql": "\nDELETE\nFROM slabs\nWHERE NOT EXISTS (SELECT 1 FROM slices WHERE slices.db_slab_id = slabs.id)\nAND slabs.db_buffered_slab_id IS NULL\n"}
2024-04-23T10:49:42Z WARN  sql  transaction attempt 1/7 failed, retry in 200ms,  err: context deadline exceeded
2024-04-23T10:49:42Z WARN  sql  transaction attempt 2/7 failed, retry in 500ms,  err: context deadline exceeded
2024-04-23T10:49:43Z WARN  sql  transaction attempt 3/7 failed, retry in 1s,  err: context deadline exceeded
2024-04-23T10:49:44Z WARN  sql  transaction attempt 4/7 failed, retry in 3s,  err: context deadline exceeded
2024-04-23T10:49:47Z WARN  sql  transaction attempt 5/7 failed, retry in 10s,  err: context deadline exceeded
2024-04-23T10:49:57Z WARN  sql  transaction attempt 6/7 failed, retry in 10s,  err: context deadline exceeded
Apr 24, 2024
Apr 24, 2024
Apr 24, 2024

Not doing so caused slight differences in the returned value. e.g. it seemed like the number of usable hosts can improve by manually tweaking settings but no recommendation was returned.

Deployed this to Arequipa for testing @alexfreska If you set the target to 300 it should give you a recommendation now.

--- Current contracts ---
Contracts: 266
--- Target contracts ---
Contracts.Amount: 300
--- Before ---
MaxDownloadPrice: 1 KS
{
  "hosts": 722,
  "usable": 273,
  "unusable": {
    "blocked": 0,
    "gouging": {
      "contract": 12,
      "download": 109,
      "gouging": 131,
      "pruning": 7,
      "upload": 24
    },
    "notAcceptingContracts": 83,
    "notScanned": 86
  },
  "recommendation": {
    "gougingSettings": {
      "maxRPCPrice": "10000000000000000000",
      "maxContractPrice": "5000000000000000000000000",
      "maxDownloadPrice": "1155000000000000000000000000",
      "maxUploadPrice": "333333333000000000000000000",
      "maxStoragePrice": "154166666667",
      "hostBlockHeightLeeway": 12,
      "minPriceTableValidity": 600000000000,
      "minAccountExpiry": 86400000000000,
      "minMaxEphemeralAccountBalance": "1000000000000000000000000",
      "migrationSurchargeMultiplier": 10
    }
  }
}
--- After ---
MaxDownloadPrice: 1.03 KS
{
  "hosts": 722,
  "usable": 274,
  "unusable": {
    "blocked": 0,
    "gouging": {
      "contract": 12,
      "download": 107,
      "gouging": 131,
      "pruning": 7,
      "upload": 24
    },
    "notAcceptingContracts": 83,
    "notScanned": 86
  },
  "recommendation": {
    "gougingSettings": {
      "maxRPCPrice": "10000000000000000000",
      "maxContractPrice": "5000000000000000000000000",
      "maxDownloadPrice": "1135575000000000000000000000",
      "maxUploadPrice": "333333333000000000000000000",
      "maxStoragePrice": "154166666667",
      "hostBlockHeightLeeway": 12,
      "minPriceTableValidity": 600000000000,
      "minAccountExpiry": 86400000000000,
      "minMaxEphemeralAccountBalance": "1000000000000000000000000",
      "migrationSurchargeMultiplier": 10
    }
  }
}
Apr 23, 2024
  • Fixed an issue where file health slab keys were showing up as repeated and incorrect.
Apr 23, 2024
Apr 23, 2024

This removes the random data fill of volumes, enabling Linux and macOS to use sparse files again. However, this does have the side effect of "breaking" size detection when initializing and growing volumes, with no easy solution.I think that's acceptable to reduce hardware wear and tear and increase volume initialization performance.

Another option is to throw this behind an opt-in config flag for advanced users. That would prevent beginners from over-initializing their volumes while allowing advanced users to take advantage of filesystem optimizations.

Apr 23, 2024

Add chain index type to easily revert all entities introduced at a certain index by making use of CASCADE deletes

View full activity feed →