We have our monitoring solution in place. We are now in control of our cardano-node services, that are running as a background process.
The most important reason to get this up and running quickly, is the expiry of the KES period. KES is a key generation algorithm that is used in cryptography and network security. In the case of the Cardano Node, it is represented by “hot keys” a signing key and operational certificate that are applied to the stakingpools. The KES period is the time frame in which the signing key and operation certificate are valid, in other words, they expire. As a pool operator, you want your stakepool “hot keys” updated before the KES period expires. If you are too late with updating these files on your stakepool, you won’t be able to forge(mint) blocks with your stakepool nodes. Which will be bad for the performance of your pool and for the delegators in it.
We are making use of alerts, to detect the KES period expiration, and notify ourselves through Telegram in such events. The notifications will repeat every 30 min before the KES period expires so they can’t be missed.
The dashboard above, shows us the most relevant information:
Chain status; current epoch, slots in epoch and the current height
Server resource utilization for all our nodes over a period of time
KES period and expiry period of our “hot keys” (this is where we put an alert)
Numbers of blocks we forged as stakepool over a period of time
Grafana, the monitoring tool we use, to setup alerts that sends them out to channels. We are using the Telegram channel to stay in touch with the Cardano community. It’s an obvious choice to get alerts through a channel already in use.
Above is an example scenario, where we defined an alert, in case the KES period gets less than 100. We received an alert about this in our Telegram ₳4PROFIT stakepool alerter group. In such case, we can generate a new set of “hot keys” in time, to restart the stakepool before the stakepool gets in a state where it is unable to forge blocks.
Of course there are other alternatives instead of using Grafana and Telegram for alerting. For example you could also use the ELK stack (a popular monitoring stack). For us using Grafana and Telegram is sufficient to do the job. Depending on your needs, you should choose the tools that work best for you.
The performance of our stake pool has been excellent so far and we are one of the top performing stake pools on the HTN (Haskell Test Net).
In the last 2 days, we were able to forge 110 blocks, on average ~2.3 blocks per hour.
We are striving to keep up with this performance for our potential delegators on Mainnet. Which is expected to launch in August 2020.
The performance the last day was heavily influenced by the change of the d-factor by IOHK. Before the changes were made, there were returns for stake pool operators that were based on 90% of the blocks, now the return is based on only 15%. This change made by IOHK was influencing not only us, but the rewards for all stake pool operators as well. Only 30 blocks of the 110 blocks were forged after that change, other top performing pool operators reported similar results.
Tomorrow there will be a change to Cardano-node 1.14.0. This version comes with a new genesis and this means the pool operator have to re-register their pools on basis of new keys and certificates. This is a first real test in anticipation of the switch to Mainnet for the pool operators.
There is a new tag of the Cardano node out since 13/06/2020, that contains many new improvements. You can read about this on the Cardano node github page.
This involves work to be done for the pool operator, building a new cardano-node and cardano-cli (command line interface), via the Haskell compiler on the source code from the repository. More on this in part II of this blog.
One of the exciting improvements is that we now can lift the rewards we earn on our stake to the stake pool, either as stakepool owner or as delegator. This allows us to compound our stake or move the rewards to another address.
How does this work?
You can create a payment and staking address with the cardano-cli (command line interface), this is a tool by which a pool operator or delegator can transact with the Cardano blockchain.
Once we have a payment and staking address generated, we are able to register the staking address on the blockchain and check the status of that registration.
With the staking address being registered on the blockchain. We are able to delegate the staking address to a stake pool of choice, to earn rewards on our stake. The stake is our owned tADA tokens that are stored on our payment address. The rewards are earned by pool operators that operate and maintain a Cardano node, that mint blocks. Once they are assigned to their pool, based on a lottery system, minting a block, is the validation of the assigned block that contains one or more transactions. Validation is done on basis of cryptography by the pool operators node. This ensures the transactions in the assigned block, are not altered by a malicious party. Once the cryptographic verification is done and correct, the node publishes the results back to the other nodes on the network. This checks the outcome to agree or disagree with it. If the majority agrees with the outcome, the pool operator receives a reward and is successfully minting a block. The rewards that the pool operator receives for successfully minting a block are coming from a fund that is setup by the Cardano foundation. Which also collects all the costs that are involved with issuing a transaction on the network, known as fee or network costs. The rewards the pool operator receives for successfully minting blocks, are being shared with delegators to the pool, after the deduction of operational costs. The delegator receives the rewards automatically on their staking address, based on their weight of the total stake delegated to the pool.
There are several factors involved in this lottery system, to get a block assigned as stake pool. We will try to explain below based on the rewards formula that comes from the delegation design specification document. This can be found on page 38 in the document.
What we understand from this formula is that, rewards are based on the factor σ. In which σ is the total pool stake, i.e. the total delegated stake, plus pool owners pledge flesh in the game, or long term locked investment. The left part of the formula is independent of the pledge and dependent on the saturation of the pool, in case of a high saturation a=0.1. If all blocks are minted, the rewards will be ~91% of the maximum rewards, the pool can receive from the total epoch reward R. The other ~9% of the rewards is dependent on the second part of the formula, is based on the amount of pledge the owner locked up in the pool. If there is no pledge there won’t be any additional rewards from the other ~9% and if the pool is fully pledged it will receive 100% of that ~9%. We currently don’t know what a fully pledged pool means, but this is something we have to find out once the mainnet is going live.
Saturation is a term used to indicate that a particular stake pool has more stake delegated to it than is ideal for the network. Once a pool reaches the point of saturation it will offer diminishing rewards. The saturation mechanism was designed to prevent centralization by encouraging delegators to delegate to different stake pools, and operators to set up alternative pools, so they can continue earning maximum rewards. Therefore, exists to preserve the interests of both ada holders delegating their stake and stake pool operators.
The goal is to avoid any single pool becoming too large – thereby disincentivising delegation to other pools – and receiving a disproportionate amount of the rewards. The health of the network is partly determined by having a high number of active stake pools with a balanced amount of stakes delegated to them. The more numerous and geographically diverse the network’s pools are, the better.
In part II of this blog we will explain more about the technical details, of creating a payment and stake address and why it is important to get into an early stage familiar with this process.
Today we had one of those disastrous days!! We managed to get our website wiped, rushing to get some things finished. Since the backup was not in place yet we lost our content.
We’ve learned our lesson!! Creating a snapshot after this post and working tomorrow on re-posting the lost valuable content for our subscribers. Got to keep them up to date on the progress of our pool.
Today ADA4PROFIT became member of the Cardano Topology Alliance. An initiative by Marcel from TITAN staking.
A big thank you to Marcel for this initiative and for inviting ADA4PROFIT to become one of the members to provide reliable relays, we are honored. ADA4PROFIT is supporting your great initiative by increasing our global presence of relay nodes. Assuring we are connected properly ourselves and offering this to other pool operators.
In the moment of writing this post, ADA4PROFIT is providing 3 relay nodes. Our goal is to have a few more nodes up and running soon, thus creating a group to interconnect reliable relays on the Cardano Network.
Below you will find a brief description of what the topology alliance is trying to solve.
Currently there is no transparency, regarding whether our relays are listed in any other topology file. If listed, whether this external relay was restarted using that single topology file, each pool operator can only hope that others list his/her relays.
This bears the risk of not being properly connected to get our blocks on chain. Especially new pool operators struggle to be listed on established topology lists.
To increase the chance of being properly connected to the network, members of this group promise to utilize one or more of their relays, to list all members of this alliance.
Effect: If any one member is listed on a well connected relay, each of the members of this group are also connected to the network.
To increase the chance of being well connected, each member should enlist on the IOHK peer list, and try to ask other pool operators directly to include their relays in their lists as fixed/static entries.
On this day, 06/14/2020, we have our first relay node for the Cardano blockchain live in the USA.
What is the idea behind having a relay node in the USA?
Better global coverage for the ability to mint blocks.
Achieve decentralization to secure the network.
Reduction of a single point of failure, running relay nodes from one location on one data center.
Installing and making the new relay node operational, took us less than half an hour. Updating the topology, restarting the pool and other relay nodes was the most time consuming, but all went smoothly.
Preparation of the VPS machine took about an hour of, installation and configuration of a user, SSH, firewall, chrony, docker, docker-compose, system tuning and security hardening. This is something we can further optimize in the near future, once we get our infrastructure fully deployed by using Ansible. This could save us valuable time if we have to do this task over and over again.
With all being said, we are very happy with the result!