Releases: Consensys/teku
22.9.1
This is a recommended update containing improvements to post-merge performance issues on MainNet.
There are no changes from the previous 22.9.1-RC1 so users who are running the RC do not need to upgrade.
Downloads
- Available as
22.9.1
on Dockerhub - Download the binary distribution:
Breaking Changes
- The
--initial-state
and--eth1-deposit-contract-address
options has been removed from thevalidator-client
subcommand. They have been ignored for some time but are now completely removed. - LevelDB configuration has been changed to increase the maximum number of files it may open at a time. Systems with low open file handle limits may encounter issues.
We recommend at least 2048 file handles or preferably unlimited. Theulimit
command can be used to check and configure file handle limits.
Performance Improvements
This release significantly improves Teku's performance post-merge, but we will continue to investigate further improvements.
Please note that if the execution client is slow to respond, particularly when importing new blocks, that will increase the rate of incorrect head votes and increase the likelihood of attestations not being included.
To see more information on where time is being spent when importing blocks, users can enable late block alerts with the --Xmetrics-block-timing-tracking-enabled
option. This option will log a warning when a block import completes more than 4 seconds into the slot (after attestations will be created) with a break down of where time was spent. For example:
Late Block Import *** Block: e83a322a714b1b364ab59b2cb8666076fa015a8a5b4f1a8b9829306fc249f787 (4718961) proposer 425317 arrival 681ms, pre-state_retrieved +11ms, processed +306ms, execution_payload_result_received +3173ms, begin_importing +0ms, transaction_prepared +0ms, transaction_committed +1ms, completed +15ms
The labels used for timings are somewhat Teku developer focussed, but some key steps are:
arrival
- the time since the start of the slot that the block was first received. Large values here usually indicate that the block was produced late.state_retrieved
- the time it took to prepare the pre-state to apply the block to.processed
- the time it took Teku to process the block.execution_payload_result_received
- the time Teku spent waiting for the execution client to finish importing the block.
The remaining entries are Teku processing time to complete importing the block, updating fork choice records and storing the data to disk.
So in the example above the block was late to import because of a slow response from the execution client.
Additions and Improvements
- Enables asynchronous database updates by default. This ensures slow disk access or LevelDB compactions don't cause delays in the beacon node
- Increased leveldb open files to 1000 files by default. If teku fails to start, update maximum open files to 2048 or unlimited
- Reduced number of blocks held in in-memory cache to reduce need for old-gen GC with the larger post-merge blocks
- Make Validator Client connect to a failover event stream (if failovers are configured) when the current Beacon Node is not synced
- Detect Lodestar clients in
libp2p_connected_peers_current
metrics - Reduce CPU and Memory consumption in shuffling, which will improve epoch transition performance
- Faster peer discovery on startup
Bug Fixes
- Fixed an issue where epoch cache priming sometimes happened synchronously, causing delays in time tick processing in the beacon node
- Fix
/eth/v1/validator/register_validator
responding with a 400 status code and a misleading error message in case of exceptions - Update snakeyaml dependency to resolve cve-2022-25857 which could result in excessive memory usage when parsing YAML content
- Fixed an issue where the range requested for deposit logs was not reduced when using only
--ee-endpoint
leading to persistent timeouts with execution clients
Upcoming Breaking Changes
- The
validator_beacon_node_published_attestation_total
,validator_beacon_node_published_aggregate_total
,
validator_beacon_node_send_sync_committee_messages_total
,validator_beacon_node_send_sync_committee_contributions_total
andvalidator_beacon_node_published_block_total
metrics have been deprecated in favour of the newvalidator_beacon_node_requests_total
metric.
The old metrics will be removed in a future release. An update to the Teku Dashboard that uses the new metric is available. - The
/eth/v1/debug/beacon/states/:state_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/debug/beacon/states/:state_id
- The
/eth/v1/beacon/blocks/:block_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/beacon/blocks/:block_id
- The
/eth/v1/validator/blocks/:slot
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/validator/blocks/:slot
- The
/eth/v1/debug/beacon/heads
endpoint has been deprecated in favor of the v2 Bellatrix endpoint/eth/v2/debug/beacon/heads
22.9.1-RC1
This is a release candidate containing improvements to post-merge performance issues on MainNet that has received less testing than our normal releases.
MainNet users should only upgrade if they are confident in their ability to rollback if required. We encourage all testnet users to upgrade to assist with testing.
Downloads
- Available as
22.9.1-RC1
on Dockerhub - Download the binary distribution:
Breaking Changes
- The
--initial-state
and--eth1-deposit-contract-address
options has been removed from thevalidator-client
subcommand. They have been ignored for some time but are now completely removed. - LevelDB configuration has been changed to increase the maximum number of files it may open at a time. Systems with low open file handle limits may encounter issues.
We recommend at least 2048 file handles or preferably unlimited. Theulimit
command can be used to check and configure file handle limits.
Additions and Improvements
- Enables asynchronous database updates by default. This ensures slow disk access or LevelDB compactions don't cause delays in the beacon node. Can be disabled if needed via
--Xdata-storage-async-enabled=false
- Increased leveldb open files to 1000 files by default. If teku fails to start, update maximum open files to 2048 or unlimited
- Reduced number of blocks held in in-memory cache to reduce need for old-gen GC with the larger post-merge blocks
- Make Validator Client connect to a failover event stream (if failovers are configured) when the current Beacon Node is not synced
- Detect Lodestar clients in
libp2p_connected_peers_current
metrics - Reduce CPU and Memory consumption in shuffling, which will improve epoch transition performance
- Faster peer discovery on startup
Bug Fixes
- Fixed an issue where epoch cache priming sometimes happened synchronously, causing delays in time tick processing in the beacon node
- Fix
/eth/v1/validator/register_validator
responding with a 400 status code and a misleading error message in case of exceptions - Fix a
NullPointerException
for the gas limit when a proposer config is used and builder is enabled - Update snakeyaml dependency to resolve cve-2022-25857 which could result in excessive memory usage when parsing YAML content
- Fixed an issue where the range requested for deposit logs was not reduced when using only
--ee-endpoint
leading to persistent timeouts with execution clients
Upcoming Breaking Changes
- The
validator_beacon_node_published_attestation_total
,validator_beacon_node_published_aggregate_total
,
validator_beacon_node_send_sync_committee_messages_total
,validator_beacon_node_send_sync_committee_contributions_total
andvalidator_beacon_node_published_block_total
metrics have been deprecated in favour of the newvalidator_beacon_node_requests_total
metric.
The old metrics will be removed in a future release. An update to the Teku Dashboard that uses the new metric is available. - The
/eth/v1/debug/beacon/states/:state_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/debug/beacon/states/:state_id
- The
/eth/v1/beacon/blocks/:block_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/beacon/blocks/:block_id
- The
/eth/v1/validator/blocks/:slot
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/validator/blocks/:slot
- The
/eth/v1/debug/beacon/heads
endpoint has been deprecated in favor of the v2 Bellatrix endpoint/eth/v2/debug/beacon/heads
22.9.0
This is a required update. All users should upgrade as soon as possible.
IMPORTANT: Mainnet nodes must be made "merge ready"
All Mainnet nodes will need to prepare for the merge, including:
- Connecting to a local execution client's engine API with the
--ee-endpoint
option. - Validators will need to set a default fee recipient with
--validators-proposer-default-fee-recipient
.
Both Teku and your execution client will need to be updated to prepare for the merge. You should not receive any warning or error messages about the execution client being unavailable.
See the Teku guide on preparing for the merge for more information.
Downloads
- Available as
22.9.0
on Dockerhub - Download the binary distribution:
Bug Fixes
- Resolves an issue with public key validation.
Upcoming Breaking Changes
- The
validator_beacon_node_published_attestation_total
,validator_beacon_node_published_aggregate_total
,
validator_beacon_node_send_sync_committee_messages_total
,validator_beacon_node_send_sync_committee_contributions_total
andvalidator_beacon_node_published_block_total
metrics have been deprecated in favour of the newvalidator_beacon_node_requests_total
metric.
The old metrics will be removed in a future release. An update to the Teku Dashboard that uses the new metric is available. - The
/eth/v1/debug/beacon/states/:state_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/debug/beacon/states/:state_id
- The
/eth/v1/beacon/blocks/:block_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/beacon/blocks/:block_id
- The
/eth/v1/validator/blocks/:slot
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/validator/blocks/:slot
- The
/eth/v1/debug/beacon/heads
endpoint has been deprecated in favor of the v2 Bellatrix endpoint/eth/v2/debug/beacon/heads
22.8.2
This is a strongly recommended update that fixes a corner case which could lead to producing invalid blocks.
IMPORTANT: Mainnet nodes must be made "merge ready"
All Mainnet nodes will need to prepare for the merge, including:
- Connecting to a local execution client's engine API with the
--ee-endpoint
option. - Validators will need to set a default fee recipient with
--validators-proposer-default-fee-recipient
.
Both Teku and your execution client will need to be updated to prepare for the merge. You should not receive any warning or error messages about the execution client being unavailable.
See the Teku guide on preparing for the merge for more information.
Downloads
- Available as
22.8.2
on Dockerhub - Download the binary distribution:
Additions and Improvements
- Added Gas Limit APIs (GET/POST/DELETE)
- Skip finding the PoW block that first satisfies the minimum genesis time condition when the genesis state is already known. Fixes an incompatibility with Nethermind's backwards sync for historic blocks.
- Circuit breaker logic added when interacting with Builder endpoint
- VC using failover nodes for retrieving the network spec when
--network=auto
and multiple beacon nodes are configured - Teku now confirms the first successful connection to the execution client by logging
Execution Client is online
.
Bug Fixes
- Fixed a corner case that could lead to producing invalid blocks
- Fixed
io.libp2p.core.InternalErrorException: [peerHandler] not initialized yet
error from jvm-libp2p - Fixed VC incompatibility with Nimbus BN for the
/eth/v1/validator/beacon_committee_subscriptions
endpoint
Upcoming Breaking Changes
- The
validator_beacon_node_published_attestation_total
,validator_beacon_node_published_aggregate_total
,
validator_beacon_node_send_sync_committee_messages_total
,validator_beacon_node_send_sync_committee_contributions_total
andvalidator_beacon_node_published_block_total
metrics have been deprecated in favour of the newvalidator_beacon_node_requests_total
metric.
The old metrics will be removed in a future release. An update to the Teku Dashboard that uses the new metric is available. - The
/eth/v1/debug/beacon/states/:state_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/debug/beacon/states/:state_id
- The
/eth/v1/beacon/blocks/:block_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/beacon/blocks/:block_id
- The
/eth/v1/validator/blocks/:slot
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/validator/blocks/:slot
- The
/eth/v1/debug/beacon/heads
endpoint has been deprecated in favor of the v2 Bellatrix endpoint/eth/v2/debug/beacon/heads
22.8.1
This is a required update that include the Bellatrix network upgrade and merge transition configuration. All Mainnet users must upgrade before September 6.
IMPORTANT: Mainnet nodes must be made "merge ready"
All Mainnet nodes will need to prepare for the merge, including:
- Connecting to a local execution client's engine API with the
--ee-endpoint
option. - Validators will need to set a default fee recipient with
--validators-proposer-default-fee-recipient
.
Both Teku and your execution client will need to be updated to prepare for the merge. Until the execution client is updated Teku will regularly report an error like:
Merge *** Transition Configuration error: local TerminalTotalDifficulty and TerminalBlockHash not matching remote Execution Client values
local configuration: TransitionConfiguration{terminalTotalDifficulty=0x000000000000000000000000000000000000000000000c70d808a128d7380000, terminalBlockHash=0x0000000000000000000000000000000000000000000000000000000000000000, terminalBlockNumber=0}
remote configuration: TransitionConfiguration{terminalTotalDifficulty=0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffbff, terminalBlockHash=0x0000000000000000000000000000000000000000000000000000000000000000, terminalBlockNumber=0}
or like:
Execution Client is offline java.io.IOException: -32000: invalid ttd: execution <nil> consensus 0xc70d808a128d7380000 (See log file for full stack trace)
Once the execution client has been updated, you should not continue receiving these errors or other warnings about the execution client being unavailable.
Please continue to monitor for further Teku updates prior to the merge and be ready to upgrade if necessary.
See the Teku guide on preparing for the merge for more information.
Downloads
- Available as
22.8.1
on Dockerhub - Download the binary distribution:
Additions and Improvements
- Sets the Bellatrix fork epoch and TTD for The Merge on MainNet
- Automatically add
/eth/v2/debug/beacon/states/finalized
to the initial state file url (--initial-state
CLI option) when the provided url doesn't work and doesn't contain any path.
This simplifies using the standard REST API to retrieve the initial state as just the base URL can be specified (e.g.--initial-state https://<credentials@eth2-beacon-mainnet.infura.io
) - Ability to configure multiple beacon nodes for a single validator client using
--beacon-node-api-endpoints
CLI option - Primed cache for new justified checkpoints to reduce time required to run fork choice immediately after justification
- Remain in optimistic mode when there are no viable branches in the block tree because blocks from every branch were marked INVALID during optimistic sync (ethereum/consensus-specs#2955)
Bug Fixes
- Fixed
NullPointerException
when checking for the terminal PoW block while the EL was syncing - Changed log level of
Ignoring calculated payload
message to debug level. - Fixed repeated timeout exceptions when requesting deposit logs from the
--ee-endpoint
Upcoming Breaking Changes
- The
validator_beacon_node_published_attestation_total
,validator_beacon_node_published_aggregate_total
,
validator_beacon_node_send_sync_committee_messages_total
,validator_beacon_node_send_sync_committee_contributions_total
andvalidator_beacon_node_published_block_total
metrics have been deprecated in favour of the newvalidator_beacon_node_requests_total
metric.
The old metrics will be removed in a future release. An update to the Teku Dashboard that uses the new metric is available. - The
/eth/v1/debug/beacon/states/:state_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/debug/beacon/states/:state_id
- The
/eth/v1/beacon/blocks/:block_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/beacon/blocks/:block_id
- The
/eth/v1/validator/blocks/:slot
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/validator/blocks/:slot
- The
/eth/v1/debug/beacon/heads
endpoint has been deprecated in favor of the v2 Bellatrix endpoint/eth/v2/debug/beacon/heads
22.8.0
This is an optional update containing bug fixes, optimizations and minor new features.
Downloads
- Available as
22.8.0
on Dockerhub - Download the binary distribution:
IMPORTANT: Prater nodes must be made "merge ready"
See the Teku guide on preparing for the merge for more information.
Additions and Improvements
- Support for the external builder API (e.g. mev-boost)
- Enables fork choice before block proposals by default on MainNet (previously on by default on testnets only)
- Optimisations in jvm-libp2p to reduce CPU usage
- Updated Sepolia bootnodes
- Enabled progressive balance tracking optimisation on MainNet
- Replaced separate metrics for each beacon node request with one metric
beacon_node_requests_total
. Some old metrics were kept for backwards compatibility, but will be removed in future versions.
Bug Fixes
--ee-endpoint
option was not used to retrieve deposits for networks where Bellatrix was not yet scheduled- Fix
latestValidHash
with invalid Execution Payload in response from execution engine didn't trigger appropriate ForkChoice changes - Remove incorrect error about potentially finalizing an invalid execution payload when importing a block with an invalid payload
Upcoming Breaking Changes
- The
validator_beacon_node_published_attestation_total
,validator_beacon_node_published_aggregate_total
,
validator_beacon_node_send_sync_committee_messages_total
,validator_beacon_node_send_sync_committee_contributions_total
andvalidator_beacon_node_published_block_total
metrics have been deprecated in favour of the newvalidator_beacon_node_requests_total
metric.
The old metrics will be removed in a future release. An update to the Teku Dashboard that uses the new metric is available. - The
/eth/v1/debug/beacon/states/:state_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/debug/beacon/states/:state_id
- The
/eth/v1/beacon/blocks/:block_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/beacon/blocks/:block_id
- The
/eth/v1/validator/blocks/:slot
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/validator/blocks/:slot
- The
/eth/v1/debug/beacon/heads
endpoint has been deprecated in favor of the v2 Bellatrix endpoint/eth/v2/debug/beacon/heads
22.7.0
This is a required update for the Prater testnet and recommended for other users.
IMPORTANT: Prater nodes must be made "merge ready"
All Prater nodes will need to prepare for the merge, including:
- Connecting to a local execution client's engine API with the
--ee-endpoint
option. - Validators will need to set a default fee recipient with
--validators-proposer-default-fee-recipient
.
See the Teku guide on preparing for the merge for more information.
Downloads
- Available as
22.7.0
on Dockerhub - Download the binary distribution:
Additions and Improvements
- Added Bellatrix fork epoch and terminal total difficulty to schedule the Prater/Görli merge
- Support
--network=goerli
as an alias for--network=prater
- Added a panda banner to celebrate the merge completing
- Log a warning instead of a verbose error if node is syncing while performing sync committee duties
- Distributions created from the same git commit and docker image will be identical
- Optimised storage of latest vote information by batching updates
- Ensured dependencies are up to date
- Validator Registration signature integration with external signer
- Teku-specific Beacon Node API on
/teku/v1/beacon/deposit_snapshot
providing finalized Deposit Tree Snapshot according to the draft EIP-4881
Bug Fixes
- Fix not rendering emoticons correctly in graffiti when running in a Docker container
- Fix resource leak from closed SSE connections
Upcoming Breaking Changes
- The
/eth/v1/debug/beacon/states/:state_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/debug/beacon/states/:state_id
- The
/eth/v1/beacon/blocks/:block_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/beacon/blocks/:block_id
- The
/eth/v1/validator/blocks/:slot
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/validator/blocks/:slot
- The
/eth/v1/debug/beacon/heads
endpoint has been deprecated in favor of the v2 Bellatrix endpoint/eth/v2/debug/beacon/heads
22.6.1
This is a required update for the Sepolia testnet and recommended for other users.
Downloads
- Available as
22.6.1
on Dockerhub - Download the binary distribution:
Additions and Improvements
- Updated terminal total difficulty (TTD) configuration for Sepolia testnet
- Further optimization of epoch transitions
- Added additional Sepolia bootnode
- Optimised attestation gossip validation by using the current head state to validate more attestations
- Avoid revalidating signatures for attestations re-added to the memory pool during a re-org
Bug Fixes
- Fixed the beacon-rest-api opening a listen port on too many interfaces
- Fixed issue where blocks received via gossip may not be processed if an ancestor's execution payload returned an error from the execution engine.
- Increased the default queue size limit for validator tasks to better support running very large numbers of validators.
- Fixed issue where execution engine connections using web sockets may fail during the merge transition
Upcoming Breaking Changes
- The
/eth/v1/debug/beacon/states/:state_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/debug/beacon/states/:state_id
- The
/eth/v1/beacon/blocks/:block_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/beacon/blocks/:block_id
- The
/eth/v1/validator/blocks/:slot
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/validator/blocks/:slot
- The
/eth/v1/debug/beacon/heads
endpoint has been deprecated in favor of the v2 Bellatrix endpoint/eth/v2/debug/beacon/heads
22.6.0
This is a recommended upgrade with configuration updates and optimisations.
Downloads
- Available as
22.6.0
on Dockerhub - Download the binary distribution:
Breaking Changes
- The command-line option
--validators-performance-tracking-enabled
has been removed in favour of--validators-performance-tracking-mode
Additions and Improvements
- Added configuration for Sepolia testnet.
- Reduced memory requirements for storing the deposit merkle tree.
- Enable spec change to ignore weightings from attestations from equivocating validators.
- Fork choice before proposals is now enabled by default for testnets. It can be disabled with
--Xfork-choice-before-proposing-enabled=false
if required. - Updated the Ropsten network configuration to include the correct terminal difficulty.
- Updated to BLST 0.3.8
- Reduced CPU usage when processing epoch transition.
Bug Fixes
- Fixed issue where the REST API may return content as SSZ instead of JSON if the header
Accept: */*
was specified. - Fixed issue where sync committee aggregations were skipped, but reported failed because there were no signatures to aggregate.
- Fixed division by zero when estimating time to TTD during a period with no new PoW blocks.
- Fixed issue where validator client could receive a 500 rather than 503 when getting duties if the beacon node was syncing.
Upcoming Breaking Changes
- The
/eth/v1/debug/beacon/states/:state_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/debug/beacon/states/:state_id
- The
/eth/v1/beacon/blocks/:block_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/beacon/blocks/:block_id
- The
/eth/v1/validator/blocks/:slot
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/validator/blocks/:slot
- The
/eth/v1/debug/beacon/heads
endpoint has been deprecated in favor of the v2 Bellatrix endpoint/eth/v2/debug/beacon/heads
v22.5.2
This is a recommended upgrade with configuration updates and optimisations.
Known Issues
- Some REST API requests may respond with SSZ instead of JSON when the header
Accept: */*
is sent. To ensure JSON is returned either omit theAccept
header or useAccept: application/json
.
Downloads
- Available as
22.5.2
on Dockerhub - Download the binary distribution:
Breaking Changes
- Removed
/teku/v1/beacon/states/{state_id}
, use/eth/v2/debug/beacon/states/{state_id}
instead.
Additions and Improvements
- Updated Ropsten testnet config to include extremely high TTD and enable proposer boost.
- Changed the default maximum peers count from 74 to 100 (
--p2p-peer-upper-bound 74
was old setting) - Update proposer boost weighting to 40%.
- Update libp2p
BeaconBlocksByRange
to only return the first block if the step is greater than 1, in line with 1.20 beacon chain networking spec. - Added
failOnRejectedCount
query parameter to liveness endpoint.
Bug Fixes
- Resolve a performance degradation in batch signature verification on machines with multiple, slower CPUs.
Upcoming Breaking Changes
- The
/eth/v1/debug/beacon/states/:state_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/debug/beacon/states/:state_id
- The
/eth/v1/beacon/blocks/:block_id
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/beacon/blocks/:block_id
- The
/eth/v1/validator/blocks/:slot
endpoint has been deprecated in favor of the v2 Altair endpoint/eth/v2/validator/blocks/:slot
- The
/eth/v1/debug/beacon/heads
endpoint has been deprecated in favor of the v2 Bellatrix endpoint/eth/v2/debug/beacon/heads
- The commandline option
--validators-performance-tracking-enabled
has been deprecated in favour of--validators-performance-tracking-mode