Added

• Added distributed tracing in the OpenTelemetry trace format to the node, currently focused at the LOOPP Plugin development effort. This includes a new set of Tracing TOML configurations. The default for collecting traces is off - you must explicitly enable traces and setup a valid OpenTelemetry collector. Refer to .github/tracing/README.md for more details.
• Added a new, optional WebServer authentication option that supports LDAP as a user identity provider. This enables user login access and user roles to be managed and provisioned via a centralized remote server that supports the LDAP protocol, which can be helpful when running multiple nodes. See the documentation for more information and config setup instructions. There is a new [WebServer].AuthenticationMethod config option, when set to ldap requires the new [WebServer.LDAP] config section to be defined, see the reference docs/core.toml.
• New prom metrics for mercury transmit queue:
  mercury_transmit_queue_delete_error_count
mercury_transmit_queue_insert_error_count
mercury_transmit_queue_push_error_count
  Nops should consider alerting on these.
• Mercury now implements a local cache for fetching prices for fees, which ought to reduce latency and load on the mercury server, as well as increasing performance. It is enabled by default and can be configured with the following new config variables:

  [Mercury]
  
  # Mercury.Cache controls settings for the price retrieval cache querying a mercury server
  [Mercury.Cache]
  # LatestReportTTL controls how "stale" we will allow a price to be e.g. if
  # set to 1s, a new price will always be fetched if the last result was
  # from 1 second ago or older.
  # 
  # Another way of looking at it is such: the cache will _never_ return a
  # price that was queried from now-LatestReportTTL or before.
  # 
  # Setting to zero disables caching entirely.
  LatestReportTTL = "1s" # Default
  # MaxStaleAge is that maximum amount of time that a value can be stale
  # before it is deleted from the cache (a form of garbage collection).
  # 
  # This should generally be set to something much larger than
  # LatestReportTTL. Setting to zero disables garbage collection.
  MaxStaleAge = "1h" # Default
  # LatestReportDeadline controls how long to wait for a response from the
  # mercury server before retrying. Setting this to zero will wait indefinitely.
  LatestReportDeadline = "5s" # Default
  

• New prom metrics for the mercury cache:
  mercury_cache_fetch_failure_count
mercury_cache_hit_count
mercury_cache_wait_count
mercury_cache_miss_count
• Added new EVM.OCR TOML config fields DeltaCOverride and DeltaCJitterOverride for overriding the config DeltaC.
• Mercury v0.2 has improved consensus around current block that uses the most recent 5 blocks instead of only the latest one
• Two new prom metrics for mercury, nops should consider adding alerting on these:
  • mercury_insufficient_blocks_count
  • mercury_zero_blocks_count

Changed

• PromReporter no longer directly reads txm related status from the db, and instead uses the txStore API.
• L2Suggested mode is now called SuggestedPrice
• Console logs will now escape (non-whitespace) control characters
• Following EVM Pool metrics were renamed:
  • evm_pool_rpc_node_states → multi_node_states
  • evm_pool_rpc_node_num_transitions_to_alive → pool_rpc_node_num_transitions_to_alive
  • evm_pool_rpc_node_num_transitions_to_in_sync → pool_rpc_node_num_transitions_to_in_sync
  • evm_pool_rpc_node_num_transitions_to_out_of_sync → pool_rpc_node_num_transitions_to_out_of_sync
  • evm_pool_rpc_node_num_transitions_to_unreachable → pool_rpc_node_num_transitions_to_unreachable
  • evm_pool_rpc_node_num_transitions_to_invalid_chain_id → pool_rpc_node_num_transitions_to_invalid_chain_id
  • evm_pool_rpc_node_num_transitions_to_unusable → pool_rpc_node_num_transitions_to_unusable
  • evm_pool_rpc_node_highest_seen_block → pool_rpc_node_highest_seen_block
  • evm_pool_rpc_node_num_seen_blocks → pool_rpc_node_num_seen_blocks
  • evm_pool_rpc_node_polls_total → pool_rpc_node_polls_total
  • evm_pool_rpc_node_polls_failed → pool_rpc_node_polls_failed
  • evm_pool_rpc_node_polls_success → pool_rpc_node_polls_success

Removed

• Removed Optimism2 as a supported gas estimator mode

Fixed

• Corrected Ethereum Sepolia LinkContractAddress to 0x779877A7B0D9E8603169DdbD7836e478b4624789
• Fixed a bug that caused the Telemetry Manager to report incorrect health

Upcoming Required Configuration Changes

Starting in v2.9.0:

• TelemetryIngress.URL and TelemetryIngress.ServerPubKey will no longer be allowed. Any TOML configuration that sets this fields will prevent the node from booting. These fields will be replaced by [[TelemetryIngress.Endpoints]]
• P2P.V1 will no longer be supported and must not be set in TOML configuration in order to boot. Use P2P.V2 instead. If you are using both, V1 can simply be removed.