eth_subscriptionLogs with topics filter problem

System information

Erigon version: erigon version 2022.04.3-alpha-87421417

OS & Version: Linux (Alpine 3.15.4 in Docker)

Commit hash : 8742141

Expected behaviour

When I'm subscribe to erigon node with topics filter - I must see transactions that corresponds to topics.

Actual behaviour

I see no any transactions at all, but they are (according to ETH/BSC explorer). If I don't use any filters (no topics) - I see all transactions, but I need only transactions with topics I'm interested in.

Steps to reproduce the behaviour

const Web3 = require("web3");
const web3 = new Web3(new Web3.providers.WebsocketProvider("wss://NODE_IP:NODE_PORT"));

let options = {
    topics: [
        null,
        '0x0000000000000000000000002faf487a4414fe77e2327f0bf4ae2a264a776ad2',
        null
    ]
};

let subscription = web3.eth.subscribe('logs', options, (err, event) => {
    if (err) throw err;
});

subscription.on('data', event => console.log(event))
subscription.on('changed', changed => console.log(changed))
subscription.on('error', err => { throw err })
subscription.on('connected', nr => console.log(nr))

The minimal code above works well with gEth nodes and getblock.io node.

So, what I miss?

I think subscription to logs not supported: https://github.com/ledgerwatch/erigon/tree/devel/cmd/rpcdaemon#rpc-implementation-status
eth_getLogs - works and fast

I think subscription to logs not supported: https://github.com/ledgerwatch/erigon/tree/devel/cmd/rpcdaemon#rpc-implementation-status
eth_getLogs - works and fast

But subscription to logs without topics - works great

const Web3 = require("web3");
const web3 = new Web3(new Web3.providers.WebsocketProvider("wss://NODE_IP:NODE_PORT"));

let subscription = web3.eth.subscribe('logs', {}, (err, event) => {
    if (err) throw err;
});

subscription.on('data', event => console.log(event))
subscription.on('changed', changed => console.log(changed))
subscription.on('error', err => { throw err })
subscription.on('connected', nr => console.log(nr))

This works. But when I add topics to options it's shows nothing((

Don’t know, maybe bug. Need somebody to check.

Could you please try
let options = { topics: [ '0x0000000000000000000000002faf487a4414fe77e2327f0bf4ae2a264a776ad2' ] };

instead of
let options = { topics: [ null, '0x0000000000000000000000002faf487a4414fe77e2327f0bf4ae2a264a776ad2', null ] };
?

I'm running similar issues with my erigon node running on Ubuntu 22.04 on commit aa79853. If I only have one log subscription it works fine, but running multiple at once with different topic/address queries causes unexpected behaviour. I'm testing using go-ethereum with this simple program, which subscribes to logs with the option to filter by address 0xdac17f958d2ee523a2206206994597c13d831ec7 (USDT) using the flag -usdt, and the option to filter by topics ["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"] (ERC20 Transfer event hash) using the flag -erc20

package main

import (
	"context"
	"flag"
	"fmt"
	"os"

	ethereum "github.com/ethereum/go-ethereum"
	ethCommon "github.com/ethereum/go-ethereum/common"
	"github.com/ethereum/go-ethereum/core/types"
	ethClient "github.com/ethereum/go-ethereum/ethclient"
)

const (
	usdtAddress         = "0xdac17f958d2ee523a2206206994597c13d831ec7"
	erc20TransferTopic0 = "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
)

func main() {
	websocketAddress := os.Getenv("ERIGON_WEBSOCKET_URL")
	if websocketAddress == "" {
		fmt.Println("Please set ERIGON_WEBSOCKET_URL environment variable")
		return
	}
	client, err := ethClient.Dial(websocketAddress)
	if err != nil {
		fmt.Printf("Failed to connect to websocket: %v\n", err)
		return
	}

	erc20, uniswap := parseCliArgs()

	var filter ethereum.FilterQuery
	if erc20 {
		filter.Topics = [][]ethCommon.Hash{{ethCommon.HexToHash(erc20TransferTopic0)}}
	}
	if uniswap {
		filter.Addresses = []ethCommon.Address{ethCommon.HexToAddress(usdtAddress)}
	}

	receivedLogs := make(chan types.Log)
	subscription, err := client.SubscribeFilterLogs(context.Background(), filter, receivedLogs)
	if err != nil {
		fmt.Printf("Unable to create subscription: %v\n", err)
		return
	}
	defer subscription.Unsubscribe()
	fmt.Println("Subscription successful")
	errChan := subscription.Err()

	for {
		select {
		case log := <-receivedLogs:
			fmt.Printf("Received log %d of block %d from address %s and topic0 %s\n",
				log.Index,
				log.BlockNumber,
				log.Address.String(),
				log.Topics[0].String(),
			)
		case err := <-errChan:
			fmt.Printf("Subscription error encountered: %v\n", err)
			return
		}
	}
}

// Parses command line flags, and returns whether or not to filter by the ERC20
// transfer topic 0 hash, and whether or not to filter by USDT address
func parseCliArgs() (bool, bool) {
	erc20TransferFlag := flag.Bool("erc20", false, "Whether to filter logs by topic 0 hash "+erc20TransferTopic0)
	usdtFlag := flag.Bool("usdt", false, "Whether to filter logs by address "+usdtAddress)
	flag.Parse()
	return *erc20TransferFlag, *usdtFlag
}

When I run this program multiple time concurrently, I get the following behaviour:

Running one with no flags and one with the -erc20 flag, neither receive any logs
Running one with no flags and one with the -usdt flag, neither receive any logs
Running one with no flags, one with the -erc20 flag, and one with both flags, the former two erroneously only receive logs from the usdt address
Running one with no flags, one with the -usdt flag, and one with both flags, the former two erroneously only receive logs with the erc20 transfer topic 0 hash
Running one with no flags, one with the -usdt flag, one with the -erc20 flag, and one with both flags, all four only receive logs with the erc20 transfer topic 0 hash from the usdt address
Running one with the -usdt flag, one with the -erc20 flag, and one with both flags, all behave as expected
Running one with no flags, and one with both flags, both behave as expected

This is a blocking bug for us to use Chainlink with Erigon. We require topic filtering on log streams.

Closing issue as fix is merged.

I'm not sure this is fixed, as we continue to see this issue. I cannot use eth_getLogs with any filters (address, topics). I can only get the full bloom from a log range. It's not the end of the world as I can filter in the client, but it adds a lot of latency and bandwidth requirements which in turn make it lose its advantage against geth.

@rcastellaj

Hi @rcastellaj, could you send some examples over of what you're running and I can take another look, thanks.

@hexoscott

@hexoscott This looks related to a similar issue with examples of usage and intermittent/different responses being generated: #6598

@samsondav

I don't believe this was fixed. As @samsondav said, this is required for chainlink.

I am running erigon v2.36.0 and I get the following error when trying to subscribe to a topic on goerli:

root@ip<erigonIP> bin]# curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"eth_subscribe","params":["logs", {"address": "0x005B0d11379c4c04C0B726eE0BE55feb50b59f81", "topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"]}], "id":1}' <Node address and port>
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"notifications not supported"}}

My system info:
Kernel -> Linux version 4.14.243-185.433.amzn2.x86_64
Distribution -> (gcc version 7.3.1 20180712 (Red Hat 7.3.1-13) (GCC))

@danieloadebimpe

I'm taking a look into this now, could you send me the flags you're using @danieloadebimpe please

@hexoscott

@hexoscott

command:
      - erigon
      - --datadir=/home/erigon/.local/share/erigon
      - --authrpc.jwtsecret=/home/erigon/.local/share/erigon/jwt.hex
      - --chain=goerli
      - --externalcl
      - --private.api.addr=0.0.0.0:9090
      - --http
      - --http.port=8546
      - --http.api=eth,erigon,engine,net,web3
      - --snapshots=false
      - --rpc.accessList=/home/erigon/rules.json
      - --http.addr=0.0.0.0
      - --http.vhosts=*
      - --ws
      - --metrics
      - --metrics.port=6060
      - --authrpc.addr=0.0.0.0
      - --authrpc.port=8551
      - --authrpc.vhosts=*
      ```

Just a quick check that eth_subscribe is in your rules.json?

Yes, heres the file:

{ "allow": [ "web3_clientVersion", "web3_sha3", "net_listening", "net_peerCount", "net_version", "eth_blockNumber", "eth_chainID", "eth_chainId", "eth_protocolVersion", "eth_syncing", "eth_gasPrice",
    "eth_maxPriorityFeePerGas", "eth_feeHistory", "eth_getBlockByHash", "eth_getBlockByNumber", "eth_getBlockTransactionCountByHash", "eth_getBlockTransactionCountByNumber",
    "eth_getUncleByBlockHashAndIndex", "eth_getUncleByBlockNumberAndIndex", "eth_getUncleCountByBlockHash", "eth_getUncleCountByBlockNumber", "eth_getTransactionByHash", "eth_getRawTransactionByHash",
    "eth_getTransactionByBlockHashAndIndex", "eth_retRawTransactionByBlockHashAndIndex", "eth_getTransactionByBlockNumberAndIndex", "eth_retRawTransactionByBlockNumberAndIndex",
    "eth_getTransactionReceipt", "eth_getBlockReceipts", "eth_estimateGas", "eth_getBalance", "eth_getCode", "eth_getTransactionCount", "eth_getStorageAt", "eth_call", "eth_callBundle",
    "eth_createAccessList", "eth_newFilter", "eth_newBlockFilter", "eth_newPendingTransactionFilter", "eth_getFilterChanges", "eth_getFilterLogs", "eth_uninstallFilter",
    "eth_getLogs", "eth_getProof", "eth_sendRawTransaction", "eth_coinbase", "eth_hashrate", "eth_submitHashrate", "eth_getWork", "eth_submitWork", "**eth_subscribe**", "eth_unsubscribe",
    "debug_accountRange", "debug_accountAt", "debug_getModifiedAccountsByNumber", "debug_getModifiedAccountsByHash", "debug_storageRangeAt", "debug_traceBlockByHash", "debug_traceBlockByNumber",
    "debug_traceTransaction", "debug_traceCall", "trace_call", "trace_callMany", "trace_rawTransaction", "trace_replayBlockTransactions", "trace_replayTransaction", "trace_block", "trace_filter",
    "trace_get", "trace_transaction", "txpool_content", "txpool_status", "erigon_getHeaderByHash", "erigon_getHeaderByNumber", "erigon_getLogsByHash", "erigon_forks", "erigon_issuance",
    "erigon_GetBlockByTimestamp"
  ]
}

Ah I see the issue here, you need to open a websocket connection to erigon first then send your request via that. Subscriptions are only enabled via sockets.

You'll get back a subscription ID which you can then use to cancel the subscription in the future if you wish.

An example sent via postman:

@hexoscott

@hexoscott is this issue being actively worked on? we've been seeing this issue with eth_getLogs per #6598. is there any additional info i could provide which would be helpful?

@ca42

Hi @ca42, it is, but I'm juggling it with other issues. I'll be taking another look into this today.

@ca42

@ca42 - do you have any recent examples that are re-produceable? Could you let me know the chain as well.

I'm going to close this issue down, the problem mentioned above around eth_getLogs was initially an mdbx version issue from the looks of things which is now resolved, and a typo in a test file leading to inconsistent results. Chances are the mdbx fix will have resolved this issue?

If the issue still persists, please leave examples and information such as chain and anything else that would be helpful and I'll pick the thread back up.

Navigation Menu

Repository navigation

eth_subscriptionLogs with topics filter problem #4030

System information

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

eth_subscriptionLogs with topics filter problem #4030

Description

System information

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

Activity

AskAlexSharov commented on Apr 30, 2022

d-polikhranidi commented on Apr 30, 2022

AskAlexSharov commented on Apr 30, 2022

primalcs commented on Jun 23, 2022

dan-nathan commented on Jun 29, 2022

samsondav commented on Aug 8, 2022

revitteth commented on Sep 16, 2022

rcastellaj commented on Sep 27, 2022

hexoscott commented on Sep 28, 2022

drcliche commented on Jan 26, 2023

danieloadebimpe commented on Feb 6, 2023

hexoscott commented on Feb 6, 2023

danieloadebimpe commented on Feb 6, 2023

hexoscott commented on Feb 6, 2023

danieloadebimpe commented on Feb 6, 2023

hexoscott commented on Feb 6, 2023

ca42 commented on Feb 16, 2023

hexoscott commented on Feb 20, 2023

hexoscott commented on Feb 20, 2023

hexoscott commented on Feb 27, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions