MDB_READERS_FULL: Environment maxreaders limit reached

[moneroexamples]

From here:
#121 (comment)

2019-01-23 05:59:21.882     7fd86ffd7700        WARNING blockchain.db.lmdb      src/blockchain_db/lmdb/db_lmdb.cpp:73Failed to create a read transaction for the db: MDB_READERS_FULL: Environment maxreaders limit reached               
2019-01-23 05:59:21,886 [I,OM,TxSearch.cpp:virtual void xmreg::TxSearch::operator()():139] A1zPXK: analyzing 200 blocks from 31800 to 31999 out of 1129675 blocks                                                                         
2019-01-23 05:59:21,918 [I,OM,TxSearch.cpp:virtual void xmreg::TxSearch::operator()():139] 9tE2Xc: analyzing 200 blocks from 155000 to 155199 out of 1129675 blocks                                                                       
2019-01-23 05:59:21,947 [E,OM,TxSearch.cpp:virtual void xmreg::TxSearch::operator()():605] 9sfsS3: exception in TxSearch!    

Happens at about 120 TxSearch threads.

This will also stop monero daemon:

status
2019-01-23 06:37:44.000     7fc006d0c700        WARNING blockchain.db.lmdb      src/blockchain_db/lmdb/db_lmdb.cpp:73Failed to create a read transaction for the db: MDB_READERS_FULL: Environment maxreaders limit reached               
2019-01-23 06:37:44.005     7fc006d0c700        ERROR   default contrib/epee/include/console_handler.h:392      Exception at [console_handler], what=Failed to create a read transaction for the db: MDB_READERS_FULL: Environment maxreaders limit reached

Before discussed in monero here: monero-project/monero#2741

[moneroexamples]

@italocoin-project

The fix is available in the branch https://github.com/moneroexamples/openmonero/tree/sync_microcore for testing. I had to add a tread pool, so that now all blockchain access operations are executed by several threads in the pool, rather than by individual account TxSearch threads. Many files had to be modified for this change, and probably still some code cleaning/reorganizations/tests-fixing are needed, but it should work now. I successfully tested it by importing 200 accounts from zero blockchain height at the same time, without issues.

How large the thread pool is (no of threads in it) can be set using this option: https://github.com/moneroexamples/openmonero/blob/sync_microcore/config/config.json#L61 or leave it to default value.

Logging systems was also updated. At the begionig you will be able to see size of the tread pool, e.g.:

[I,OM,main.cpp:int main(int, const char**):157] Thread pool size: 3 threads

Also size of the job queue is being shown from time to time:

void xmreg::CurrentBlockchainStatus::monitor_blockchain():49] PoolQueue size: 0

p.s.
Initially I tried blockchain access through rpc calls, as discussed, in this branch: https://github.com/moneroexamples/openmonero/tree/new_rpc But the performance was terrible. Accessing blockchain's lmdb directly is simply much faster than going through rpc. And a thread pool would be needed anyway, as rpc calls need to be synchronized, and you cant have hundreds of rpc connections open to blockchain as they would be timing-out, dropping etc. which I also checked.

[italocoin-project]

This looks awesome, i will test it out and let you know if i encounter issues 🥇

Initially I tried blockchain access through rpc calls, as discussed, in this branch: https://github.com/moneroexamples/openmonero/tree/new_rpc But the performance was terrible.

yes, i can imagine that accessing the lmdb directly is way faster, but at that time rpc connection to me was an ideal solution, the thread pool is better and more efficient for sure

[italocoin-project]

Markus did a thread pool in TxSearch that looks verry good and works well, check it out

ipbc-dev@de508d1

[mbehm]

The above commit linked by italo is just a quick implementation to change the whole TxSearch thread handling to be pooled, it's not really good to spawn a new thread for each wallet in any case because of needless context switching. In testing I can spam more connections than the restbed thread can handle, will happily keep syncing all of the ones that get through, it's also been running in a production env without issues for multiple days now*.

Few things I've been planning but haven't yet gotten around to:

1. a config variable/default to number of cores for the thread pool size
2. clean up io_service loop & stop flag to a class
3. move the thread_pool & io_service to be part of eg. MicroCore to clean up the static initialization hacks
4. add some logging on pool queue size etc

*The backport I did for older production instance we have still encountered an MDB_READERS_FULL error after the change, but I haven't been able to replicate it on either branch with the fix applied.

[moneroexamples]

@italocoin-project @mbehm

Thanks for link and info about TxSearch pool. Seems as a natural progression of having TxSearch done through pool as well. This would also help to resolve the issue of mysql/mariadb connections. At present each TxSearch thread has its own db connection, which in my recent tests, lead to reaching max number of db connection that mariadb/mysql can handle by default (150; can be increased in db settings, so its not hard limit).

rpc connection to me was an ideal solution,

monero also has ZeroMQ. Thus this could be potentially used as a middle ground between rpc and direct access to lmdb. However, at present I don't know what is its current status in monero, nor have work done with it.

*The backport I did for older production instance we have still encountered an MDB_READERS_FULL error after the change, but I haven't been able to replicate it on either branch with the fix applied.

TxSearch threads are not the only ones accessing blockchain. It could be that somewhere else, some code is "leaking" and accessing lmdb without going through the pool. Basically have to check where mcore (a pointer/reference to instance of MicroCore) is being used and replace it with the access through the pool. Hope this helps.

MicroCore to clean up the static initialization

MicoreCore in openmonero will also be replaced one day. Already made septate repo xmregcore for common classes, utility tools, to be shared among moneroexamples projects (and whoever find them useful). One of those classes will be UniversalIdentifier.hpp which will replace identification of outputs/inputs/paymentid in openmonero, onion explorer, transactions-export, future new projects, etc.

Similarly, there is also new project jsonifymonerotx aiming at creating complete json representation of txs and blocks. So that they can be used in unit testing of openmonero and other projects which need testing operations on txs and blocks, without accessing real blockchain for that data.

Both these new project require more work, readme updates, so they are not yet fully complete, but they are getting there.