Don't fetch schema version on all connections

[muzarski]

During internal discussion, we concluded that drivers unnecessarily executes SELECT schema_version FROM system.local WHERE key='local' query on all connections when it awaits schema agreement. Each table (including system tables) is defined "per-node" and all shards read from it.

There is a room for optimization - we can execute the query on just one connection to the specific node.

Part of the discussion for reference:

Analogously, when driver checks for schema agreement, it executes SELECT schema_version FROM system.local WHERE key='local' on each shard and checks if queried schema_versions match.

This is obviously inefficient, there is only one scylla.local table per node, and all shards read from it.

[Lorak-mmk]

One thing that should be carefully considered is error handling.

In the current version, assuming we have multiple connections to a given node, it is ok for one of them to fail ( well not really because of a bug - #1240 - but lets assume it is working correctly, as it is for USE KEYSPACE). We will just ignore the broken connections, unless all of them are broken.

If we do the change naively this can happen:

• We have 2 connections to some node
• We want to await schema agreement
• We choose one of the connections
• It breaks
• Now we have no result from this node, despite still having an open connection to it.
• It can result in the driver falsely claiming agreement.

How to solve it? Probably with some retries.

[mykaul]

How to solve it? Probably with some retries.

Why? We have multiple connections, go to the next one? (I assume it failed due to overload, for example - that could be on the shard level or node level).

[Lorak-mmk]

How to solve it? Probably with some retries.

Why? We have multiple connections, go to the next one? (I assume it failed due to overload, for example - that could be on the shard level or node level).

That is what I meant. We failed to fetch schema_version on some connection, so we retry on another connection to this node. I should have been more specific, sorry.