diff --git a/_build_scripts/slack-find-author.sh b/_build_scripts/slack-find-author.sh index b910d619..af037601 100644 --- a/_build_scripts/slack-find-author.sh +++ b/_build_scripts/slack-find-author.sh @@ -10,6 +10,7 @@ git_slack_map=( ["Charlie Harr"]="<@U044XTHRVFA>" ["Connor Shorten"]="<@U03FRH53SUT>" ["Daniel Madalitso Phiri"]="<@U060UJ41YBC>" + ["DanielleWashington"]="<@U088SBVDCET>" ["Dirk Kulawiak"]="<@U03MWHK4KV3>" ["Duda Nogueira"]="<@U05K3K9M82F>" ["dyma solovei"]="<@U07NGR323JR>" diff --git a/_includes/collections-count-limit.mdx b/_includes/collections-count-limit.mdx index 2c9dd21c..1079633a 100644 --- a/_includes/collections-count-limit.mdx +++ b/_includes/collections-count-limit.mdx @@ -1,3 +1,3 @@ :::info -To ensure optimal performance, Weaviate **limits the number of collections per instance**. The default limit is `100` collections and it can be adjusted via the [`MAXIMUM_ALLOWED_COLLECTIONS_COUNT`](/docs/weaviate/config-refs/env-vars) environment variable. +To ensure optimal performance, Weaviate **limits the number of collections per instance**. The default limit is `100` collections and it can be adjusted via the [`MAXIMUM_ALLOWED_COLLECTIONS_COUNT`](/docs/deploy/config-guides/env-vars/index.md) environment variable. ::: \ No newline at end of file diff --git a/_includes/dynamic-index-async-req.mdx b/_includes/dynamic-index-async-req.mdx index 2a3d3d68..45eda073 100644 --- a/_includes/dynamic-index-async-req.mdx +++ b/_includes/dynamic-index-async-req.mdx @@ -1,3 +1,3 @@ :::info Dynamic index requires `ASYNC_INDEXING` -Dynamic indexes require asynchronous indexing. To enable asynchronous indexing in a self-hosted Weaviate instance, set the `ASYNC_INDEXING` [environment variable](/docs/weaviate/config-refs/env-vars#general) to `true`. If your instance is hosted in Weaviate Cloud, use the Weaviate Cloud console to enable asynchronous indexing. +Dynamic indexes require asynchronous indexing. To enable asynchronous indexing in a self-hosted Weaviate instance, set the `ASYNC_INDEXING` [environment variable](/docs/deploy/config-guides/env-vars#general) to `true`. If your instance is hosted in Weaviate Cloud, use the Weaviate Cloud console to enable asynchronous indexing. ::: \ No newline at end of file diff --git a/docs/academy/deployment/k8s/30_setup_weaviate.mdx b/docs/academy/deployment/k8s/30_setup_weaviate.mdx index dbb25836..f1259e1f 100644 --- a/docs/academy/deployment/k8s/30_setup_weaviate.mdx +++ b/docs/academy/deployment/k8s/30_setup_weaviate.mdx @@ -139,7 +139,7 @@ Congratulations! You have successfully deployed Weaviate on your local Kubernete To upgrade to `1.25` or higher from a pre-`1.25` version, you must delete the deployed `StatefulSet`, update the helm chart to version `17.0.0` or higher, and re-deploy Weaviate. -See the [1.25 migration guide for Kubernetes](/docs/weaviate/more-resources/migration/weaviate-1-25) for more details. +See the [1.25 migration guide for Kubernetes](docs/deploy/migration/weaviate-1-25.md) for more details. ::: ## Questions and feedback diff --git a/docs/academy/deployment/k8s/90_next_steps.mdx b/docs/academy/deployment/k8s/90_next_steps.mdx index bbe15bb2..43847199 100644 --- a/docs/academy/deployment/k8s/90_next_steps.mdx +++ b/docs/academy/deployment/k8s/90_next_steps.mdx @@ -24,14 +24,14 @@ If you are interested in the architecture behind replication and sharding, pleas Then, check out the section on [resource planning](/docs/weaviate/concepts/resources.md) to estimate how much resources each node of a particular size might be able to handle, and how to plan your cluster accordingly. -If you have replication configured, you can apply [tunable consistency](/docs/weaviate/configuration/replication.md) concepts to read/write operations as well as queries. +If you have replication configured, you can apply [tunable consistency](docs/deploy/config-guides/replication.md) concepts to read/write operations as well as queries. ### Further reading - How-to guides - - The [How-to: Manage collections](/docs/weaviate/manage-collections/index.mdx) and [How-to: Manage objects](/docs/weaviate/manage-objects/index.mdx) guides show how to perform data operations (i.e. create, read, update, delete collections and objects within them).. - [How-to: search](/docs/weaviate/search/index.mdx): Code examples for all types of search operations. - - [How-to: configure Weaviate](/docs/weaviate/configuration/index.mdx): Guides for configuring Weaviate, such as [PQ](/docs/weaviate/configuration/compression/pq-compression.md) and [BQ](/docs/weaviate/configuration/compression/bq-compression.md) compression, [backups](/docs/weaviate/configuration/backups.md) and [replication](/docs/weaviate/configuration/replication.md). + - [How-to: configure Weaviate](/docs/weaviate/configuration/index.mdx): Guides for configuring Weaviate, such as [PQ](/docs/weaviate/configuration/compression/pq-compression.md) and [BQ](/docs/weaviate/configuration/compression/bq-compression.md) compression, [backups](docs/deploy/config-guides/backups.md) and [replication](docs/deploy/config-guides/replication.md). - [Concepts guides](/docs/weaviate/concepts/index.md): Guides for understanding Weaviate's architecture and concepts. - [API reference](/docs/weaviate/api/index.mdx): Detailed information about Weaviate's APIs. diff --git a/docs/academy/js/_snippets/intro_next_steps.mdx b/docs/academy/js/_snippets/intro_next_steps.mdx index df7f9d1f..38652e4e 100644 --- a/docs/academy/js/_snippets/intro_next_steps.mdx +++ b/docs/academy/js/_snippets/intro_next_steps.mdx @@ -9,7 +9,7 @@ Some of our more popular resources include: - How-to guides - The [How-to: Manage collections](/docs/weaviate/manage-collections/index.mdx) and [How-to: Manage objects](/docs/weaviate/manage-objects/index.mdx) guides show how to perform data operations (i.e. create, read, update, delete collections and objects within them).. - [How-to: search](/docs/weaviate/search/index.mdx): Code examples for all types of search operations. - - [How-to: configure Weaviate](/docs/weaviate/configuration/index.mdx): Guides for configuring Weaviate, such as [PQ](/docs/weaviate/configuration/compression/pq-compression.md) and [BQ](/docs/weaviate/configuration/compression/bq-compression.md) compression, [backups](/docs/weaviate/configuration/backups.md) and [replication](/docs/weaviate/configuration/replication.md). + - [How-to: configure Weaviate](/docs/weaviate/configuration/index.mdx): Guides for configuring Weaviate, such as [PQ](/docs/weaviate/configuration/compression/pq-compression.md) and [BQ](/docs/weaviate/configuration/compression/bq-compression.md) compression, [backups](docs/deploy/config-guides/backups.md) and [replication](docs/deploy/config-guides/replication.md). - [Concepts guides](/docs/weaviate/concepts/index.md): Guides for understanding Weaviate's architecture and concepts. - [API reference](/docs/weaviate/api/index.mdx): Detailed information about Weaviate's APIs. diff --git a/docs/academy/js/_snippets/intro_next_steps_js.mdx b/docs/academy/js/_snippets/intro_next_steps_js.mdx index 854b7b3a..88969c59 100644 --- a/docs/academy/js/_snippets/intro_next_steps_js.mdx +++ b/docs/academy/js/_snippets/intro_next_steps_js.mdx @@ -9,7 +9,7 @@ Some of our more popular resources include: - How-to guides - The [How-to: Manage collections](/docs/weaviate/manage-collections/index.mdx) and [How-to: Manage objects](/docs/weaviate/manage-objects/index.mdx) guides show how to perform data operations (i.e. create, read, update, delete collections and objects within them).. - [How-to: search](/docs/weaviate/search/index.mdx): Code examples for all types of search operations. - - [How-to: configure Weaviate](/docs/weaviate/configuration/index.mdx): Guides for configuring Weaviate, such as [PQ](/docs/weaviate/configuration/compression/pq-compression.md) and [BQ](/docs/weaviate/configuration/compression/bq-compression.md) compression, [backups](/docs/weaviate/configuration/backups.md) and [replication](/docs/weaviate/configuration/replication.md). + - [How-to: configure Weaviate](/docs/weaviate/configuration/index.mdx): Guides for configuring Weaviate, such as [PQ](/docs/weaviate/configuration/compression/pq-compression.md) and [BQ](/docs/weaviate/configuration/compression/bq-compression.md) compression, [backups](docs/deploy/config-guides/backups.md) and [replication](docs/deploy/config-guides/replication.md). - [Concepts guides](/docs/weaviate/concepts/index.md): Guides for understanding Weaviate's architecture and concepts. - [API reference](/docs/weaviate/api/index.mdx): Detailed information about Weaviate's APIs. diff --git a/docs/academy/py/_snippets/intro_next_steps.mdx b/docs/academy/py/_snippets/intro_next_steps.mdx index 02bf0dbf..6e6ad0b9 100644 --- a/docs/academy/py/_snippets/intro_next_steps.mdx +++ b/docs/academy/py/_snippets/intro_next_steps.mdx @@ -9,7 +9,7 @@ Some of our more popular resources include: - How-to guides - The [How-to: Manage collections](/docs/weaviate/manage-collections/index.mdx) and [How-to: Manage objects](/docs/weaviate/manage-objects/index.mdx) guides show how to perform data operations (i.e. create, read, update, delete collections and objects within them).. - [How-to: search](/docs/weaviate/search/index.mdx): Code examples for all types of search operations. - - [How-to: configure Weaviate](/docs/weaviate/configuration/index.mdx): Guides for configuring Weaviate, such as [PQ](/docs/weaviate/configuration/compression/pq-compression.md) and [BQ](/docs/weaviate/configuration/compression/bq-compression.md) compression, [backups](/docs/weaviate/configuration/backups.md) and [replication](/docs/weaviate/configuration/replication.md). + - [How-to: configure Weaviate](/docs/weaviate/configuration/index.mdx): Guides for configuring Weaviate, such as [PQ](/docs/weaviate/configuration/compression/pq-compression.md) and [BQ](/docs/weaviate/configuration/compression/bq-compression.md) compression, [backups](docs/deploy/config-guides/backups.md) and [replication](docs/deploy/config-guides/replication.md). - [Concepts guides](/docs/weaviate/concepts/index.md): Guides for understanding Weaviate's architecture and concepts. - [API reference](/docs/weaviate/api/index.mdx): Detailed information about Weaviate's APIs. diff --git a/docs/academy/py/multitenancy/200_setup.mdx b/docs/academy/py/multitenancy/200_setup.mdx index e4350e1f..1e447969 100644 --- a/docs/academy/py/multitenancy/200_setup.mdx +++ b/docs/academy/py/multitenancy/200_setup.mdx @@ -178,7 +178,7 @@ services: ### Configuration highlights -You may have seen Docker configurations elsewhere ([e.g. Docs](../../../weaviate/installation/docker-compose.md), [Academy](../starter_text_data/101_setup_weaviate/20_create_instance/20_create_docker.mdx)). But these highlighted configurations may be new to you: +You may have seen Docker configurations elsewhere ([e.g. Docs](docs/deploy/installation-guides/docker-installation.md), [Academy](../starter_text_data/101_setup_weaviate/20_create_instance/20_create_docker.mdx)). But these highlighted configurations may be new to you: - `ASYNC_INDEXING`: This will enable asynchronous indexing. This is useful for high-volume data insertion, and enables us to use the `dynamic` index type, which you will learn about later on. - `ENABLE_MODULES`: We enable `offload-s3` to demonstrate tenant offloading later on. Offloading helps us to manage inactive users' data efficiently. diff --git a/docs/academy/py/zero_to_mvp/101_hello_weaviate/40_set_up.mdx b/docs/academy/py/zero_to_mvp/101_hello_weaviate/40_set_up.mdx index 528c751d..d7d25daf 100644 --- a/docs/academy/py/zero_to_mvp/101_hello_weaviate/40_set_up.mdx +++ b/docs/academy/py/zero_to_mvp/101_hello_weaviate/40_set_up.mdx @@ -39,7 +39,7 @@ Running your own instance provides you with the same Weaviate code base as a WCD This course doesn't cover self-managed instances. We will cover Docker and Kubernetes in separate course units in the future. -If you are familiar with either solution and want to use them to install Weaviate now, see the documentation for [Docker-Compose](/docs/weaviate/installation/docker-compose.md) or [Kubernetes](/docs/weaviate/installation/kubernetes.md). +If you are familiar with either solution and want to use them to install Weaviate now, see the documentation for [Docker-Compose](docs/deploy/installation-guides/docker-installation.md) or [Kubernetes](docs/deploy/installation-guides/k8s-installation.md). ### Embedded Weaviate diff --git a/docs/academy/py/zero_to_mvp/103_schema_and_imports/20_schema.mdx b/docs/academy/py/zero_to_mvp/103_schema_and_imports/20_schema.mdx index bc295f92..b847bb8e 100644 --- a/docs/academy/py/zero_to_mvp/103_schema_and_imports/20_schema.mdx +++ b/docs/academy/py/zero_to_mvp/103_schema_and_imports/20_schema.mdx @@ -122,7 +122,7 @@ Modules are enabled at the instance level through its configuration. You can see :::info What is a `module`, exactly? By now, you've probably seen mentions of Weaviate `modules` here and there. Modules are optional Weaviate components used to enhance and customize its capabilities.


-Weaviate Academy units will generally assume WCD usage, which is pre-configured with a set of modules. We will cover how to enable modules for local instances in another unit, or you can see our [Docker installation page](/docs/weaviate/installation/docker-compose.md). +Weaviate Academy units will generally assume WCD usage, which is pre-configured with a set of modules. We will cover how to enable modules for local instances in another unit, or you can see our [Docker installation page](docs/deploy/installation-guides/docker-installation.md). ::: WCD instances come pre-configured with a number of modules. For example, the response below shows that the `text2vec-openai` module is available, so we can use it in our schema. diff --git a/docs/academy/theory/101_hello_weaviate/90_next_steps.mdx b/docs/academy/theory/101_hello_weaviate/90_next_steps.mdx index 00f90bc0..af611ca1 100644 --- a/docs/academy/theory/101_hello_weaviate/90_next_steps.mdx +++ b/docs/academy/theory/101_hello_weaviate/90_next_steps.mdx @@ -23,7 +23,7 @@ And for TypeScript/JavaScript: - How-to guides - The [How-to: Manage collections](/docs/weaviate/manage-collections/index.mdx) and [How-to: Manage objects](/docs/weaviate/manage-objects/index.mdx) guides show how to perform data operations (i.e. create, read, update, delete collections and objects within them).. - [How-to: search](/docs/weaviate/search/index.mdx): Code examples for all types of search operations. - - [How-to: configure Weaviate](/docs/weaviate/configuration/index.mdx): Guides for configuring Weaviate, such as [PQ](/docs/weaviate/configuration/compression/pq-compression.md) and [BQ](/docs/weaviate/configuration/compression/bq-compression.md) compression, [backups](/docs/weaviate/configuration/backups.md) and [replication](/docs/weaviate/configuration/replication.md). + - [How-to: configure Weaviate](/docs/weaviate/configuration/index.mdx): Guides for configuring Weaviate, such as [PQ](/docs/weaviate/configuration/compression/pq-compression.md) and [BQ](/docs/weaviate/configuration/compression/bq-compression.md) compression, [backups](docs/deploy/config-guides/backups.md) and [replication](docs/deploy/config-guides/replication.md). - [Concepts guides](/docs/weaviate/concepts/index.md): Guides for understanding Weaviate's architecture and concepts. - [API reference](/docs/weaviate/api/index.mdx): Detailed information about Weaviate's APIs. diff --git a/docs/cloud/manage-clusters/create.mdx b/docs/cloud/manage-clusters/create.mdx index e66eb4d7..3af981a1 100644 --- a/docs/cloud/manage-clusters/create.mdx +++ b/docs/cloud/manage-clusters/create.mdx @@ -130,7 +130,7 @@ The following advanced configuration settings are available: Enable async indexing
  • - Enable async replication + Enable async replication
  • Allow all CORS origins diff --git a/docs/cloud/manage-clusters/status.mdx b/docs/cloud/manage-clusters/status.mdx index ac0b36c6..df1b47f1 100644 --- a/docs/cloud/manage-clusters/status.mdx +++ b/docs/cloud/manage-clusters/status.mdx @@ -72,7 +72,7 @@ Serverless clusters also have an `Advanced configuration` section to configure o ## API Endpoint {#api-endpoint} -To get cluster details programmatically, use the [`nodes`](/docs/weaviate/config-refs/nodes.md) REST endpoint. +To get cluster details programmatically, use the [`nodes`](/docs/deploy/config-guides/nodes.md) REST endpoint. import APIOutputs from '/_includes/rest/node-endpoint-info.mdx'; diff --git a/docs/contributor-guide/weaviate-core/setup.md b/docs/contributor-guide/weaviate-core/setup.md index 0174aa37..db171ea2 100644 --- a/docs/contributor-guide/weaviate-core/setup.md +++ b/docs/contributor-guide/weaviate-core/setup.md @@ -31,8 +31,7 @@ To run the server locally with the OpenAI module. The default configuration is `local-development` which will run the server locally with the `text2vec-contextionary` and `backup-filesystem` modules. -You can also create your own configuration. For instance, you can clone an entry (`local-all-openai-cohere-google` is a good start) and add the required [environment variables](../../weaviate/config-refs/env-vars/index.md). - +You can also create your own configuration. For instance, you can clone an entry (`local-all-openai-cohere-google` is a good start) and add the required [environment variables](docs/deploy/config-guides/env-vars/index.md). ## Running with Docker @@ -54,7 +53,7 @@ tools/dev/restart_dev_environment.sh --prometheus && tools/dev/run_dev_server.sh ``` :::info -This setup is for contributors to the Weaviate code base. If you are an end-user of Weaviate looking for a Prometheus-enabled example, please see [this documentation page](/docs/weaviate/configuration/monitoring.md) or this [example](https://github.com/weaviate/weaviate-examples/tree/main/monitoring-prometheus-grafana). +This setup is for contributors to the Weaviate code base. If you are an end-user of Weaviate looking for a Prometheus-enabled example, please see [this documentation page](/docs/deploy/config-guides/monitoring.md) or this [example](https://github.com/weaviate/weaviate-examples/tree/main/monitoring-prometheus-grafana). ::: Below are more examples of running Weaviate with Docker. diff --git a/docs/deploy/aws/_category_.json b/docs/deploy/aws/_category_.json deleted file mode 100644 index 13cc9bd6..00000000 --- a/docs/deploy/aws/_category_.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "label": "AWS", - "position": 1 -} \ No newline at end of file diff --git a/docs/deploy/aws/index.mdx b/docs/deploy/aws/index.mdx deleted file mode 100644 index 713a3823..00000000 --- a/docs/deploy/aws/index.mdx +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: AWS and Weaviate -description: AWS docs -sidebar_position: 0 ---- - -# Tacetve a gemit quisquam praestat meruisse gradientis - -## Nec sum vagatur nupta tantus - -Lorem markdownum pugnando ventoque et ducta et et adurat locus inde cervos -refeci Phaethon clivum linguaque Salmacis geras coeperat mollia. Leucothoen Ceyx -est viret nostraque; meum utque latens -quae, patrios plebe inpositum: torvum. **Stipite fluminaque videri**; ictu -duabus sua *condita monstrum*. Imo undis: aut niveo sidoniae; placato sorores -at, quae perque. *Candore dolore*, et secum rastra ceratis et sollicitat vincula -matri mei alios in rigore murmur foedataque, cum. - -Et solvit ex comitante super luctus. Unda posse perque venitque quod algae -lymphis postquam optime. - -## Flammis levi harena illi membra caudam - -Ipse inducitur stridore Aiacem intonsum! Est arma, hausitque *tu* litora -aetasque, at fuit Aegea vides. Non laborum longi fortisque *putes bracchia* -umbras te fingit enim manum! - -Ingentia sive Idaeo serpere, dives inpia. Oculos in aras ergo bracchiaque vera, -omnis solutus vellet: et. Est et putat nec obvia unda -totidem amisit *coniurataeque sic arabat* mea. Caune -quodque **addita** truncis quid latebra: incubat pallorque turribus et ignarus -signum pectore quicquid Abantiades locus. **Ante** aliqua Aiacis deferre, ferunt -hunc omnia, et hic Aconteus nec. - -Odit festas dixit et quas fidissima, *est umbrae matrona* palude. Ait et quod -pectora circa. - -## Elegit sim ulla poteramus stridente cessit auras - -Tanta *Elinque*. Mea quae cultum Palameden Aglauros si sacra miserere quam ego, -**e scires** imo sonitusque tumidarum fremit quae ictu? Tum est sub vate vasto, -Galanthida in potui huc et temptat Cenaeo praeterque septem caede. - -> Suo ipse cubitique et vento in onerosa adunco, non. **In paelice Quirites** -> sunt stant sinebat superstes differt non per potentia illiusque nostris -> Hecateidos? Tempore haud imagine facta ora, uni quam fatorum. - -Altis animalibus pampinus venit mirabilis busto: tamen hunc, **salutant** -Stheneleia vultibus pectore. Evocat posuere anxia. - -Aper est; in tuum medius, responderat tamen: magis vix! Amor regno tamen tuli -remige facitote **reverentia**: loqui vade pignora. diff --git a/docs/deploy/config-guides/async-rep.md b/docs/deploy/config-guides/async-rep.md new file mode 100644 index 00000000..2743c863 --- /dev/null +++ b/docs/deploy/config-guides/async-rep.md @@ -0,0 +1,179 @@ +--- + +title: Async Replication + +--- + +Introduced to GA in the 1.29 release, Async Replication is a mechanism used to ensure eventual consistency across nodes in a distributed cluster. It works as a background process that automatically keeps nodes in sync without requiring user queries. Previously, consistency was achieved through "read repair" which involved nodes comparing data during a read request and exchanging missing or outdated information. This approach guarantees eventual consistency without requiring read operations. + +:::info + +This applies solely to data objects, as metadata consistency is treated differently (through RAFT consensus). +::: + +### Under the Hood + +- Async replication operates as a background process either per tenant (in a multi-tenant collection) or per shard (in a non-multi-tenant collection). +- It is disabled by default but can be enabled through collection configuration changes, similar to setting the replication factor. + +## Environment Variable Deep Dive + +These environment variables can be used to fine-tune behavior for your specific use case or deployment environment. + +:::tip +The optimal values for these variables will ultimately depend on factors like: data size, network conditions, write patterns, and the desired level of eventual consistency. +::: + +## Use Cases + +### General + +
    + + Feature Control +#### `ASYNC_REPLICATION_DISABLED` +Globally disables the entire async replication feature. + +- Its default value is `false`. +- **Use case**: This is useful when you have many tenants or collections where a temporary global disable is needed, like during debugging or critical maintenance. +- **Special Considerations**: + - This overrides any collection configuration. + +
    + +
    +Replication Control + +#### `ASYNC_REPLICATION_PROPAGATION_LIMIT` +Defines the maximum number of objects that will be propagated in a single async replication iteration (after one hash tree comparison). + - By default is set to 10,000. + - **Use Case(s)**: Can be adjusted based on network capacity and the desired rate of convergence. + - **Considerations**: Even if more than this number of differences are detected, only this many objects will be propagated in the current iteration. Subsequent iterations will handle the remaining differences. + + +#### `ASYNC_REPLICATION_PROPAGATION_DELAY` +Introduces a delay before considering an object for propagation. Only objects older than this delay are considered. + - By default it is set to 30 seconds. + - **Use Case(s)**: If an object is inserted into one node but the insertion is still in progress, the hash comparison might detect it. This delay prevents the async replication from trying to propagate it before the local write operation is fully complete. + - **Considerations**: This should be set based on the typical write latency of the system. +
    + +
    + Operational Visibility + +#### `ASYNC_REPLICATION_LOGGING_FREQUENCY` +Controls how often the background async replication process logs its activity. + - By default it is set to 5 seconds. + - **Use Case(s)**: Increasing the frequency provides more detailed logs, while decreasing it reduces log verbosity. +
    + +### Performance Tuning + +
    + + Memory Optimization + +#### `ASYNC_REPLICATION_HASHTREE_HEIGHT` +Customizes the height of the hash tree built by each node to represent its locally stored data. +- By default the value is set to 16 which is roughly 2MB of RAM per shard on each node. +- **Use case(s)**: + - In multi-tenant setups with a large number of tenants, reducing the hash tree would minimize the memory footprint. + - For very large collections, a larger hash tree could be more beneficial for more efficient identification of differing data ranges. +- **Special Considerations**: + - Modification of the hash tree height requires rebuilding the hash tree on each node, which involves iterating over all existing objects. + +
    + +
    + + Throughput and Concurrency + +#### `ASYNC_REPLICATION_PROPAGATION_CONCURRENCY` +Controls the number of concurrent goroutines (or threads) used to send batches of objects during the propagation phase. + - By default it is set to 5. + - **Considerations**: Increasing concurrency can improve propagation speed, but needs to be balanced with potential resource contention (CPU, network). + +
    + +
    + + Batch Processing + +#### `ASYNC_REPLICATION_DIFF_BATCH_SIZE` +Sets the number of object metadata fetched per request during the comparison phase. + - By default it is set to 1000. + - **Use Case(s)**: May be increased to potentially improve performance if network latency is low and nodes can handle larger requests. + - **Considerations**: Fetching metadata in batches optimizes network communication. + + +#### `ASYNC_REPLICATION_PROPAGATION_BATCH_SIZE` +Sets the maximum number of objects included in each batch when propagating data to a remote node. + - By default is set to 100. + - **Use Case(s)**: + - For large objects, reducing the batch size can help manage memory usage during propagation. The batch size could be similar to the batch size used during initial data insertion. + - For smaller objects, increasing the batch size might improve propagation efficiency by reducing the overhead of individual requests, but needs to be balanced with potential memory pressure. + - **Considerations**: This setting is particularly important for large objects, as larger batches can lead to higher memory consumption during transmission. Multiple batches may be sent within a single iteration to reach the `ASYNC_REPLICATION_PROPAGATION_LIMIT`. + +
    + +### Consistency Tuning + +
    + + Synchronization Frequency +#### `ASYNC_REPLICATION_FREQUENCY` +Defines how often each node initiates the process of comparing its local data (via the hash tree) with other nodes storing the same shard. This regularly checks for inconsistencies, even if no changes have been explicitly triggered. +- It's default value is 30 seconds. +- **Use Case(s)** + - Decreasing the frequency can be beneficial for applications that require faster convergence to eventual consistency. + - Increasing the frequency can be beneficial for reducing the load on the system by relaxing the eventual consistency. + +#### `ASYNC_REPLICATION_FREQUENCY_WHILE_PROPAGATING` +Defines a shorter frequency for subsequent comparison and propagation attempts when a previous propagation cycle did not complete (i.e., not all detected differences were synchronized). + - By default it is set to 20 milliseconds. + - **Use Case(s)**: When inconsistencies are known to exist, this expedites the synchronization process. + - **Considerations**: This is activated after a propagation cycle detects differences but does not propagate all of them due to limits. + +
    + +
    + Node Status Monitoring + +#### `ASYNC_REPLICATION_ALIVE_NODES_CHECKING_FREQUENCY` +Defines the frequency at which the system checks for changes in the availability of nodes within the cluster. + - By default it is set to 5 seconds. + - **Use Case(s)**: When a node rejoins the cluster after a period of downtime, it is highly likely to be out of sync. This setting ensures that the replication process is initiated promptly. + +
    + +
    +Timeout Management + +#### `ASYNC_REPLICATION_DIFF_PER_NODE_TIMEOUT` +Defines the maximum time to wait for a response when requesting object metadata from a remote node during the comparison phase, this prevents indefinite blocking if a node is unresponsive. + - By default is set to 10 seconds. + - **Use Case(s)**: May need to be increased in environments with high network latency or potentially slow-responding nodes. + +#### `ASYNC_REPLICATION_PROPAGATION_TIMEOUT` +Sets the maximum time allowed for a single propagation request (sending actual object data) to a remote node. + - By default is set to 30 seconds. + - **Use Case(s)**: May need to be increased in scenarios with high network latency, large object sizes (e.g., images, vectors), or when sending large batches of objects. + - **Considerations**: Network latency, batch size, and the size of the objects being propagated can all affect timeouts. + +
    + + + +### Further Resources + +- [Concepts: Replication](https://weaviate.io/developers/weaviate/concepts/replication-architecture/consistency) + +- [Replication How-To](docs/deploy/config-guides/replication.md#async-replication-settings) + +- [Environment Variables](docs/deploy/config-guides/env-vars/index.md#async-replication) + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/deploy/config-guides/authentication.md b/docs/deploy/config-guides/authentication.md new file mode 100644 index 00000000..89bc07df --- /dev/null +++ b/docs/deploy/config-guides/authentication.md @@ -0,0 +1,338 @@ +--- +title: Authentication +image: og/docs/configuration.jpg +# tags: ['authentication'] +--- + +:::info Authentication and authorization +Authentication and authorization are closely related concepts, and sometimes abbreviated as `AuthN` and `AuthZ`. Authentication (`AuthN`) is the process of verifying the identity of a user, while authorization (`AuthZ`) is the process of determining what permissions the user has. +::: + +Weaviate controls access through user authentication via API keys or OpenID Connect (OIDC), with an option for anonymous access. Users can then be assigned different [authorization](./authorization.md) levels, as shown in the diagram below. + +```mermaid +flowchart LR + %% Define main nodes + Request["Client
    Request"] + AuthCheck{"AuthN
    Enabled?"} + AccessCheck{"Check
    AuthZ"} + Access["✅ Access
    Granted"] + Denied["❌ Access
    Denied"] + + %% Define authentication method nodes + subgraph auth ["AuthN"] + direction LR + API["API Key"] + OIDC["OIDC"] + AuthResult{"Success?"} + end + + %% Define connections + Request --> AuthCheck + AuthCheck -->|"No"| AccessCheck + AuthCheck -->|"Yes"| auth + API --> AuthResult + OIDC --> AuthResult + AuthResult -->|"Yes"| AccessCheck + AuthResult -->|"No"| Denied + + AccessCheck -->|"Pass"| Access + AccessCheck -->|"Fail"| Denied + + %% Style nodes + style Request fill:#ffffff,stroke:#B9C8DF,color:#130C49 + style AuthCheck fill:#ffffff,stroke:#B9C8DF,color:#130C49 + style AccessCheck fill:#ffffff,stroke:#B9C8DF,color:#130C49 + style Access fill:#ffffff,stroke:#B9C8DF,color:#130C49 + style Denied fill:#ffffff,stroke:#B9C8DF,color:#130C49 + style API fill:#ffffff,stroke:#B9C8DF,color:#130C49 + style OIDC fill:#ffffff,stroke:#B9C8DF,color:#130C49 + style AuthResult fill:#ffffff,stroke:#B9C8DF,color:#130C49 + + %% Style subgraph + style auth fill:#ffffff,stroke:#130C49,stroke-width:2px,color:#130C49 +``` + +For example, a user logging in with the API key `jane-secret` may be granted administrator permissions, while another user logging in with the API key `ian-secret` may be granted read-only permissions. + +In summary, Weaviate allows the following authentication methods: + +- API key +- OpenID Connect (OIDC) +- Anonymous access **(no authentication, strongly discouraged except for development or evaluation)** + +Note that API key and OIDC authentication can be both enabled at the same time. + +The way to configure authentication differs by your deployment method, depending on whether you are running Weaviate in Docker or Kubernetes. Below, we provide examples for both. + +:::info What about Weaviate Cloud (WCD)? +For Weaviate Cloud (WCD) instances, authentication is pre-configured with OIDC and API key access. You can [authenticate against Weaviate](/docs/weaviate/connections/connect-cloud.mdx) with your WCD credentials using OIDC, or [with API keys](/docs/cloud/platform/manage-api-keys). +::: + +## API Key Authentication + +API key authentication is a simple and effective way to authenticate users. Each user is assigned a unique API key, which is used to authenticate the user. + +Note that you can either: +- Set one user for all API keys, or +- Define one user per API key (the number of users must match the number of API keys) + +Make sure all listed users are also configured in the authorization settings. + +### API keys: Docker + +API key authentication can be configured using environment variables. In Docker Compose, set them in the configuration file (`docker-compose.yml`) such as in the following example: + +```yaml +services: + weaviate: + ... + environment: + ... + # Disable anonymous access. + AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'false' + + # Enables API key authentication. + AUTHENTICATION_APIKEY_ENABLED: 'true' + + # List one or more keys in plaintext separated by commas. Each key corresponds to a specific user identity below. + AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'user-a-key,user-b-key' + + # List one or more user identities, separated by commas. Each identity corresponds to a specific key above. + AUTHENTICATION_APIKEY_USERS: 'user-a,user-b' +``` + +This configuration: +- Disables anonymous access +- Enables API key authentication +- Defines plaintext API keys in `AUTHENTICATION_APIKEY_ALLOWED_KEYS` +- Associates users with the API keys in `AUTHENTICATION_APIKEY_USERS` + +These users can now be assigned permissions based on the authorization settings. + +### API keys: Kubernetes + +For Kubernetes deployments using Helm, API key authentication can be configured in the `values.yaml` file under the `authentication` section. Here's an example configuration: + +```yaml +authentication: + anonymous_access: + # Disable anonymous access. + enabled: false + + apikey: + # Enables API key authentication. + enabled: true + + # List one or more keys in plaintext separated by commas. Each key corresponds to a specific user identity below. + allowed_keys: + - user-a-key + - user-b-key + + # List one or more user identities, separated by commas. Each identity corresponds to a specific key above. + users: + - user-a + - user-b +``` + +This configuration: + +- Disables anonymous access +- Enables API key authentication +- Defines plaintext API keys in `allowed_keys` +- Associates users with the API keys in `users` + +:::warning Environment Variables Take Precedence +If you configure API keys using environment variables, those settings will take precedence over the values in `values.yaml`. To use the Helm values configuration, ensure you have not set the corresponding environment variables. +::: + +For enhanced security in production environments, you can store API keys in Kubernetes secrets and reference them using environment variables instead of storing them as plaintext in the Helm values. + +## OIDC Authentication + +OIDC authentication requires the resource (Weaviate) to validate tokens issued by an identity provider. The identity provider authenticates the user and issues tokens, which are then validated by Weaviate. + +In an example setup, a Weaviate instance acts as the resource, Weaviate Cloud (WCD) acts as the identity provider, and the Weaviate client acts on behalf of the user. + +Any "OpenID Connect" compatible token issuer that implements OpenID Connect Discovery is compatible with Weaviate. + +This document discusses how to configure Weaviate as the resource. + +
    + + More about OIDC + + +With [OpenID Connect](https://openid.net/connect/) (based on OAuth2), an +external identity provider and token issuer ('token issuer' hereafter) is responsible for managing users. + +OIDC authentication requires obtaining a valid token from the token issuer so that it can be sent in the header of any request to Weaviate. This applies to both REST and GraphQL requests. + +When Weaviate receives a token (JSON Web Token or JWT), it verifies +that it was indeed signed by the configured token issuer. If the signature is +correct, all contents of the token are trusted, which authenticates the user based on the information in the token. + +
    + +### OIDC: Docker + +To configure Weaviate for OIDC-based authentication, add the following environment variables to your configuration file. + +An example `docker-compose.yml` file looks like this: + +```yaml +services: + weaviate: + ... + environment: + ... + # enabled (optional - defaults to false) turns OIDC auth on. All other fields in + # this section will only be validated if enabled is set to true. + AUTHENTICATION_OIDC_ENABLED: 'true' + + # issuer (required) tells weaviate how to discover the token issuer. This + # endpoint must implement the OpenID Connect Discovery spec, so that weaviate + # can retrieve the issuer's public key. + # + # The example URL below uses the path structure commonly found with keycloak + # where an example realm 'my-weaviate-usecase' was created. The exact + # path structure depends on the token issuer. See the token issuer's documentation + # about which endpoint implements OIDC Discovery. + AUTHENTICATION_OIDC_ISSUER: 'http://my-token-issuer/auth/realms/my-weaviate-usecase' + + # client_id (required unless skip_client_id_check is set to true) tells + # Weaviate to check for a particular OAuth 2.0 client_id in the audience claim. + # This is to prevent that a token which was signed by the correct issuer + # but never intended to be used with Weaviate can be used for authentication. + # + # For more information on what clients are in OAuth 2.0, see + # https://tools.ietf.org/html/rfc6749#section-1.1 + AUTHENTICATION_OIDC_CLIENT_ID: 'my-weaviate-client' + + # username_claim (required) tells Weaviate which claim in the token to use for extracting + # the username. The username will be passed to the authorization plugin. + AUTHENTICATION_OIDC_USERNAME_CLAIM: 'email' + + # skip_client_id_check (optional, defaults to false) skips the client_id + # validation in the audience claim as outlined in the section above. + # Not recommended to set this option as it reduces security, only set this + # if your token issuer is unable to provide a correct audience claim + AUTHENTICATION_OIDC_SKIP_CLIENT_ID_CHECK: 'false' + + # scope (optional) these will be used by clients as default scopes for authentication + AUTHENTICATION_OIDC_SCOPES: '' +``` + +:::info OIDC and Azure +As of November 2022, we were aware of some differences in Microsoft Azure's OIDC implementation compared to others. If you are using Azure and experiencing difficulties, [this external blog post](https://xsreality.medium.com/making-azure-ad-oidc-compliant-5734b70c43ff) may be useful. +::: + +### OIDC: Kubernetes + +For Kubernetes deployments using Helm, OIDC authentication can be configured in the `values.yaml` file under the `authentication` section. Here's an example configuration: + +```yaml +authentication: + anonymous_access: + # Disable anonymous access. + enabled: false + oidc: + # enabled (optional - defaults to false) turns OIDC auth on. All other fields in + # this section will only be validated if enabled is set to true. + enabled: true + + # issuer (required) tells weaviate how to discover the token issuer. This + # endpoint must implement the OpenID Connect Discovery spec, so that weaviate + # can retrieve the issuer's public key. + # + # The example URL below uses the path structure commonly found with keycloak + # where an example realm 'my-weaviate-usecase' was created. The exact + # path structure depends on the token issuer. See the token issuer's documentation + # about which endpoint implements OIDC Discovery. + issuer: 'http://my-token-issuer/auth/realms/my-weaviate-usecase' + + # client_id (required unless skip_client_id_check is set to true) tells + # Weaviate to check for a particular OAuth 2.0 client_id in the audience claim. + # This is to prevent that a token which was signed by the correct issuer + # but never intended to be used with Weaviate can be used for authentication. + # + # For more information on what clients are in OAuth 2.0, see + # https://tools.ietf.org/html/rfc6749#section-1.1 + client_id: 'my-weaviate-client' + + # username_claim (required) tells Weaviate which claim in the token to use for extracting + # the username. The username will be passed to the authorization plugin. + username_claim: 'email' + + # skip_client_id_check (optional, defaults to false) skips the client_id + # validation in the audience claim as outlined in the section above. + # Not recommended to set this option as it reduces security, only set this + # if your token issuer is unable to provide a correct audience claim + skip_client_id_check: 'false' + + # scope (optional) these will be used by clients as default scopes for authentication + scopes: '' + + # groups_claim: '' +``` + +### Note: Configuring the OIDC token issuer + +Configuring the OIDC token issuer is outside the scope of this document, but here are a few options as a starting point: + +- For simple use-cases such as for a single user, you can use Weaviate Cloud (WCD) as the OIDC token issuer. To do so: + - Make sure you have a WCD account (you can [sign up here](https://console.weaviate.cloud/)). + - In the Docker Compose file (e.g. `docker-compose.yml`), specify: + - `https://auth.wcs.api.weaviate.io/auth/realms/SeMI` as the issuer (in `AUTHENTICATION_OIDC_ISSUER`), + - `wcs` as the client id (in `AUTHENTICATION_OIDC_CLIENT_ID`), and + - enable the adminlist (`AUTHORIZATION_ADMINLIST_ENABLED: 'true'`) and add your WCD account email as the user (in `AUTHORIZATION_ADMINLIST_USERS`) . + - `email` as the username claim (in `AUTHENTICATION_OIDC_USERNAME_CLAIM`). + +- If you need a more customizable setup you can use commercial OIDC providers like [Okta](https://www.okta.com/). +- As another alternative, you can run your own OIDC token issuer server, which may be the most complex but also configurable solution. Popular open-source solutions include Java-based [Keycloak](https://www.keycloak.org/) and Golang-based [dex](https://github.com/dexidp/dex). + +:::info +By default, Weaviate validates that the token includes a specified client id in the audience claim. If your token issuer does not support this feature, you can turn it off as outlined in the configuration section below. +::: + +## Anonymous Access + +Weaviate can be configured to accept anonymous requests. This is strongly discouraged except for development or evaluation purposes. + +Users that send requests without explicit authentication are authenticated as `user: anonymous`. + +You can use the authorization plugin to specify which permissions to apply to this `anonymous` user. If anonymous access is disabled altogether, any request without an allowed authentication scheme returns `401 Unauthorized`. + +### Anonymous access: Docker + +To enable anonymous access in Docker Compose, add the following environment variable to your configuration file: + +```yaml +services: + weaviate: + ... + environment: + ... + AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' +``` + +### Anonymous access: Kubernetes + +To enable anonymous access in Kubernetes, add the following configuration to your `values.yaml` file: + +```yaml +authentication: + anonymous_access: + enabled: true +``` + +## Further resources + +- [Configuration: Authorization and RBAC](./authorization.md) +- [References: Environment variables / Authentication and Authorization](/docs/deploy/config-guides/env-vars#authentication-and-authorization) + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/weaviate/configuration/authorization.md b/docs/deploy/config-guides/authorization.md similarity index 93% rename from docs/weaviate/configuration/authorization.md rename to docs/deploy/config-guides/authorization.md index fdab62fe..58e5c776 100644 --- a/docs/weaviate/configuration/authorization.md +++ b/docs/deploy/config-guides/authorization.md @@ -1,6 +1,5 @@ --- title: Authorization -sidebar_position: 35 image: og/docs/configuration.jpg # tags: ['authorization'] --- @@ -79,7 +78,7 @@ In the Admin list authorization scheme, [anonymous users](#anonymous-users) can The way to configure authorization differs by your deployment method, depending on whether you are running Weaviate in Docker or Kubernetes. Below, we provide examples for both. :::info What about Weaviate Cloud (WCD)? -For Weaviate Cloud (WCD) instances, authorization is pre-configured with Admin list access. You can [authenticate against Weaviate](../connections/connect-cloud.mdx) with your WCD credentials using OIDC, or [with admin or read-only API keys](/docs/cloud/platform/manage-api-keys). +For Weaviate Cloud (WCD) instances, authorization is pre-configured with Admin list access. You can [authenticate against Weaviate](/docs/weaviate/connections/connect-cloud.mdx) with your WCD credentials using OIDC, or [with admin or read-only API keys](/docs/cloud/platform/manage-api-keys).
    RBAC access will be available in WCD in a future release. @@ -93,7 +92,7 @@ Role-based access control (RBAC) is generally available in Weaviate from version Role-based access control (RBAC) is a method of restricting access to resources based on the roles of users. In Weaviate, RBAC allows you to define **roles** and assign **permissions** to those roles. Users can then be assigned to roles, and inherit the permissions associated with those roles. -Check out the dedicated **[RBAC documentation](./rbac/index.mdx)** for instructions on how to [configure RBAC](./rbac/configuration.md) in your Weaviate instance and examples on how to [manage roles an users](./rbac/manage-roles.mdx). +Check out the dedicated **[RBAC documentation](/docs/weaviate/configuration/rbac/index.mdx)** for instructions on how to [configure RBAC](/docs/deploy/config-guides/configuring-rbac.md) in your Weaviate instance and examples on how to [manage roles an users](/docs/weaviate/configuration/rbac/manage-roles.mdx). ## Admin list @@ -250,8 +249,8 @@ Weaviate can be configured to provide undifferentiated access, by disabling auth ## Further resources - [Configuration: Authentication](./authentication.md) -- [Configuration: RBAC](./rbac/index.mdx) -- [References: Environment variables / Authentication and Authorization](../config-refs/env-vars/index.md#authentication-and-authorization) +- [Configuration: RBAC](/docs/weaviate/configuration/rbac/index.mdx) +- [References: Environment variables / Authentication and Authorization](docs/deploy/config-guides/env-vars/index.md#authentication-and-authorization) ## Questions and feedback diff --git a/docs/weaviate/configuration/backups.md b/docs/deploy/config-guides/backups.md similarity index 93% rename from docs/weaviate/configuration/backups.md rename to docs/deploy/config-guides/backups.md index 3731185a..b1914eb0 100644 --- a/docs/weaviate/configuration/backups.md +++ b/docs/deploy/config-guides/backups.md @@ -1,6 +1,5 @@ --- title: Backups -sidebar_position: 12 image: og/docs/configuration.jpg --- @@ -28,8 +27,8 @@ Weaviate's Backup feature is designed to work natively with cloud technology. Mo :::caution Important backup considerations -- **Version Requirements**: If you are running Weaviate `v1.23.12` or older, you must [update](../more-resources/migration/index.md) to `v1.23.13` or higher before restoring a backup to prevent data corruption. -- **[Multi-tenancy](../concepts/data.md#multi-tenancy) limitations**: Backups will only include `active` tenants. `Inactive` or `offloaded` tenants in multi-tenant collections will not be included. Be sure to [activate](../manage-collections/multi-tenancy.mdx#manage-tenant-states) any required tenants before creating a backup. +- **Version Requirements**: If you are running Weaviate `v1.23.12` or older, you must [update](/docs/deploy/migration/index.md) to `v1.23.13` or higher before restoring a backup to prevent data corruption. +- **[Multi-tenancy](docs/weaviate/concepts/data.md#multi-tenancy) limitations**: Backups will only include `active` tenants. `Inactive` or `offloaded` tenants in multi-tenant collections will not be included. Be sure to [activate](docs/weaviate/manage-collections/multi-tenancy.mdx#manage-tenant-states) any required tenants before creating a backup. ::: ## Backup Quickstart @@ -62,15 +61,6 @@ Restart Weaviate to apply the new configuration. Then, you are ready to start a /> - - - - - - - - - - - - - - - - - - REST API or [programmatically using a client library](./manage-roles.mdx). +You can connect to your instance with the root user in order to [create new users](docs/weaviate/configuration/rbac/manage-users.mdx) which can be assigned custom roles and permissions using the REST API or [programmatically using a client library](docs/weaviate/configuration/rbac/manage-roles.mdx). import DynamicUserManagement from '/_includes/configuration/dynamic-user-management.mdx'; @@ -104,7 +103,7 @@ This configuration: - Enables RBAC - Configures `root-user` as a user with built-in admin permissions -You can connect to your instance with the root user in order to [create new users](./manage-users.mdx) which can be assigned custom roles and permissions using the REST API or [programmatically using a client library](./manage-roles.mdx). +You can connect to your instance with the root user in order to [create new users](docs/weaviate/configuration/rbac/manage-users.mdx) which can be assigned custom roles and permissions using the REST API or [programmatically using a client library](docs/weaviate/configuration/rbac/manage-roles.mdx). ## RBAC and performance @@ -120,10 +119,10 @@ Here are some tips to optimize performance when using RBAC: ## Further resources -- [RBAC: Overview](./index.mdx) -- [RBAC: Manage roles](./manage-roles.mdx) -- [RBAC: Manage users](./manage-users.mdx) -- [RBAC: Tutorial](../../tutorials/rbac.mdx) +- [RBAC: Overview](docs/weaviate/configuration/rbac/index.mdx) +- [RBAC: Manage roles](docs/weaviate/configuration/rbac/manage-roles.mdx) +- [RBAC: Manage users](docs/weaviate/configuration/rbac/manage-users.mdx) +- [RBAC: Tutorial](docs/deploy/tutorials/rbac.mdx) ## Questions and feedback diff --git a/docs/weaviate/config-refs/env-vars/_category_.json b/docs/deploy/config-guides/env-vars/_category_.json similarity index 100% rename from docs/weaviate/config-refs/env-vars/_category_.json rename to docs/deploy/config-guides/env-vars/_category_.json diff --git a/docs/weaviate/config-refs/env-vars/index.md b/docs/deploy/config-guides/env-vars/index.md similarity index 85% rename from docs/weaviate/config-refs/env-vars/index.md rename to docs/deploy/config-guides/env-vars/index.md index 0ff1627b..8a02ae3c 100644 --- a/docs/weaviate/config-refs/env-vars/index.md +++ b/docs/deploy/config-guides/env-vars/index.md @@ -5,7 +5,7 @@ image: og/docs/configuration.jpg # tags: ['HNSW'] --- -To configure Weaviate in a [Docker](../../installation/docker-compose.md) or a [Kubernetes](../../installation/kubernetes.md) deployment, you can set these environment variables. +To configure Weaviate in a [Docker](/docs/deploy/installation-guides/docker-installation.md) or a [Kubernetes](/docs/deploy/installation-guides/k8s-installation.md) deployment, you can set these environment variables. :::info Boolean environment variables For Boolean environment variables, `"on"`, `"enabled"`, `"1"`, and `"true"` are interpreted as `true`. @@ -35,14 +35,14 @@ import APITable from '@site/src/components/APITable'; | `RAFT_ENABLE_ONE_NODE_RECOVERY` | Enable running the single node recovery routine on restart. This is useful if the default hostname has changed and a single node cluster believes there are supposed to be two nodes. | `boolean` | `false` | | `DEFAULT_VECTORIZER_MODULE` | Default vectorizer module - will be overridden by any class-level value defined in the schema | `string` | `text2vec-contextionary` | | `DISABLE_LAZY_LOAD_SHARDS` | New in v1.23. When `false`, enable lazy shard loading to improve mean time to recovery in multi-tenant deployments. | `string` | `false` | -| `DISABLE_TELEMETRY` | Disable [telemetry](../telemetry.md) data collection | boolean | `false` | -| `DISK_USE_READONLY_PERCENTAGE` | If disk usage is higher than the given percentage all shards on the affected node will be marked as `READONLY`, meaning all future write requests will fail. See [Disk Pressure Warnings and Limits for details](/docs/weaviate/configuration/persistence.md#disk-pressure-warnings-and-limits). | `string - number` | `90` | -| `DISK_USE_WARNING_PERCENTAGE` | If disk usage is higher than the given percentage a warning will be logged by all shards on the affected node's disk. See [Disk Pressure Warnings and Limits for details](/docs/weaviate/configuration/persistence.md#disk-pressure-warnings-and-limits). | `string - number` | `80` | +| `DISABLE_TELEMETRY` | Disable [telemetry](docs/deploy/config-guides/telemetry.md) data collection | boolean | `false` | +| `DISK_USE_READONLY_PERCENTAGE` | If disk usage is higher than the given percentage all shards on the affected node will be marked as `READONLY`, meaning all future write requests will fail. See [Disk Pressure Warnings and Limits for details](docs/deploy/config-guides/persistence.md#disk-pressure-warnings-and-limits). | `string - number` | `90` | +| `DISK_USE_WARNING_PERCENTAGE` | If disk usage is higher than the given percentage a warning will be logged by all shards on the affected node's disk. See [Disk Pressure Warnings and Limits for details](docs/deploy/config-guides/persistence.md#disk-pressure-warnings-and-limits). | `string - number` | `80` | | `ENABLE_API_BASED_MODULES` | Enable all API-based modules. (Experimental as of `v1.26.0`) | `boolean` | `true` | | `ENABLE_MODULES` | Specify Weaviate modules to enable | `string - comma separated names` | `text2vec-openai,generative-openai` | -| `ENABLE_TOKENIZER_GSE` | Enable the [`GSE` tokenizer](../../config-refs/schema/index.md#gse-and-trigram-tokenization-methods) for use | `boolean` | `true` | -| `ENABLE_TOKENIZER_KAGOME_JA` | Enable the [`Kagome` tokenizer for Japanese](../../config-refs/schema/index.md#kagome_ja-tokenization-method) for use (Experimental as of `v1.28.0`) | `boolean` | `true` | -| `ENABLE_TOKENIZER_KAGOME_KR` | Enable the [`Kagome` tokenizer for Korean](../../config-refs/schema/index.md#kagome_kr-tokenization-method) for use (Experimental as of `v1.25.7`) | `boolean` | `true` | +| `ENABLE_TOKENIZER_GSE` | Enable the [`GSE` tokenizer](/docs/weaviate/config-refs/schema/index.md#gse-and-trigram-tokenization-methods) for use | `boolean` | `true` | +| `ENABLE_TOKENIZER_KAGOME_JA` | Enable the [`Kagome` tokenizer for Japanese](docs/weaviate/config-refs/schema/index.md#kagome_ja-tokenization-method) for use (Experimental as of `v1.28.0`) | `boolean` | `true` | +| `ENABLE_TOKENIZER_KAGOME_KR` | Enable the [`Kagome` tokenizer for Korean](docs/weaviate/config-refs/schema/index.md#kagome_kr-tokenization-method) for use (Experimental as of `v1.25.7`) | `boolean` | `true` | | `GODEBUG` | Controls debugging variables within the runtime. [See official Go docs](https://pkg.go.dev/runtime). | `string - comma-separated list of name=val pairs` | `gctrace=1` | | `GOMAXPROCS` | Set the maximum number of threads that can be executing simultaneously. If this value is set, it be respected by `LIMIT_RESOURCES`. | `string - number` | `NUMBER_OF_CPU_CORES` | | `GOMEMLIMIT` | Set the memory limit for the Go runtime. A suggested value is between 90-80% of your total memory for Weaviate. The Go runtime tries to make sure that long-lived and temporary memory allocations do not exceed this value by making the garbage collector more aggressive as the memory usage approaches the limit. [Learn more about GOMEMLIMIT](https://weaviate.io/blog/gomemlimit-a-game-changer-for-high-memory-applications). | `string - memory limit in SI units` | `4096MiB` | @@ -53,28 +53,28 @@ import APITable from '@site/src/components/APITable'; | `LIMIT_RESOURCES` | If `true`, Weaviate will automatically attempt to auto-detect and limit the amount of resources (memory & threads) it uses to (0.8 * total memory) and (number of cores-1). It will override any `GOMEMLIMIT` values, however it will respect `GOMAXPROCS` values. | `boolean` | `false` | | `LOG_FORMAT` | Set the Weaviate logging format

    Default: Outputs log data to json. e.g.: `{"action":"startup","level":"debug","msg":"finished initializing modules","time":"2023-04-12T05:07:43Z"}`
    `text`: Outputs log data to a string. e.g. `time="2023-04-12T04:54:23Z" level=debug msg="finished initializing modules" action=startup` | `string` | | | `LOG_LEVEL` | Sets the Weaviate logging level. Default: `info`

    `panic`: Panic entries only. (new in `v1.24`)
    `fatal`: Fatal entries only. (new in `v1.24`)
    `error`: Error entries only. (new in `v1.24`)
    `warning`: Warning entries only. (new in `v1.24`)
    `info`: General operational entries.
    `debug`: Very verbose logging.
    `trace`: Even finer-grained informational events than `debug`. | `string` | | -| `MAXIMUM_ALLOWED_COLLECTIONS_COUNT` | Maximum allowed number of collections in a Weaviate node. A value of `-1` removes the limit. Default: `1000`

    Instead of raising the collections count limit, consider [rethinking your architecture](../../starter-guides/managing-collections/collections-scaling-limits.mdx).
    Added in `v1.30`| `string - number` | `20` | +| `MAXIMUM_ALLOWED_COLLECTIONS_COUNT` | Maximum allowed number of collections in a Weaviate node. A value of `-1` removes the limit. Default: `1000`

    Instead of raising the collections count limit, consider [rethinking your architecture](docs/weaviate/starter-guides/managing-collections/collections-scaling-limits.mdx).
    Added in `v1.30`| `string - number` | `20` | | `MEMORY_READONLY_PERCENTAGE` | If memory usage is higher than the given percentage all shards on the affected node will be marked as `READONLY`, meaning all future write requests will fail. (Default: `80`) | `string - number` | `75` | | `MEMORY_WARNING_PERCENTAGE` | If memory usage is higher than the given percentage a warning will be logged by all shards on the affected node's disk. (Default: `0` - i.e. no limit) | `string - number` | `85` | | `MODULES_CLIENT_TIMEOUT` | Timeout for requests to Weaviate modules. Default: `50s` | `string - duration` | `5s`, `10m`, `1h` | | `ORIGIN` | Set the http(s) origin for Weaviate | `string - HTTP origin` | `https://my-weaviate-deployment.com` | -| `PERSISTENCE_DATA_PATH` | Path to the Weaviate data store.
    [Note about file systems and performance](../../concepts/resources.md#file-system). | `string - file path` | `/var/lib/weaviate`
    Starting in v1.24, defaults to `./data`| -| `PERSISTENCE_HNSW_MAX_LOG_SIZE` | Maximum size of the HNSW [write-ahead-log](../../concepts/storage.md#hnsw-vector-index-storage). Increase this to improve log compaction efficiency, or decrease to reduce memory requirements. Default: 500MiB | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) | +| `PERSISTENCE_DATA_PATH` | Path to the Weaviate data store.
    [Note about file systems and performance](docs/weaviate/concepts/resources.md#file-system). | `string - file path` | `/var/lib/weaviate`
    Starting in v1.24, defaults to `./data`| +| `PERSISTENCE_HNSW_MAX_LOG_SIZE` | Maximum size of the HNSW [write-ahead-log](docs/weaviate/concepts/storage.md#hnsw-vector-index-storage). Increase this to improve log compaction efficiency, or decrease to reduce memory requirements. Default: 500MiB | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) | | `PERSISTENCE_LSM_ACCESS_STRATEGY` | Function used to access disk data in virtual memory. Default: `mmap` | `string` | `mmap` or `pread` | -| `PERSISTENCE_LSM_MAX_SEGMENT_SIZE` | Maximum size of a segment in the [LSM store](../../concepts/storage.md#object-and-inverted-index-store). Set this to limit disk usage spikes during compaction to ~2x the segment size. Default: no limit | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) | -| `PROMETHEUS_MONITORING_ENABLED` | If set, Weaviate collects [metrics in a Prometheus-compatible format](/docs/weaviate/configuration/monitoring.md) | `boolean` | `false` | +| `PERSISTENCE_LSM_MAX_SEGMENT_SIZE` | Maximum size of a segment in the [LSM store](docs/weaviate/concepts/storage.md#object-and-inverted-index-store). Set this to limit disk usage spikes during compaction to ~2x the segment size. Default: no limit | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) | +| `PROMETHEUS_MONITORING_ENABLED` | If set, Weaviate collects [metrics in a Prometheus-compatible format](docs/deploy/config-guides/monitoring.md) | `boolean` | `false` | | `PROMETHEUS_MONITORING_GROUP` | If set, Weaviate groups metrics for the same class across all shards. | `boolean` | `true` | | `QUERY_CROSS_REFERENCE_DEPTH_LIMIT` | Sets the maximum depth of cross-references to be resolved in a query. Defaults to 5.
    Added in `v1.24.25`, `v1.25.18`, `v1.26.5`. | `string - number` | `3` | | `QUERY_DEFAULTS_LIMIT` | Sets the default number of objects to be returned in a query. | `string - number` | `25`
    Starting in v1.24, defaults to `10`| | `QUERY_MAXIMUM_RESULTS` | Sets the maximum total number of objects that can be retrieved. | `string - number` | `10000` | | `QUERY_SLOW_LOG_ENABLED` | Log slow queries for debugging. Requires a restart to update.
    (New in 1.24.16, 1.25.3) | `boolean` | `False` | | `QUERY_SLOW_LOG_THRESHOLD` | Set a threshold time for slow query logging. Requires a restart to update.
    (New in 1.24.16, 1.25.3) | `string` | `2s`
    Values are times: `3h`, `2s`, `100ms` | -| `REINDEX_SET_TO_ROARINGSET_AT_STARTUP` | Allow Weaviate to perform a one-off re-indexing to [use Roaring Bitmaps](../../concepts/filtering.md#indexFilterable).

    Available in versions `1.18` and higher. | `boolean` | `true` | +| `REINDEX_SET_TO_ROARINGSET_AT_STARTUP` | Allow Weaviate to perform a one-off re-indexing to [use Roaring Bitmaps](docs/weaviate/concepts/filtering.md#indexFilterable).

    Available in versions `1.18` and higher. | `boolean` | `true` | | `TOKENIZER_CONCURRENCY_COUNT` | Limit the combined number of GSE and Kagome tokenizers running at the same time. Default: `GOMAXPROCS` | `string - number` | `NUMBER_OF_CPU_CORES` | | `TOMBSTONE_DELETION_CONCURRENCY` | The maximum number of cores to use for tombstone deletion. Set this to limit the number of cores used for cleanup. Default: Half of the available cores. (New in `v1.24.0`) | `string - int` | `4` | | `TOMBSTONE_DELETION_MAX_PER_CYCLE` | Maximum number of tombstones to delete per cleanup cycle. Set this to limit cleanup cycles, as they are resource-intensive. As an example, set a maximum of 10000000 (10M) for a cluster with 300 million-object shards. Default: none | `string - int` (New in `v1.24.15` / `v1.25.2`) | `10000000` | | `TOMBSTONE_DELETION_MIN_PER_CYCLE` | Minimum number of tombstones to delete per cleanup cycle. Set this to prevent triggering unnecessary cleanup cycles below a threshold. As an example, set a minimum of 1000000 (1M) for a cluster with 300 million-object shards. Default: 0 (New in `v1.24.15`, `v1.25.2`) | `string - int` | `100000` | -| `USE_GSE` | Enable the [`GSE` tokenizer](../../config-refs/schema/index.md#gse-and-trigram-tokenization-methods) for use.
    (The same as `ENABLE_TOKENIZER_GSE`. We recommend using `ENABLE_TOKENIZER_GSE` for consistency in naming with other optional tokenizers.) | `boolean` | `true` | +| `USE_GSE` | Enable the [`GSE` tokenizer](docs/weaviate/config-refs/schema/index.md#gse-and-trigram-tokenization-methods) for use.
    (The same as `ENABLE_TOKENIZER_GSE`. We recommend using `ENABLE_TOKENIZER_GSE` for consistency in naming with other optional tokenizers.) | `boolean` | `true` | | `USE_INVERTED_SEARCHABLE` | Store searchable properties using a more efficient in-disk format, designed for the BlockMax WAND algorithm. Set as `true` together with `USE_BLOCKMAX_WAND` to enable BlockMax WAND at query time.

    Added in `v1.28` default: `false`
    From `v1.30` default: `true`
    Read more | `boolean` | `true` | | `USE_BLOCKMAX_WAND` | Use BlockMax WAND algorithm for BM25 and hybrid searches. Enable it together with `USE_INVERTED_SEARCHABLE` to get the performance benefits.

    Added in `v1.28` default: `false`
    From `v1.30` default: `true`
    Read more | `boolean` | `true` | @@ -90,7 +90,7 @@ import APITable from '@site/src/components/APITable'; | Variable | Description | Type | Example Value | | --- | --- | --- | --- | -| `BACKUP_*` | Various configuration variables for backup provider modules. They are outlined in detail on the [Backups page](/docs/weaviate/configuration/backups.md). | | +| `BACKUP_*` | Various configuration variables for backup provider modules. They are outlined in detail on the [Backups page](docs/deploy/config-guides/backups.md). | | | `AZURE_BLOCK_SIZE` | The block size for Azure Blob Storage for backups. Default: `41943040` (40MB) | `int - bytes` | `10000000` | | `AZURE_CONCURRENCY` | The maximum number of parts that will be concurrently uploaded/downloaded during backup operations. Default: `1` | `int` | `3` | | `CLIP_INFERENCE_API` | The endpoint where to reach the clip module if enabled | `string` | `http://multi2vec-clip:8000` | @@ -102,7 +102,7 @@ import APITable from '@site/src/components/APITable'; | `OFFLOAD_S3_CONCURRENCY` | The maximum number of parts that will be uploaded/downloaded in parallel during offloading operations (default: `25`) | `string - number` | `10` | | `OFFLOAD_TIMEOUT` | The request timeout value, in seconds (default: `120`) | `string - number` | `60` | | `TRANSFORMERS_INFERENCE_API` | The endpoint where to reach the transformers module if enabled | `string` | `http://t2v-transformers:8080` | -| `USE_GOOGLE_AUTH` | Automatically look for Google Cloud credentials, and generate Vertex AI access tokens for Weaviate to use as needed ([read more](../../model-providers/google/index.md)). (default: `false`) | `boolean` | `true` | +| `USE_GOOGLE_AUTH` | Automatically look for Google Cloud credentials, and generate Vertex AI access tokens for Weaviate to use as needed ([read more](docs/weaviate/model-providers/google/index.md)). (default: `false`) | `boolean` | `true` | | `USE_SENTENCE_TRANSFORMERS_VECTORIZER` | (EXPERIMENTAL) Use the `sentence-transformer` vectorizer instead of the default vectorizer (from the `transformers` library). Applies to custom images only. | `boolean` | `true` | | `CLIP_WAIT_FOR_STARTUP` | If `true`, Weaviate waits for the `multi2vec-clip` module to start up before starting (default: `true`). | `boolean` | `true` | | `NER_WAIT_FOR_STARTUP` | If `true`, Weaviate waits for the `ner-transformers` module to start up before starting (default: `true`). (Available from `v1.25.27`, `v1.26.12`, `v1.27.7`) | `boolean` | `true` | @@ -119,7 +119,7 @@ import APITable from '@site/src/components/APITable'; ## Authentication and authorization :::info Authentication & Authorization documentation -For more information on authentication and authorization, see the [Authentication](../../configuration/authentication.md) and [Authorization](../../configuration/authorization.md) pages. +For more information on authentication and authorization, see the [Authentication](docs/deploy/config-guides/authentication.md) and [Authorization](docs/deploy/config-guides/authorization.md) pages. ::: ```mdx-code-block @@ -132,7 +132,7 @@ For more information on authentication and authorization, see the [Authenticatio | `AUTHENTICATION_APIKEY_ALLOWED_KEYS` | Allowed API keys.

    Each key corresponds to a specific user identity below. | `string - comma-separated list` | `jane-secret-key,ian-secret-key` | | `AUTHENTICATION_APIKEY_ENABLED` | Enable API key-based authentication | `boolean` | `false` | | `AUTHENTICATION_APIKEY_USERS` | API key-based identities.

    Each identity corresponds to a specific key above. | `string - comma-separated list` | `jane@doe.com,ian-smith` | -| `AUTHENTICATION_DB_USERS_ENABLED` | Allow runtime [user management](../../configuration/rbac/manage-users.mdx). Default: `false` | `boolean` | `true` | +| `AUTHENTICATION_DB_USERS_ENABLED` | Allow runtime [user management](docs/weaviate/configuration/rbac/manage-users.mdx). Default: `false` | `boolean` | `true` | | `AUTHENTICATION_OIDC_CLIENT_ID` | OIDC Client ID | `string` | `my-client-id` | | `AUTHENTICATION_OIDC_ENABLED` | Enable OIDC-based authentication | `boolean` | `false` | | `AUTHENTICATION_OIDC_GROUPS_CLAIM` | OIDC Groups Claim | `string` | `groups` | @@ -174,9 +174,9 @@ For more information on authentication and authorization, see the [Authenticatio | `CLUSTER_HOSTNAME` | Hostname of a node. Always set this value if the default OS hostname might change over time. | `string` | `node1` | | `CLUSTER_JOIN` | The service name of the "founding" member node in a cluster setup | `string` | `weaviate-node-1:7100` | | `HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE` | If `true`, vector cache prefill is synchronous when a node starts. The node reports ready to serve when the cache is hot. Defaults to `false`. Added in 1.24.20 and 1.25.5. | `boolean` | `false` | -| `COLLECTION_RETRIEVAL_STRATEGY`| Set collection definition retrieval behavior for a data request.

    • `LeaderOnly` (default): Always requests the definition from the leader node.
    • `LocalOnly`: Always use the local definition
    • `LeaderOnMismatch`: Requests the definition if outdated.
    ([Read more](../../concepts/replication-architecture/consistency.md#collection-definition-requests-in-queries)) (Added in `v1.27.10`, `v1.28.4`) | `string` | `LeaderOnly` | -| `RAFT_ENABLE_FQDN_RESOLVER` | If `true`, use DNS lookup instead of memberlist lookup for Raft. Added in `v1.25.15` and removed in `v1.30`. ([Read more](../../concepts/cluster.md#node-discovery)) | `boolean` | `true` | -| `RAFT_FQDN_RESOLVER_TLD` | The top-level domain to use for DNS lookup, in `[node-id].[tld]` format. Added in `v1.25.15` and removed in `v1.30`. ([Read more](../../concepts/cluster.md#node-discovery)) | `string` | `example.com` | +| `COLLECTION_RETRIEVAL_STRATEGY`| Set collection definition retrieval behavior for a data request.

    • `LeaderOnly` (default): Always requests the definition from the leader node.
    • `LocalOnly`: Always use the local definition
    • `LeaderOnMismatch`: Requests the definition if outdated.
    ([Read more](docs/weaviate/concepts/replication-architecture/consistency.md#collection-definition-requests-in-queries)) (Added in `v1.27.10`, `v1.28.4`) | `string` | `LeaderOnly` | +| `RAFT_ENABLE_FQDN_RESOLVER` | If `true`, use DNS lookup instead of memberlist lookup for Raft. Added in `v1.25.15` and removed in `v1.30`. ([Read more](docs/weaviate/concepts/cluster.md#node-discovery)) | `boolean` | `true` | +| `RAFT_FQDN_RESOLVER_TLD` | The top-level domain to use for DNS lookup, in `[node-id].[tld]` format. Added in `v1.25.15` and removed in `v1.30`. ([Read more](docs/weaviate/concepts/cluster.md#node-discovery)) | `string` | `example.com` | | `RAFT_BOOTSTRAP_EXPECT` | The number of voter notes at bootstrapping time | `string - number` | `1` | | `RAFT_BOOTSTRAP_TIMEOUT` | The time in seconds to wait for the cluster to bootstrap | `string - number` | `90` | | `RAFT_GRPC_MESSAGE_MAX_SIZE` | The maximum internal raft gRPC message size in bytes. Defaults to 1073741824 | `string - number` | `1073741824` | @@ -192,7 +192,7 @@ For more information on authentication and authorization, see the [Authenticatio :::info Added in `v1.29` The environment variables for configuring async replication have been introduced in `v1.29`. -To learn more about their usage, visit the **[replication how-to guide](/docs/weaviate/configuration/replication#async-replication-settings)**. +To learn more about their usage, visit the **[replication how-to guide](docs/deploy/config-guides/replication.md#async-replication-settings)**. ::: ```mdx-code-block diff --git a/docs/weaviate/config-refs/env-vars/runtime-config.md b/docs/deploy/config-guides/env-vars/runtime-config.md similarity index 96% rename from docs/weaviate/config-refs/env-vars/runtime-config.md rename to docs/deploy/config-guides/env-vars/runtime-config.md index 57e3ffcc..de1bb940 100644 --- a/docs/weaviate/config-refs/env-vars/runtime-config.md +++ b/docs/deploy/config-guides/env-vars/runtime-config.md @@ -73,7 +73,7 @@ If Weaviate attempts to start with an invalid runtime configuration file (e.g., When modifying the runtime configuration file for a running Weaviate instance, if the new configuration is invalid, Weaviate continues using the last valid configuration that is stored in memory. Error logs and metrics will indicate when configuration loading fails. -Weaviate provides the following [metrics to help you monitor](../../configuration/monitoring.md) runtime configuration status: +Weaviate provides the following [metrics to help you monitor](../monitoring.md) runtime configuration status: | Metric Name | Description | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | @@ -85,3 +85,9 @@ Weaviate provides the following [metrics to help you monitor](../../configuratio It's important to set up proper alerting based on these metrics and logs to quickly identify configuration issues. If any Weaviate process is failing to load its runtime configuration, it won't be able to start until the configuration is fixed. ::: + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + \ No newline at end of file diff --git a/docs/deploy/config-guides/horizontal-scaling.mdx b/docs/deploy/config-guides/horizontal-scaling.mdx new file mode 100644 index 00000000..09173050 --- /dev/null +++ b/docs/deploy/config-guides/horizontal-scaling.mdx @@ -0,0 +1,379 @@ +--- +sidebar_label: Horizontal scaling +title: Horizontal Scaling Deployment Strategies +--- + +import FilteredTextBlock from '@site/src/components/Documentation/FilteredTextBlock'; +import TabItem from '@theme/TabItem'; +import Tabs from '@theme/Tabs'; +import PyCode from '!!raw-loader!/_includes/code/howto/manage-data.collections.py'; +import PyCodeV3 from '!!raw-loader!/_includes/code/howto/manage-data.collections-v3.py'; +import TSCode from '!!raw-loader!/_includes/code/howto/manage-data.collections.ts'; +import TSCodeLegacy from '!!raw-loader!/_includes/code/howto/manage-data.collections-v2.ts'; +import JavaCode from '!!raw-loader!/_includes/code/howto/java/src/test/java/io/weaviate/docs/manage-data.classes.java'; +import GoCode from '!!raw-loader!/_includes/code/howto/go/docs/manage-data.classes_test.go'; + + +Weaviate offers two complementary superpowers for scaling your deployment: sharding and replication. +Sharding divides your data across multiple nodes, allowing you to handle datasets far larger than a single machine could process. +Meanwhile, replication creates redundant copies of your data, ensuring high availability even when individual nodes fail or need maintenance. +While each scaling method shines on its own, the true magic happens when they join forces. + +Let's explore how you can harness these capabilities to build a deployment that's both massive in scale and rock-solid in reliability! + +## Scaling Methods + +![Sharding vs Replication](./img/horiz-scaling.png) + + + +### Replication + +Replication creates redundant copies of your data, it is useful when your data needs to be highly available. + +### Sharding + +Sharding divides data across nodes, it is useful when your dataset is too large for just a single node. + + +### Choosing your strategy + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Requirement/GoalShardingReplicationBoth CombinedPrimary Consideration
    + Handle dataset too large for single node + +
    Yes
    +     		+
    No
    +     		+
    Yes
    +     		+
    +

    How much data are you storing?

    +
      +
    • + Vector dimensions and count determine memory requirements +
      • +
    • + Sharding divides this across nodes +
      • +
    +
    +
    + Improve query throughput + +
    Maybe*
    +     		+
    Yes
    +     		+
    Yes
    +     		+
    +

    Is your workload read-heavy?

    +
      +
    • + Replication allows distributing read queries across nodes +
      • +
    • + Sharding may help with certain query patterns +
      • +
    +
    +
    + Accelerate data imports + +
    Yes
    +     		+
    No
    +     		+
    Yes
    +     		+
    +

    Is import speed a priority?

    +
      +
    • + Sharding enables parallel processing of imports +
      • +
    • + Replication adds overhead during imports +
      • +
    +
    +
    + Ensure high availability + +
    No
    +     		+
    Yes
    +     		+
    Yes
    +     		+
    +

    Can you tolerate downtime?

    +
      +
    • + Replication provides redundancy if nodes fail +
      • +
    • + Without replication, shard loss = data loss +
      • +
    +
    +
    + Enable zero-downtime upgrades + +
    No
    +     		+
    Yes
    +     		+
    Yes
    +     		+
    +

    How critical is continuous operation?

    +
      +
    • + Replication allows rolling updates +
      • +
    • + Production systems typically require this capability +
      • +
    +
    +
    + Optimize resource utilization + +
    Yes
    +     		+
    Maybe*
    +     		+
    Maybe*
    +     		+
    +

    Are you resource-constrained?

    +
      +
    • + Sharding distributes load efficiently +
      • +
    • + Replication adds resource overhead +
      • +
    +
    +
    + Geographic distribution + +
    No
    +     		+
    Yes
    +     		+
    Yes
    +     		+
    +

    Do you need multi-region support?

    +
      +
    • + Replicas can be deployed across regions +
      • +
    • + Reduces latency for geographically distributed users +
      • +
    +
    +
    + +**This may serve as a partial solution and will depend on your configuration.* + +### Sharding: Divide and Conquer + +You've made the decision to shard your data, now let's get it configured: + + + + + + + + + + + + + + + + + +#### Parameters + +These parameters are used to configure your collection shards: + +| Parameter | Type | Description | +| :-------------------- | :------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `desiredCount` | integer | *Immutable, Optional*. Controls the target number of physical shards for the collection index. Defaults to the number of nodes in the cluster, but can be explicitly set lower. If set higher than the node count, some nodes will host multiple shards. | +| `virtualPerPhysical` | integer | *Immutable, Optional*. Defines how many virtual shards correspond to one physical shard, defaulting to `128`. Using virtual shards aids in reducing data movement during resharding. | +| `desiredVirtualCount` | integer | *Read-only*. Shows the target total number of virtual shards, calculated as `desiredCount * virtualPerPhysical`. | + + +### Replication: An army of clones + +Configure your data's replication to ensure it's always available: + +import RaftRFChangeWarning from '/_includes/1-25-replication-factor.mdx'; + + + +Configure replication settings, such as [async replication](docs/deploy/config-guides/replication.md#async-replication-settings) and [deletion resolution strategy](docs/weaviate/concepts/replication-architecture/consistency.md#deletion-resolution-strategies). + + + + + + + + + + + + + + + +```bash +curl \ +-X POST \ +-H "Content-Type: application/json" \ +-d '{ + "class": "Article", + "properties": [ + { + "dataType": [ + "string" + ], + "description": "Title of the article", + "name": "title" + } + ], + "replicationConfig": { + "factor": 3, + "asyncEnabled": true, + "deletionStrategy": "TimeBasedResolution" + } +}' \ +http://localhost:8080/v1/schema +``` + + + + + +In a highly available environment, combining sharding and replication leverages the power and +capabilities of both methods to be a dynamic duo that keeps your deployment highly available. +If given the opportunity, those two techniques will be your deployment's dynamic duo. +Specifically using the [`ASYNC_REPLICATION` environment variables](./async-rep.md) introduced in the 1.29 release +will allow you to unleash the full power of horizontal scaling! + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/deploy/config-guides/img/horiz-scaling.png b/docs/deploy/config-guides/img/horiz-scaling.png new file mode 100644 index 00000000..433be744 Binary files /dev/null and b/docs/deploy/config-guides/img/horiz-scaling.png differ diff --git a/docs/deploy/config-guides/img/weaviate-sample-dashboard-async-queue.png b/docs/deploy/config-guides/img/weaviate-sample-dashboard-async-queue.png new file mode 100644 index 00000000..903f8aa6 Binary files /dev/null and b/docs/deploy/config-guides/img/weaviate-sample-dashboard-async-queue.png differ diff --git a/docs/deploy/config-guides/img/weaviate-sample-dashboard-importing.png b/docs/deploy/config-guides/img/weaviate-sample-dashboard-importing.png new file mode 100644 index 00000000..09fceb95 Binary files /dev/null and b/docs/deploy/config-guides/img/weaviate-sample-dashboard-importing.png differ diff --git a/docs/deploy/config-guides/img/weaviate-sample-dashboard-lsm.png b/docs/deploy/config-guides/img/weaviate-sample-dashboard-lsm.png new file mode 100644 index 00000000..7cd3a9fd Binary files /dev/null and b/docs/deploy/config-guides/img/weaviate-sample-dashboard-lsm.png differ diff --git a/docs/deploy/config-guides/img/weaviate-sample-dashboard-objects.png b/docs/deploy/config-guides/img/weaviate-sample-dashboard-objects.png new file mode 100644 index 00000000..4b0c148b Binary files /dev/null and b/docs/deploy/config-guides/img/weaviate-sample-dashboard-objects.png differ diff --git a/docs/deploy/config-guides/img/weaviate-sample-dashboard-startup.png b/docs/deploy/config-guides/img/weaviate-sample-dashboard-startup.png new file mode 100644 index 00000000..aae320f1 Binary files /dev/null and b/docs/deploy/config-guides/img/weaviate-sample-dashboard-startup.png differ diff --git a/docs/deploy/config-guides/img/weaviate-sample-dashboard-usage.png b/docs/deploy/config-guides/img/weaviate-sample-dashboard-usage.png new file mode 100644 index 00000000..09706dca Binary files /dev/null and b/docs/deploy/config-guides/img/weaviate-sample-dashboard-usage.png differ diff --git a/docs/deploy/config-guides/img/weaviate-sample-dashboard-vector.png b/docs/deploy/config-guides/img/weaviate-sample-dashboard-vector.png new file mode 100644 index 00000000..aa3e28a1 Binary files /dev/null and b/docs/deploy/config-guides/img/weaviate-sample-dashboard-vector.png differ diff --git a/docs/deploy/config-guides/index.mdx b/docs/deploy/config-guides/index.mdx new file mode 100644 index 00000000..38e16444 --- /dev/null +++ b/docs/deploy/config-guides/index.mdx @@ -0,0 +1,90 @@ +--- +title: Configuration Guides +sidebar_position: 0 +--- + +Deploying Weaviate successfully depends on robust, well-considered configuration. +This section outlines key configuration requirements and best practices for production deployments. +Whether you are setting up a new environment or optimizing an existing one, configuration choices impact security, scalability, and reliability. + +### Configuring your deployment + +import CardsSection from "/src/components/CardsSection"; + +export const mainReferencesData = [ + + { + title: "Environment variables", + description: "Configure your environment variables for connection strings, ports, and security credentials", + link: "/docs/deploy/config-guides/env-vars", + icon: "fas fa-cog", + }, + { + title: "Scaling", + description: "Scale your deployment for high availability.", + link: "/docs/deploy/config-guides/horizontal-scaling", + icon: "fas fa-arrow-up-right-dots", + }, + { + title: "Persistence", + description: "Enable backups and configure retention policies.", + link: "/docs/deploy/config-guides/persistence", + icon: "fas fa-floppy-disk", + }, + { + title: "Monitoring, metrics, and logging", + description: "Monitor the status and health of your Weaviate instance.", + link: "/docs/deploy/config-guides/monitoring", + icon: "fas fa-chart-bar", + }, + { + title: "Access Management", + description: "Configure access management of your deployment.", + link: "/docs/deploy/config-guides/authentication", + icon: "fas fa-shield-halved", + }, + { + title: "Status", + description: "Monitor the status and health of your Weaviate instance.", + link: "/docs/deploy/config-guides/status", + icon: "fas fa-heartbeat", + }, + { + title: "Telemetry", + description: + "Understand how Weaviate telemetry works and how to configure it.", + link: "/docs/deploy/config-guides/telemetry", + icon: "fas fa-chart-line", + }, + { + title: "Cluster metadata", + description: "Access and manage metadata about your Weaviate cluster.", + link: "/docs/deploy/config-guides/meta", + icon: "fas fa-info-circle", + }, + { + title: "Cluster node data", + description: "View and manage data about individual nodes in your cluster.", + link: "/docs/deploy/config-guides/nodes", + icon: "fas fa-server", + }, + { + title: "OIDC Configuration", + description: + "Configure OpenID Connect for secure authentication with Weaviate.", + link: "/docs/deploy/config-guides/oidc", + icon: "fas fa-lock", + }, +] + +
    + +
    + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + + + diff --git a/docs/weaviate/config-refs/meta.md b/docs/deploy/config-guides/meta.md similarity index 100% rename from docs/weaviate/config-refs/meta.md rename to docs/deploy/config-guides/meta.md diff --git a/docs/deploy/config-guides/monitoring.md b/docs/deploy/config-guides/monitoring.md new file mode 100644 index 00000000..0d11fe89 --- /dev/null +++ b/docs/deploy/config-guides/monitoring.md @@ -0,0 +1,168 @@ +--- +title: Monitoring +image: og/docs/configuration.jpg +# tags: ['configuration', 'operations', 'monitoring', 'observability'] +--- + +Weaviate can expose Prometheus-compatible metrics for monitoring. A standard +Prometheus/Grafana setup can be used to visualize metrics on various +dashboards. + +Metrics can be used to measure request latencies, import +speed, time spent on vector vs object storage, memory usage, application usage, +and more. + +## Configure Monitoring + +### Enable within Weaviate + +To tell Weaviate to collect metrics and expose them in a Prometheus-compatible +format, all that's required is to set the following environment variable: + +```sh +PROMETHEUS_MONITORING_ENABLED=true +``` + +By default, Weaviate will expose the metrics at `:2112/metrics`. You +can optionally change the port to a custom port using the following environment +variable: + +```sh +PROMETHEUS_MONITORING_PORT=3456 +``` + +### Scrape metrics from Weaviate + +Metrics are typically scraped into a time-series database, such as Prometheus. +How you consume metrics depends on your setup and environment. + +The [Weaviate examples repo contains a fully pre-configured setup using +Prometheus, Grafana and some example +dashboards](https://github.com/weaviate/weaviate-examples/tree/main/monitoring-prometheus-grafana). +You can start up a full-setup including monitoring and dashboards with a single +command. In this setup the following components are used: + +* Docker Compose is used to provide a fully-configured setup that can be + started with a single command. +* Weaviate is configured to expose Prometheus metrics as outlined in the + section above. +* A Prometheus instance is started with the setup and configured to scrape + metrics from Weaviate every 15s. +* A Grafana instance is started with the setup and configured to use the + Prometheus instance as a metrics provider. Additionally, it runs a dashboard + provider that contains a few sample dashboards. + +### Multi-tenancy + +When using multi-tenancy, we suggest setting the `PROMETHEUS_MONITORING_GROUP` [environment variable](docs/deploy/config-guides/env-vars/index.md) as `true` so that data across all tenants are grouped together for monitoring. + +## Obtainable Metrics + +The list of metrics that are obtainable through Weaviate's metric system is +constantly being expanded. The complete list is in the [`prometheus.go`](https://github.com/weaviate/weaviate/blob/main/usecases/monitoring/prometheus.go) source code file. + +This page describes some noteworthy metrics and their uses. + +Typically metrics are quite granular, as they can always be aggregated later +on. For example if the granularity is "shard", you could aggregate all "shard" +metrics of the same "class" to obtain a class metrics, or aggregate all metrics +to obtain the metric for the entire Weaviate instance. + +| Metric | Description | Labels | Type | +|---|---|---|---| +| `async_operations_running` | Number of currently running async operations. The operation itself is defined through the `operation` label. | `operation`, `class_name`, `shard_name`, `path` | `Gauge` | +| `batch_delete_durations_ms` | Duration of a batch delete in ms. The `operation` label further defines what operation as part of the batch delete is being measured. Granularity is a shard of a class | `class_name`, `shard_name` | `Histogram` | +| `batch_durations_ms` | Duration of a single batch operation in ms. The `operation` label further defines what operation as part of the batch (e.g. object, inverted, vector) is being used. Granularity is a shard of a class. | `operation`, `class_name`, `shard_name` | `Histogram` | +| `index_queue_delete_duration_ms` | Duration of deleting one or more vectors from the index queue and the underlying index. | `class_name`, `shard_name`, `target_vector` | `Summary` | +| `index_queue_paused` | Whether the index queue is paused. | `class_name`, `shard_name`, `target_vector` | `Gauge` | +| `index_queue_preload_count` | Number of vectors preloaded to the index queue. | `class_name`, `shard_name`, `target_vector` | `Gauge` | +| `index_queue_preload_duration_ms` | Duration of preloading un-indexed vectors to the index queue. | `class_name`, `shard_name`, `target_vector` | `Summary` | +| `index_queue_push_duration_ms` | Duration of pushing one or more vectors to the index queue. | `class_name`, `shard_name`, `target_vector` | `Summary` | +| `index_queue_search_duration_ms` | Duration of searching for vectors in the index queue and the underlying index. | `class_name`, `shard_name`, `target_vector` | `Summary` | +| `index_queue_size` | Number of vectors in the index queue. | `class_name`, `shard_name`, `target_vector` | `Gauge` | +| `index_queue_stale_count` | Number of times the index queue has been marked as stale. | `class_name`, `shard_name`, `target_vector` | `Counter` | +| `index_queue_vectors_dequeued` | Number of vectors sent to the workers per tick. | `class_name`, `shard_name`, `target_vector` | `Gauge` | +| `index_queue_wait_duration_ms` | Duration of waiting for the workers to finish. | `class_name`, `shard_name`, `target_vector` | `Summary` | +| `lsm_active_segments` | Number of currently present segments per shard. Granularity is shard of a class. Grouped by `strategy`. | `strategy`, `class_name`, `shard_name`, `path` | `Gauge` | +| `lsm_bloom_filter_duration_ms` | Duration of a bloom filter operation per shard in ms. Granularity is shard of a class. Grouped by `strategy`. | `operation`, `strategy`, `class_name`, `shard_name` | `Histogram` | +| `lsm_segment_count` | Number of segments by level | `strategy`, `class_name`, `shard_name`, `path`, `level` | `Gauge` | +| `lsm_segment_objects` | Number of entries per LSM segment by level. Granularity is shard of a class. Grouped by `strategy` and `level`. | `operation`, `strategy`, `class_name`, `shard_name`, `path`, `level` | `Gauge` | +| `lsm_segment_size` | Size of LSM segment by level and unit. | `strategy`, `class_name`, `shard_name`, `path`, `level`, `unit` | `Gauge` | +| `object_count` | Numbers of objects present. Granularity is a shard of a class | `class_name`, `shard_name` | `Gauge` | +| `objects_durations_ms` | Duration of an individual object operation, such as `put`, `delete`, etc. as indicated by the `operation` label, also as part of a batch. The `step` label adds additional precisions to each `operation`. Granularity is a shard of a class. | `class_name`, `shard_name` | `Histogram` | +| `requests_total` | Metric that tracks all user requests to determine if it was successful or failed. | `api`, `query_type`, `class_name` | `GaugeVec` | +| `startup_diskio_throughput` | Disk I/O throughput in bytes/s at startup operations, such as reading back the HNSW index or recovering LSM segments. The operation itself is defined by the `operation` label. | `operation`, `step`, `class_name`, `shard_name` | `Histogram` | +| `startup_durations_ms` | Duration of individual startup operations in ms. The operation itself is defined through the `operation` label. | `operation`, `class_name`, `shard_name` | `Histogram` | +| `vector_index_durations_ms` | Duration of regular vector index operation, such as insert or delete. The operation itself is defined through the `operation` label. The `step` label adds more granularity to each operation. | `operation`, `step`, `class_name`, `shard_name` | `Histogram` | +| `vector_index_maintenance_durations_ms` | Duration of a sync or async vector index maintenance operation. The operation itself is defined through the `operation` label. | `opeartion`, `class_name`, `shard_name` | `Histogram` | +| `vector_index_operations` | Total number of mutating operations on the vector index. The operation itself is defined by the `operation` label. | `operation`, `class_name`, `shard_name` | `Gauge` | +| `vector_index_size` | The total capacity of the vector index. Typically larger than the number of vectors imported as it grows proactively. | `class_name`, `shard_name` | `Gauge` | +| `vector_index_tombstone_cleaned` | Total number of deleted and removed vectors after repair operations. | `class_name`, `shard_name` | `Counter` | +| `vector_index_tombstone_cleanup_threads` | Number of currently active threads for repairing/cleaning up the vector index after deletes have occurred. | `class_name`, `shard_name` | `Gauge` | +| `vector_index_tombstones` | Number of currently active tombstones in the vector index. Will go up on each incoming delete and go down after a completed repair operation. | `class_name`, `shard_name` | `Gauge` | +| `weaviate_build_info` | Provides general information about the build (What version is currently running? How long has this version been running, etc) | `version`, `revision`, `branch`, `goVersion` | | +| `weaviate_runtime_config_hash` | Hash value of the currently active runtime configuration, useful for tracking when new configurations take effect. | `sha256` | `GaugeVec` | +| `weaviate_runtime_config_last_load_success` | Indicates whether the last loading attempt was successful (`1` for success, `0` for failure). | | `Gauge` | +| `weaviate_schema_collections` | Shows the total number of collections at any given point. | | | +| `weaviate_schema_shards` | Shows the total number of shards at any given point. | `status(HOT, COLD, WARM, FROZEN)` | | +| `weaviate_internal_sample_memberlist_queue_broadcasts` | Shows the number of messages in the broadcast queue of Memberlist. | `quantile=0.5, 0.9, 0.99` | `Summary` | +| `weaviate_internal_timer_memberlist_gossip` | Shows the latency distribution of the each gossip made in Memberlist. | `quantile=0.5, 0.9, 0.99` | `Summary` | +| `weaviate_internal_counter_raft_apply` | Number of transactions in the configured interval. | | `counter` | +| `weaviate_internal_counter_raft_state_candidate` | Number of times the raft server initiated an election. | | `counter` | +| `weaviate_internal_counter_raft_state_follower` | Number of times in the configured interval that the raft server became a follower. | | `summary` | +| `weaviate_internal_counter_raft_state_leader` | Number of times the raft server became a leader. | | `counter` | +| `weaviate_internal_counter_raft_transition_heartbeat_timeout` | Number of times that the node transitioned to `candidate` state after not receiving a heartbeat message from the last known leader. | | `summary` | +| `weaviate_internal_gauge_raft_commitNumLogs` | Number of logs processed for application to the finite state machine in a single batch. | | `gauge` | +| `weaviate_internal_gauge_raft_leader_dispatchNumLogs` | Number of logs committed to disk in the most recent batch. | | `gauge` | +| `weaviate_internal_gauge_raft_leader_oldestLogAge` | | | `gauge` | +| `weaviate_internal_gauge_raft_peers` | The number of peers in the raft cluster configuration. | | `gauge` | +| `weaviate_internal_sample_raft_boltdb_logBatchSize` | | | | +| `weaviate_internal_sample_raft_boltdb_logSize` | | | | +| `weaviate_internal_sample_raft_boltdb_logsPerBatch` | | | | +| `weaviate_internal_sample_raft_boltdb_writeCapacity` | | | | +| `weaviate_internal_sample_raft_thread_fsm_saturation` | | | | +| `weaviate_internal_sample_raft_thread_main_saturation` | | | | +| `weaviate_internal_timer_raft_boltdb_getLog` | | | | +| `weaviate_internal_timer_raft_boltdb_storeLogs` | Time required to record any outstanding logs since the last request to append entries for the given node. | | `timer` | +| `weaviate_internal_timer_raft_commitTime` | Time required to commit a new entry to the raft log on the leader node. | | `summary` | +| `weaviate_internal_timer_raft_fsm_apply` | Number of logs committed by the finite state machine since the last interval. | | `summary` | +| `weaviate_internal_timer_raft_fsm_enqueue` | Time required to queue up a batch of logs for the finite state machine to apply. | | `summary` | +| `weaviate_internal_timer_raft_leader_dispatchLog` | Time required for the leader node to write a log entry to disk. | | `timer` | + +Extending Weaviate with new metrics is very easy. To suggest a new metric, see the [contributor guide](/docs/contributor-guide). + +### Versioning + +Be aware that metrics do not follow the semantic versioning guidelines of other Weaviate features. Weaviate's main APIs are stable and breaking changes are extremely rare. Metrics, however, have shorter feature lifecycles. It can sometimes be necessary to introduce an incompatible change or entirely remove a metric, for example, because the cost of observing a specific metric in production has grown too high. As a result, it is possible that a Weaviate minor release contains a breaking change for the Monitoring system. If so, it will be clearly highlighted in the release notes. + +## Sample Dashboards + +Weaviate does not ship with any dashboards by default, but here is a list of +dashboards being used by the various Weaviate teams, both during development, +and when helping users. These do not come with any support, but may still be +helpful. Treat them as inspiration to design your own dashboards which fit +your uses perfectly: + +| Dashboard | Purpose | Preview | +| --- | --- | --- | +| [Importing Data Into Weaviate](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/importing.json) | Visualize speed of import operations (including its components, such as object store, inverted index, and vector index). | ![Importing Data into Weaviate](./img/weaviate-sample-dashboard-importing.png "Importing Data Into Weaviate") | +| [Object Operations](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/objects.json) | Visualize speed of whole object operations, such as GET, PUT, etc. | ![Objects](./img/weaviate-sample-dashboard-objects.png "Objects") | +| [Vector Index](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/vectorindex.json) | Visualize the current state, as well as operations on the HNSW vector index | ![Vector Index](./img/weaviate-sample-dashboard-vector.png "Vector Index") | +| [LSM Stores](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/lsm.json) | Get insights into the internals (including segments) of the various LSM stores within Weaviate. | ![LSM Store](./img/weaviate-sample-dashboard-lsm.png "LSM Store") | +| [Startup](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/startup.json) | Visualize the startup process, including recovery operations | ![Startup](./img/weaviate-sample-dashboard-startup.png "Vector Index") | +| [Usage](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/usage.json) | Obtain usage metrics, such as number of objects imported, etc.| ![Usage](./img/weaviate-sample-dashboard-usage.png "Usage") | +| [Aysnc index queue](https://github.com/weaviate/weaviate/blob/main/tools/dev/grafana/dashboards/index_queue.json) | Observe index queue activity | ![Async index queue](./img/weaviate-sample-dashboard-async-queue.png "Async index queue") | + +## `nodes` API Endpoint + +To get collection details programmatically, use the [`nodes`](docs/deploy/config-guides/nodes.md) REST endpoint. + +import APIOutputs from '/_includes/rest/node-endpoint-info.mdx'; + + + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/weaviate/config-refs/nodes.md b/docs/deploy/config-guides/nodes.md similarity index 98% rename from docs/weaviate/config-refs/nodes.md rename to docs/deploy/config-guides/nodes.md index ff4ea134..a4dc1205 100644 --- a/docs/weaviate/config-refs/nodes.md +++ b/docs/deploy/config-guides/nodes.md @@ -1,6 +1,5 @@ --- title: Cluster node data -sidebar_position: 93 image: og/docs/configuration.jpg # tags: ['nodes', 'reference', 'configuration'] --- diff --git a/docs/weaviate/config-refs/oidc.md b/docs/deploy/config-guides/oidc.md similarity index 100% rename from docs/weaviate/config-refs/oidc.md rename to docs/deploy/config-guides/oidc.md diff --git a/docs/weaviate/configuration/persistence.md b/docs/deploy/config-guides/persistence.md similarity index 95% rename from docs/weaviate/configuration/persistence.md rename to docs/deploy/config-guides/persistence.md index 7ded1689..a1e023c3 100644 --- a/docs/weaviate/configuration/persistence.md +++ b/docs/deploy/config-guides/persistence.md @@ -1,6 +1,5 @@ --- title: Persistence -sidebar_position: 45 image: og/docs/configuration.jpg --- @@ -76,7 +75,7 @@ See [Backups](./backups.md). ## Kubernetes -For Kubernetes setup, the only thing to bear in mind is that Weaviate needs a `PersistentVolumes` through `PersistentVolumeClaims` ([more info](/docs/weaviate/installation/kubernetes.md#requirements)) but the Helm chart is already configured to store the data on an external volume. +For Kubernetes setup, the only thing to bear in mind is that Weaviate needs a `PersistentVolumes` through `PersistentVolumeClaims` ([more info](../installation-guides/k8s-installation.md#requirements)) but the Helm chart is already configured to store the data on an external volume. ## Disk Pressure Warnings and Limits @@ -104,7 +103,7 @@ In general, `mmap` may be a preferred option with memory management benefits. Ho ## Related pages -- [Configuration: Backups](./backups.md) +- [Configuration: Backups](docs/deploy/config-guides/backups.md) ## Questions and feedback diff --git a/docs/weaviate/configuration/replication.md b/docs/deploy/config-guides/replication.md similarity index 62% rename from docs/weaviate/configuration/replication.md rename to docs/deploy/config-guides/replication.md index 6e6c514f..a3091817 100644 --- a/docs/weaviate/configuration/replication.md +++ b/docs/deploy/config-guides/replication.md @@ -1,13 +1,12 @@ --- title: Replication -sidebar_position: 60 image: og/docs/configuration.jpg # tags: ['configuration', 'operations', 'monitoring', 'observability'] --- Weaviate instances can be replicated. Replication can improve read throughput, improve availability, and enable zero-downtime upgrades. -For more details on how replication is designed and built in Weaviate, see [Replication Architecture](../concepts/replication-architecture/index.md). +For more details on how replication is designed and built in Weaviate, see [Replication Architecture](docs/weaviate/concepts/replication-architecture/index.md). ## How to configure @@ -15,7 +14,7 @@ import RaftRFChangeWarning from '/\_includes/1-25-replication-factor.mdx'; -Replication is disabled by default. It can be enabled per collection in the [collection configuration](../manage-collections/multi-node-setup.mdx#replication-settings). This means you can set different replication factors per class in your dataset. +Replication is disabled by default. It can be enabled per collection in the [collection configuration](docs/weaviate/manage-collections/multi-node-setup.mdx#replication-settings). This means you can set different replication factors per class in your dataset. To enable replication, you can set one or both of the following: @@ -26,25 +25,14 @@ To enable replication, you can set one or both of the following: The `REPLICATION_MINIMUM_FACTOR` environment variable sets the minimum replication factor for all collections in the Weaviate instance. -If you set the [replication factor for a collection](#replication-factor-for-a-collection), the collection's replication factor overrides the minimum replication factor. +If you set the [replication factor for a collection](docs/deploy/config-guides/persistence.md#replication-factor-for-a-collection), the collection's replication factor overrides the minimum replication factor. -### Replication factor for a collection - -import SchemaReplication from '/\_includes/code/schema.things.create.replication.mdx'; - - - -In this example, there are three replicas. If you set the replication factor before you import data, all of the data is replicated three times. - -The replication factor can be modified after you add data to a collection. If you modify the replication factor afterwards, new data is copied across the new and pre-existing replica nodes. - -The example data schema has a [write consistency](../concepts/replication-architecture/consistency.md#tunable-write-consistency) level of `ALL`. When you upload or update a schema, the changes are sent to `ALL` nodes (via a coordinator node). The coordinator node waits for a successful acknowledgment from `ALL` nodes before sending a success message back to the client. This ensures a highly consistent schema in your distributed Weaviate setup. ## Data consistency When Weaviate detects inconsistent data across nodes, it attempts to repair the out of sync data. -Starting in v1.26, Weaviate adds [async replication](../concepts/replication-architecture/consistency.md#async-replication) to proactively detect inconsistencies. In earlier versions, Weaviate uses a [repair-on-read](../concepts/replication-architecture/consistency.md#repair-on-read) strategy to repair inconsistencies at read time. +Starting in v1.26, Weaviate adds [async replication](docs/weaviate/concepts/replication-architecture/consistency.md#async-replication) to proactively detect inconsistencies. In earlier versions, Weaviate uses a [repair-on-read](docs/weaviate/concepts/replication-architecture/consistency.md#repair-on-read) strategy to repair inconsistencies at read time. Repair-on-read is automatic. To activate async replication, set `asyncEnabled` to true in the `replicationConfig` section of your collection definition. @@ -55,12 +43,12 @@ import ReplicationConfigWithAsyncRepair from '/\_includes/code/configuration/rep ### Configure async replication settings {#async-replication-settings} :::info Added in `v1.29` -The [environment variables](/docs/weaviate/config-refs/env-vars#async-replication) for configuring async replication (`ASYNC_*`) have been introduced in `v1.29`. +The [environment variables](/docs/deploy/config-guides/env-vars/index.md#async-replication) for configuring async replication (`ASYNC_*`) have been introduced in `v1.29`. ::: Async replication helps achieve consistency for data replicated across multiple nodes. -Update the following [environment variables](/docs/weaviate/config-refs/env-vars#async-replication) to configure async replication for your particular use case. +Update the following [environment variables](/docs/deploy/config-guides/env-vars/index.md#async-replication) to configure async replication for your particular use case. #### Logging @@ -76,7 +64,7 @@ Update the following [environment variables](/docs/weaviate/config-refs/env-vars - **Monitor node availability:** `ASYNC_REPLICATION_ALIVE_NODES_CHECKING_FREQUENCY` Trigger comparisons whenever there’s a change in node availability. - **Configure hash tree height:** `ASYNC_REPLICATION_HASHTREE_HEIGHT` - Specify the size of the hash tree, which helps narrow down data differences by comparing hash digests at multiple levels instead of scanning entire datasets. See [this page](../concepts/replication-architecture/consistency.md#memory-and-performance-considerations-for-async-replication) for more information on the memory and performance considerations for async replication. + Specify the size of the hash tree, which helps narrow down data differences by comparing hash digests at multiple levels instead of scanning entire datasets. See [this page](docs/weaviate/concepts/replication-architecture/consistency.md#memory-and-performance-considerations-for-async-replication) for more information on the memory and performance considerations for async replication. - **Batch size for digest comparison:** `ASYNC_REPLICATION_DIFF_BATCH_SIZE` Define the number of objects whose digest (e.g., last update time) is compared between nodes before propagating actual objects. @@ -103,7 +91,7 @@ Tweak these settings based on your cluster size and network latency to achieve o ## How to use: Queries -When you add (write) or query (read) data, one or more replica nodes in the cluster will respond to the request. How many nodes need to send a successful response and acknowledgment to the coordinator node depends on the `consistency_level`. Available [consistency levels](../concepts/replication-architecture/consistency.md) are `ONE`, `QUORUM` (replication_factor / 2 + 1) and `ALL`. +When you add (write) or query (read) data, one or more replica nodes in the cluster will respond to the request. How many nodes need to send a successful response and acknowledgment to the coordinator node depends on the `consistency_level`. Available [consistency levels](docs/weaviate/concepts/replication-architecture/consistency.md) are `ONE`, `QUORUM` (replication_factor / 2 + 1) and `ALL`. The `consistency_level` can be specified at query time: @@ -113,7 +101,7 @@ curl "http://localhost:8080/v1/objects/{ClassName}/{id}?consistency_level=ONE" ``` :::note -In v1.17, only [read queries that get data by ID](../manage-objects/read.mdx#get-an-object-by-id) had a tunable consistency level. All other object-specific REST endpoints (read or write) used the consistency level `ALL`. Starting with v1.18, all write and read queries are tunable to either `ONE`, `QUORUM` (default) or `ALL`. GraphQL endpoints use the consistency level `ONE` (in both versions). +In v1.17, only [read queries that get data by ID](docs/weaviate/manage-objects/read.mdx#get-an-object-by-id) had a tunable consistency level. All other object-specific REST endpoints (read or write) used the consistency level `ALL`. Starting with v1.18, all write and read queries are tunable to either `ONE`, `QUORUM` (default) or `ALL`. GraphQL endpoints use the consistency level `ONE` (in both versions). ::: import QueryReplication from '/\_includes/code/replication.get.object.by.id.mdx'; @@ -122,7 +110,8 @@ import QueryReplication from '/\_includes/code/replication.get.object.by.id.mdx' ## Related pages -- [Concepts: Replication Architecture](../concepts/replication-architecture/index.md) +- [Concepts: Replication Architecture](docs/weaviate/concepts/replication-architecture/index.md) +- [Configurinfg Async Replication](./async-rep.md) ## Questions and feedback diff --git a/docs/weaviate/config-refs/status.md b/docs/deploy/config-guides/status.md similarity index 100% rename from docs/weaviate/config-refs/status.md rename to docs/deploy/config-guides/status.md diff --git a/docs/weaviate/config-refs/telemetry.md b/docs/deploy/config-guides/telemetry.md similarity index 93% rename from docs/weaviate/config-refs/telemetry.md rename to docs/deploy/config-guides/telemetry.md index 2512a0e7..4bc2b65b 100644 --- a/docs/weaviate/config-refs/telemetry.md +++ b/docs/deploy/config-guides/telemetry.md @@ -22,7 +22,7 @@ Weaviate does not collect any other telemetry information. ## Disabling Telemetry Data -To disable telemetry data collection, add this line to your [system configuration](./env-vars/index.md) file: +To disable telemetry data collection, add this line to your [system configuration](docs/deploy/config-guides/env-vars/index.md) file: ```bash DISABLE_TELEMETRY=true diff --git a/docs/deploy/faqs/index.md b/docs/deploy/faqs/index.md new file mode 100644 index 00000000..e2fc4e80 --- /dev/null +++ b/docs/deploy/faqs/index.md @@ -0,0 +1,106 @@ +--- + +title: Deployment FAQs + +--- + +Looking for guidance on deploying Weaviate in a production environment? Whether you're scaling up from a proof-of-concept to an enterprise-wide implementation, migrating from another solution, or optimizing for specific workloads, you'll find practical answers to streamline your deployment process. Our goal is to help you achieve reliable performance, maintain data integrity, and minimize operational overhead as you harness the power of vector search for your applications! + + +#### Q1: Why did my cluster suddenly become read-only? + +
    + + Answer + +This almost always means you've run out of disk space. Weaviate protects itself by switching to read-only mode when disk usage exceeds a configured threshold. You'll need to increase the disk size available to Weaviate and then reset the read-only status. + +
    + +#### Q2: How do I deploy Weaviate using AWS Marketplace? + +
    + + Answer + +This [page](../installation-guides/aws-installation.md) will walk you through all the necessary steps to deploy Weaviate using AWS marketplace. + +
    + +#### Q3: How do I deploy Weaviate using GCP Marketplace? + +
    + + Answer + +This [page](../installation-guides/gcp-installation.md) will walk you through all the necessary steps to deploy Weaviate using GCP marketplace. + +
    + +#### Q4: Is there a recommended limit for collections? + +
    + + Answer + +If you plan to create more than **20** collections, you may want to consider multi-tenancy instead for better scaling and performance. + +**Additional information:** [Scaling limits with collections](docs/weaviate/starter-guides/managing-collections/collections-scaling-limits.mdx) + +
    + +#### Q5: What are some common issues that occur during deployment? + +
    + + Answer + +Common issues that occur during deployment include: + +- The cluster becoming `read-only`. +- Query results being inconsistent. +- Nodes unable to maintain consensus. +- Creating too many collections. + +#### Further resources + +For further information, the [troubleshooting page](./troubleshooting.md) will help with some of the common issue encountered. + +
    + +#### Q6: What's the difference between Weaviate and other databases? + +
    + + Answer + +Weaviate has complex processes which means that ingestion and deletion require more steps than other types of databases. Data ingestion tends to take longer than a traditional database because of vectorization and deleting objects can be expensive due to the embedding costs. +
    + +#### Q7: Do my resources free up immediately after deleting objects? + +
    + + Answer + +No, it does not happen instantaneously. When you delete objects a tombstone is created. The data removal and index cleanup occur as background processes. + +
    + +#### Q8: What's the difference between client timeouts and module timeouts? + +
    + + Answer + +- **Client timeout:** These are timeouts between the client and Weaviate server. + +- **Module timeout:** These are timeouts that occurs when Weaviate interacts with external modules like LLMs and vectorizers. + +
    + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + \ No newline at end of file diff --git a/docs/deploy/faqs/troubleshooting.md b/docs/deploy/faqs/troubleshooting.md new file mode 100644 index 00000000..239a691a --- /dev/null +++ b/docs/deploy/faqs/troubleshooting.md @@ -0,0 +1,85 @@ +--- + +title: Deployment Troubleshooting Guide + +--- + +So you've deployed Weaviate and you're fully immersed in the world of vectors when suddenly you encounter a puzzling mystery. This page will serve as your handbook for when things go awry in "Vector Land!" + +Consider every error message a clue to solving the mystery you're encountering. The [LOG_LEVEL](/deploy/config-guides/env-vars#LOG_LEVEL) environment variable helps you to solve any mysteries you encounter. The various levels of logging will allow you to right-size the precise amount of information you need to solve any Vector Land mysteries. + + + +## Common issues and solutions + +### The cluster is not accepting new information and there are disk space or `read-only` error messages in the logs. + +
    + +Answer + +#### Identifying the issue +As a first step, you'll want to examine your cluster's logs to identify the problem. If after checking the logs of your cluster you see error messages that include phrases like "read-only" or "disk space," then your cluster is more than likely in a `read-only` state due to insufficient disk space. + +#### Resolving the issue + +To solve this mystery, you'll need to increase the available disk space for your nodes. Once the disk space is increased, then you'll need to manually mark the affected shards or collections as writeable again. +You can also set the [`MEMORY_WARNING_PERCENTAGE`](docs/deploy/config-guides/env-vars/index.md#MEMORY_WARNING_PERCENTAGE) environment variable to issue warnings when the memory limit is near. +
    + +### You're receiving inconsistent query results. + +
    + + Answer + +#### Identifying the issue + +To confirm and identify the issue, you'll want to first run the same query multiple times to confirm that the results are inconsistent. If the inconsistent results are persisting, then you probably have asynchronous replication disabled for your deployment. + +#### Resolving the issue + +Check your settings to check if you have asynchronous replication enabled. If `async_replication_disabled` is set to "true" then you'll need to set that variable to "false." Once it is enabled, the logs will show messages that indicate successful peers checks and synchronization for the nodes. +
    + +### Your nodes won't communicate, join a cluster, or maintain consensus. + +
    + + Answer + +#### Identifying the issue + +To confirm and identify the issue, you'll want to first run the same query multiple times to confirm that the results are inconsistent. If the inconsistent results are persisting, then you probably have asynchronous replication disabled for your deployment. + +#### Resolving the issue + +Check your settings to check if you have asynchronous replication enabled. If `async_replication_disabled` is set to "true" then you'll need to set that variable to "false." Once it is enabled, the logs will show messages that indicate successful peers checks and synchronization for the nodes. Additionally, test the [live and ready REST endpoints](/docs/weaviate/api/rest#tag/well-known/GET/.well-known/live) and check the network configuration of the nodes. +
    + +### You've downgraded and now your clusters won't reach the `Ready` state. + +
    + + Answer + +#### Identifying the issue + +If you have a multi-node instance running `1.28.13+`, `1.29.5+`, or `1.30.2+` and have downgraded to a `v1.27.x` version earlier than `1.27.26`. + +#### Resolving the issue + +If you need to downgrade Weaviate to `v1.27.x`, use `1.27.26` or higher. + +- [Migration guides](../migration/index.md) + +
    + + +As you continue your adventures in Vector Land, remember that even the most seasoned vector detectives encounter mysterious cases from time to time. Behind every error message lies not just a problem, but the clue you need to run Weaviate in its most optimal form! + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + \ No newline at end of file diff --git a/docs/deploy/index.mdx b/docs/deploy/index.mdx index 37231681..d88d5afc 100644 --- a/docs/deploy/index.mdx +++ b/docs/deploy/index.mdx @@ -1,52 +1,98 @@ --- -title: Deployment overview +title: Overview description: Deployment overview page sidebar_position: 0 --- -# Tacetve a gemit quisquam praestat meruisse gradientis +import CardsSection from "/src/components/CardsSection"; +import DeploymentCards from "/src/components/DeploymentCards"; +import styles from "/src/components/CardsSection/styles.module.scss"; -## Nec sum vagatur nupta tantus +# Deploying Weaviate -Lorem markdownum pugnando ventoque et ducta et et adurat locus inde cervos -refeci Phaethon clivum linguaque Salmacis geras coeperat mollia. Leucothoen Ceyx -est viret nostraque; meum utque latens -quae, patrios plebe inpositum: torvum. **Stipite fluminaque videri**; ictu -duabus sua *condita monstrum*. Imo undis: aut niveo sidoniae; placato sorores -at, quae perque. *Candore dolore*, et secum rastra ceratis et sollicitat vincula -matri mei alios in rigore murmur foedataque, cum. +:::info[Section under construction!] -Et solvit ex comitante super luctus. Unda posse perque venitque quod algae -lymphis postquam optime. +This is a new section being created, please bear with us as we add in new content! -## Flammis levi harena illi membra caudam +::: -Ipse inducitur stridore Aiacem intonsum! Est arma, hausitque *tu* litora -aetasque, at fuit Aegea vides. Non laborum longi fortisque *putes bracchia* -umbras te fingit enim manum! +Weaviate offers multiple deployment options to satisfy your specific use case in production. -Ingentia sive Idaeo serpere, dives inpia. Oculos in aras ergo bracchiaque vera, -omnis solutus vellet: et. Est et putat nec obvia unda -totidem amisit *coniurataeque sic arabat* mea. Caune -quodque **addita** truncis quid latebra: incubat pallorque turribus et ignarus -signum pectore quicquid Abantiades locus. **Ante** aliqua Aiacis deferre, ferunt -hunc omnia, et hic Aconteus nec. +This section hosts common deployment topics, including Kubernetes, cloud providers, and best practices, along with detailed tutorials and how-to guides. +Weaviate is designed for: -Odit festas dixit et quas fidissima, *est umbrae matrona* palude. Ait et quod -pectora circa. +- **Scalability** – Handle billions of vector data points efficiently. +- **High-Performance Search** – Power AI applications with real-time vector retrieval. +- **Flexible Integrations** – Connect with various machine learning models and data sources. +- **Cloud & On-Prem Deployment** – Deploy on Weaviate Cloud, Kubernetes, or managed cloud services. -## Elegit sim ulla poteramus stridente cessit auras +## Deployment Options -Tanta *Elinque*. Mea quae cultum Palameden Aglauros si sacra miserere quam ego, -**e scires** imo sonitusque tumidarum fremit quae ictu? Tum est sub vate vasto, -Galanthida in potui huc et temptat Cenaeo praeterque septem caede. +Choose the best deployment method based on your needs: -> Suo ipse cubitique et vento in onerosa adunco, non. **In paelice Quirites** -> sunt stant sinebat superstes differt non per potentia illiusque nostris -> Hecateidos? Tempore haud imagine facta ora, uni quam fatorum. +export const deploymentCardsData = [ + { + tabs: [ + { label: "Evaluation", active: true }, + { label: "Deployment", active: true }, + { label: "Production", active: true }, + ], + header: "Weaviate Cloud", + bgImage: "/img/site/hex-weaviate.svg", + bgImageLight: "/img/site/hex-weaviate-light.svg", + listItems: [ + "From evaluation (sandbox) to production", + "Serverless (infrastructure managed by Weaviate)", + "(Optional) Data replication (high-availability)", + "(Optional) Zero-downtime updates", + ], + button: { + text: "Set up a WCD instance", + link: "/docs/cloud/manage-clusters/create", + }, + }, + { + tabs: [ + { label: "Evaluation", active: true }, + { label: "Deployment", active: true }, + { label: "Production", active: false }, + ], + header: "Docker", + bgImage: "/img/site/docker-doc-icon.svg", + bgImageLight: "/img/site/docker-doc-icon-light.svg", + listItems: [ + "For local evaluation & development", + "Local inference containers", + "Multi-modal models", + "Customizable configurations", + ], + button: { + text: "Deploy Weaviate with Docker", + link: "/docs/deploy/installation-guides/docker-installation", + }, + }, + { + tabs: [ + { label: "Evaluation", active: false }, + { label: "Deployment", active: true }, + { label: "Production", active: true }, + ], + header: "Kubernetes", + bgImage: "/img/site/kubernetes-doc-icon.svg", + bgImageLight: "/img/site/kubernetes-doc-icon-light.svg", + listItems: [ + "From development to production", + "Local inference containers", + "Multi-modal models", + "Customizable configurations", + "Self-deploy or Marketplace deployment", + "(Optional) Zero-downtime updates", + ], + button: { + text: "Deploy Weaviate with Kubernetes", + link: "/docs/deploy/installation-guides/k8s-installation", + }, + }, +]; -Altis animalibus pampinus venit mirabilis busto: tamen hunc, **salutant** -Stheneleia vultibus pectore. Evocat posuere anxia. - -Aper est; in tuum medius, responderat tamen: magis vix! Amor regno tamen tuli -remige facitote **reverentia**: loqui vade pignora. + \ No newline at end of file diff --git a/docs/deploy/installation-guides/aws-cli.md b/docs/deploy/installation-guides/aws-cli.md new file mode 100644 index 00000000..fda3564b --- /dev/null +++ b/docs/deploy/installation-guides/aws-cli.md @@ -0,0 +1,206 @@ +--- +title: EKS Installation +description: Deploy Weaviate on EKS using the AWS CLI +--- + +Weaviate can be deployed on an EKS cluster using the `eksctl` command-line tool that creates and manages clusters. By the end of this document, you'll have all the necessary information to create an EKS cluster using the command line, add persistent storage to your cluster and then deploy Weaviate onto the cluster. + +:::info Prerequisites + +- Helm installed +- The AWS CLI installed with the latest version +- `kubectl` installed +- `eksctl` installed +::: + +
    + AWS policies needed + +Ensure that you have adequate permissions to create and interact wth an EKS cluster. The following policies should provide you the adequate permissions to create your cluster: + +- eks:CreateCluster +- eks:DescribeCluster +- eks:ListClusters +- eks:UpdateClusterConfig +- eks:DeleteCluster +- iam:CreateRole +- iam:AttachRolePolicy +- iam:PutRolePolicy +- iam:GetRole +- iam:ListRolePolicies +- iam:ListAttachedRolePolicies +- ec2:DescribeSubnets +- ec2:DescribeVpcs +- ec2:DescribeSecurityGroups +- ec2:CreateSecurityGroup +- ec2:AuthorizeSecurityGroupIngress +- ec2:RevokeSecurityGroupIngress +- cloudformation:CreateStack +- cloudformation:DescribeStacks +- cloudformation:UpdateStack +- cloudformation:DeleteStack +- ec2:CreateTags +- ec2:DescribeInstances +- ec2:DescribeNetworkInterfaces +- ec2:DescribeAvailabilityZones + +
    + +#### Verify your tools + +Before starting, ensure that your tools are installed: + +```bash +helm version +aws --version +kubectl version +eksctl version +``` + + + +### Step 1: Create the Cluster + +To create your cluster, prepare a `yaml` file that with a name of your choosing (e.g. `eks-cluster.yaml`) + +```yaml +apiVersion: eksctl.io/v1alpha5 +kind: ClusterConfig +metadata: + name: + region: + version: "1.31" + +managedNodeGroups: + - name: node-group-name + labels: { role: worker } + instanceType: t3.large # Choose your instance type + desiredCapacity: 3 # Number of nodes + minSize: 2 # Minimum number of nodes for autoscaling + maxSize: 5 # Maximum number of nodes for autoscaling + privateNetworking: true # Use private networking + volumeSize: 80 # Root volume size in GB + volumeType: gp3 # Root volume type + +addons: + - name: vpc-cni + version: latest + attachPolicyARNs: + - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy + - name: coredns + version: latest + - name: kube-proxy + version: latest + - name: aws-ebs-csi-driver + version: latest + wellKnownPolicies: + ebsCSIController: true # Enable EBS CSI driver +``` + +This creates an EKS cluster within your specified region with an autoscaling node group. There are 3 nodes for high availability and having autoscaling enabled allows for the cluster to dynamically adjust resources based on demand. + +#### Run this command to create your EKS cluster: + +```bash +eksctl create cluster -f +``` + +#### Enable `kubectl` to interact with the newly created cluster: + +```bash +aws eks --region update-kubeconfig --name +``` + +#### Verify that the cluster has been created and that you are able to interact with it: + +```bash +kubectl get nodes +``` + +### Step 2: Add Storage Class + +After creating your cluster and verifying that you can interact with it , you'll need to create a `storageclass.yaml` file: +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: +provisioner: ebs.csi.aws.com +parameters: + type: gp3 + encrypted: "true" +reclaimPolicy: Retain +volumeBindingMode: Immediate +allowVolumeExpansion: true +``` + +After creating the storage class, apply it: +```bash +kubectl apply -f .yaml +``` + + +#### Verify your storage class and has been created and applied + +```bash +kubectl get sc +``` + +### Step 3: Add Weaviate to EKS + +After adding persistent storage to your cluster, you can now deploy Weaviate into it. + +#### Create a Weaviate namespace: + +```bash +kubectl create namespace weaviate +``` + +#### Add the Weaviate Helm chart: + +```bash +helm repo add weaviate https://weaviate.github.io/weaviate-helm +helm repo update +``` + +After you've added the Weaviate Helm chart, configure the `values.yaml` file before you deploy Weaviate on the cluster. + +```bash +helm show values weaviate/weaviate > values.yaml +``` + +Before deploying Weaviate, change the `storgeclass` and ensure that you have replicas specified in your `values.yaml` file. + +```yaml +storage: + size: 32Gi + storageClassName: "" +``` + +```yaml +replicas: 3 +``` + +#### Deploy Weaviate on your cluster: + +```bash +helm upgrade --install weaviate weaviate/weaviate \ + --namespace weaviate \ + --values values.yaml \ +``` + +#### Verify your deployment + +```bash +kubectl get pods -n weaviate +``` + +## Further Resources + +- [Persistent storage for Kubernetes](https://aws.amazon.com/blogs/storage/persistent-storage-for-kubernetes/) + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/deploy/installation-guides/aws-installation.md b/docs/deploy/installation-guides/aws-installation.md new file mode 100644 index 00000000..8af042d5 --- /dev/null +++ b/docs/deploy/installation-guides/aws-installation.md @@ -0,0 +1,65 @@ +--- +title: AWS Marketplace +description: Install Weaviate through the AWS Marketplace for quick cloud deployment. +image: og/docs/installation.jpg +# tags: ['installation', 'AWS Marketplace'] +--- + +import ReactPlayer from 'react-player/lazy' + + + +Through the AWS Marketplace you can launch a Weaviate serverless instance billed directly from AWS. + +:::info Prerequisites +- An AWS account with sufficient credit / payment method. +- (Recommended) Familiarity with AWS and the AWS console. +::: + +You can use [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-ng2dfhb4yjoic?sr=0-3&ref_=beagle&applicationId=AWSMPContessa) to launch a Weaviate serverless instance. + +## Installation instructions + +1. Go to Weaviate's [AWS Marketplace listing](https://aws.amazon.com/marketplace/pp/prodview-ng2dfhb4yjoic?sr=0-3&ref_=beagle&applicationId=AWSMPContessa). +1. Subscribe to the product by following the instructions on the page. + 1. Click View Purchase Options, then go to the next page. + 2. Review the pricing, terms and conditions, and click Subscribe. +From there you will be prompted to set up your account on [Weaviate Cloud](docs/cloud/index.mdx). + +:::info +
    + + A little background information + +- When you deploy Weaviate Serverless Cloud through the AWS Marketplace, you're subscribing to a Software as a Service (SaaS) solution that is specifically built for AWS customers. + +- AWS will notify you once your Weaviate serverless cluster is available. + +**This solution is ideal for:** + +- Organizations requiring AWS billing integration. +- Organizations with regulatory requirements who need specific regional deployments. + +
    + +::: + +### Billing + +You will be charged for Weaviate directly by AWS. + +:::warning + +If you cancel your Weaviate AWS marketplace subscription, your Weaviate organization and its clusters will be deleted by Weaviate. + +::: + +### Other marketplace offerings + +- [Weaviate Kubernetes Cluster](https://aws.amazon.com/marketplace/pp/prodview-cicacyv63r43i?sr=0-1&ref_=beagle&applicationId=AWSMPContessa) this product uses a CloudFormation template to create an EKS cluster with Weaviate already installed. + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/weaviate/installation/docker-compose.md b/docs/deploy/installation-guides/docker-installation.md similarity index 93% rename from docs/weaviate/installation/docker-compose.md rename to docs/deploy/installation-guides/docker-installation.md index f0df6de4..1577bb51 100644 --- a/docs/weaviate/installation/docker-compose.md +++ b/docs/deploy/installation-guides/docker-installation.md @@ -1,6 +1,5 @@ --- title: Docker -sidebar_position: 2 image: og/docs/installation.jpg # tags: ['installation', 'Docker'] --- @@ -37,8 +36,8 @@ You can customize your Weaviate configuration by creating a `docker-compose.yml` ## Sample Docker Compose file This starter Docker Compose file allows: -* Use of any [API-based model provider integrations](../model-providers/index.md) (e.g. `OpenAI`, `Cohere`, `Google`, and `Anthropic`). - * This includes the relevant embedding model, generative, and reranker [integrations](../model-providers/index.md). +* Use of any [API-based model provider integrations](docs/weaviate/model-providers/index.md) (e.g. `OpenAI`, `Cohere`, `Google`, and `Anthropic`). + * This includes the relevant embedding model, generative, and reranker [integrations](/docs/weaviate/model-providers/index.md). * Searching pre-vectorized data (without a vectorizer). * Mounts a persistent volume called `weaviate_data` to `/var/lib/weaviate` in the container to store data. @@ -121,16 +120,16 @@ volumes: ... ``` -This setup enables API-key based [authentication](../configuration/authentication.md) and role-based access control [authorization](../configuration/authorization.md). +This setup enables API-key based [authentication](docs/deploy/config-guides/authentication.md) and role-based access control [authorization](docs/deploy/config-guides/authorization.md). It defines the users `user-a` and `user-b` and corresponding keys `user-a-key` and `user-b-key` which serve as authentication credentials for connecting to your Weaviate instance. -The user `user-a` is granted admin access rights using the **Role-based access control (RBAC)** method. A custom role can be assigned to the user `user-b` by following the [authorization and RBAC guide](../configuration/authorization.md). +The user `user-a` is granted admin access rights using the **Role-based access control (RBAC)** method. A custom role can be assigned to the user `user-b` by following the [authorization and RBAC guide](docs/deploy/config-guides/authorization.md).     -Edit the `docker-compose.yml` file to suit your needs. You can add or remove [environment variables](#environment-variables), change the port mappings, or add additional [model provider integrations](../model-providers/index.md), such as [Ollama](../model-providers/ollama/index.md), or [Hugging Face Transformers](../model-providers/transformers/index.md). +Edit the `docker-compose.yml` file to suit your needs. You can add or remove [environment variables](#environment-variables), change the port mappings, or add additional [model provider integrations](/docs/weaviate/model-providers/index.md), such as [Ollama](/docs/weaviate/model-providers/ollama/index.md), or [Hugging Face Transformers](/docs/weaviate/model-providers/transformers/index.md). To start your Weaviate instance, run this command from your shell: @@ -151,7 +150,7 @@ import DocsConfigGen from '@site/src/components/DockerConfigGen'; You can use environment variables to control your Weaviate setup, authentication and authorization, module settings, and data storage settings. :::info List of environment variables -A comprehensive of list environment variables [can be found on this page](../config-refs/env-vars/index.md). +A comprehensive of list environment variables [can be found on this page](/docs/deploy/config-guides/env-vars/index.md). ::: ## Example configurations @@ -261,7 +260,7 @@ To configure Weaviate to use multiple host nodes, follow these steps: - Set the `RAFT_BOOTSTRAP_EXPECT` for each node with the number of voters. - Optionally, set the hostname for each node using `CLUSTER_HOSTNAME`. -(Read more about [horizontal replication in Weaviate](../concepts/cluster.md).) +(Read more about [horizontal replication in Weaviate](/docs/weaviate/concepts/cluster.md).) So, the Docker Compose file includes environment variables for the "founding" member that look like this: @@ -290,7 +289,7 @@ And the other members' configurations may look like this: RAFT_BOOTSTRAP_EXPECT: 3 ``` -Below is an example configuration for a 3-node setup. You may be able to test [replication](../configuration/replication.md) examples locally using this configuration. +Below is an example configuration for a 3-node setup. You may be able to test [replication](docs/deploy/config-guides/replication.md) examples locally using this configuration.
    diff --git a/docs/deploy/installation-guides/gcp-installation.md b/docs/deploy/installation-guides/gcp-installation.md new file mode 100644 index 00000000..c3c6f808 --- /dev/null +++ b/docs/deploy/installation-guides/gcp-installation.md @@ -0,0 +1,55 @@ +--- +title: GCP Marketplace +description: Set up Weaviate using Google Cloud Marketplace for simplified deployment. +image: og/docs/installation.jpg +tags: ['installation', 'Google Cloud Marketplace'] +--- + +A Weaviate cluster is easy to deploy with Google Cloud Marketplace (GCP). + +:::info Prerequisites + +- A Google Cloud account with sufficient credit / payment method. +- (Recommended) Familiarity with Google Cloud and the Google Cloud console. +::: + +## Installation instructions + +1. Go to Weaviate's [Google Cloud Marketplace listing](https://console.cloud.google.com/marketplace/product/weaviate-gcp-mktplace/weaviate) page and click Subscribe. +1. Configure and deploy Weaviate by following the on-screen instructions. + +Once completed, you will have a [Weaviate serverless cloud](docs/cloud/index.mdx) deployment. + +:::info +
    + + A little background information + +- When you deploy Weaviate Serverless Cloud through the GCP Marketplace, you're subscribing to a Software as a Service (SaaS) solution that is specifically built for GCP customers. + +- GCP will notify you once your Weaviate serverless cluster is available. + +**This solution is ideal for:** + +- Organizations requiring GCP billing integration. +- Organizations with regulatory requirements who need specific regional deployments. + +
    + +::: + +## Billing + +You will be charged for Weaviate and associated resources directly by Google Cloud. + +:::warning + +If you cancel your Weaviate GCP marketplace subscription, your Weaviate organization and its clusters will be deleted by Weaviate. + +::: + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/weaviate/installation/kubernetes.md b/docs/deploy/installation-guides/k8s-installation.md similarity index 97% rename from docs/weaviate/installation/kubernetes.md rename to docs/deploy/installation-guides/k8s-installation.md index c9b43685..45ebbca7 100644 --- a/docs/weaviate/installation/kubernetes.md +++ b/docs/deploy/installation-guides/k8s-installation.md @@ -122,8 +122,8 @@ In this example, the key `readonly-key` will authenticate a user as the `readonl OIDC authentication is also enabled, with WCD as the token issuer/identity provider. Thus, users with WCD accounts could be authenticated. This configuration sets `someuser@weaviate.io` as an admin user, so if `someuser@weaviate.io` were to authenticate, they will be given full (read and write) privileges. For further, general documentation on authentication and authorization configuration, see: -- [Authentication](../configuration/authentication.md) -- [Authorization](../configuration/authorization.md) +- [Authentication](../config-guides/authentication.md) +- [Authorization](../config-guides/authorization.md) #### Run as non-root user @@ -165,7 +165,7 @@ The above command (`helm upgrade...`) is idempotent. In other words, you can run To upgrade to `1.25` or higher from a pre-`1.25` version, you must delete the deployed `StatefulSet`, update the helm chart to version `17.0.0` or higher, and re-deploy Weaviate. -See the [1.25 migration guide for Kubernetes](../more-resources/migration/weaviate-1-25.md) for more details. +See the [1.25 migration guide for Kubernetes](../migration/weaviate-1-25.md) for more details. ## Additional Configuration Help diff --git a/docs/deploy/migration/_category_.json b/docs/deploy/migration/_category_.json new file mode 100644 index 00000000..aecbb8a4 --- /dev/null +++ b/docs/deploy/migration/_category_.json @@ -0,0 +1,4 @@ +{ + "label": "Migration guides", + "position": 8 +} diff --git a/docs/weaviate/more-resources/migration/archive.md b/docs/deploy/migration/archive.md similarity index 99% rename from docs/weaviate/more-resources/migration/archive.md rename to docs/deploy/migration/archive.md index 87117745..786f1984 100644 --- a/docs/weaviate/more-resources/migration/archive.md +++ b/docs/deploy/migration/archive.md @@ -1,7 +1,7 @@ --- title: Archive -sidebar_position: 90 image: og/docs/more-resources.jpg +sidebar_position: 10 # tags: ['migration'] --- diff --git a/docs/weaviate/more-resources/migration/index.md b/docs/deploy/migration/index.md similarity index 70% rename from docs/weaviate/more-resources/migration/index.md rename to docs/deploy/migration/index.md index 28ff4e1d..2528385a 100644 --- a/docs/weaviate/more-resources/migration/index.md +++ b/docs/deploy/migration/index.md @@ -1,6 +1,6 @@ --- title: Migration Guide -sidebar_position: 10 +sidebar_position: 0 image: og/docs/more-resources.jpg # tags: ['migration'] --- @@ -13,7 +13,7 @@ Weaviate is under active development, with new features and improvements being a When upgrading Weaviate, we recommend that you: -1. Create a complete [backup](../../configuration/backups.md) of your current Weaviate instance before beginning any upgrade process. +1. Create a complete [backup](docs/deploy/config-guides/backups.md) of your current Weaviate instance before beginning any upgrade process. 1. Plan to upgrade one minor version at a time, always using the latest patch version of each minor release. This approach of upgrading one minor version at a time helps to minimize the risk of issues during the upgrade process, by mirroring our testing and release process. Upgrading to the latest patch version of each minor release ensures that you have the latest bugfixes and improvements. @@ -41,11 +41,11 @@ Between `v1.25` and `v1.27`, there are two minor versions, `v1.26` and `v1.27`. ### Raft Migration (v1.25.0+) -Weaviate `v1.25.0` introduced Raft [as the consensus algorithm for cluster metadata](../../concepts/replication-architecture/cluster-architecture.md#metadata-replication-raft). This requires a one-time migration of the cluster metadata. +Weaviate `v1.25.0` introduced Raft [as the consensus algorithm for cluster metadata](/docs/weaviate/concepts/replication-architecture/cluster-architecture.md#metadata-replication-raft.). This requires a one-time migration of the cluster metadata. -In [Docker-based self-hosted instances](../../installation/docker-compose.md), the migration is automatic. +In [Docker-based self-hosted instances](docs/deploy/installation-guides/docker-installation.md), the migration is automatic. -In [Kubernetes-based self-hosted instances](../../installation/kubernetes.md), you must perform a manual migration step. For more information, see the [Weaviate `v1.25.0` migration guide](./weaviate-1-25.md). +In [Kubernetes-based self-hosted instances](docs/deploy/installation-guides/k8s-installation.md), you must perform a manual migration step. For more information, see the [Weaviate `v1.25.0` migration guide](./weaviate-1-25.md). This was a significant change to the Weaviate architecture. Accordingly, we suggest performing another backup after upgrading to `v1.25.latest`, before proceeding with further upgrades to ensure that you have a recent backup. @@ -59,6 +59,16 @@ If you are upgrading from a version before `v1.23.13`, we recommend that you: 2. Upgrade to at least `v1.23.13` (preferably to `v1.23.16`) or higher, using the [general upgrade instructions above](#general-upgrade-instructions). 3. Restore your backup to the upgraded instance. +## Downgrades + +### RAFT Snapshots (v1.28.13+, v1.29.5+, v1.30.2+) + +Multi-node instances of Weaviate running `1.28.13+`, `1.29.5+`, or `1.30.2+` may experience problems if downgraded to a `v1.27.x` version earlier than `1.27.26`. The cluster may not reach a **Ready** state due to a change in the way that RAFT snapshots are stored in the database. + +A fix for this issue will be released with `1.27.26`, which safely handles the downgrade path to `1.27`. + +If you need to downgrade Weaviate to `v1.27.x`, use `1.27.26` or higher. + ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; diff --git a/docs/weaviate/more-resources/migration/weaviate-1-25.md b/docs/deploy/migration/weaviate-1-25.md similarity index 95% rename from docs/weaviate/more-resources/migration/weaviate-1-25.md rename to docs/deploy/migration/weaviate-1-25.md index 501334f7..c5268d4b 100644 --- a/docs/weaviate/more-resources/migration/weaviate-1-25.md +++ b/docs/deploy/migration/weaviate-1-25.md @@ -1,6 +1,6 @@ --- title: 1.25 (For Kubernetes users) -sidebar_position: 10 +sidebar_position: 2 image: og/docs/more-resources.jpg # tags: ['migration'] --- @@ -55,7 +55,7 @@ If your deployment is on another namespace, modify the instructions below accord ### (Optional) Backup -Before proceeding with the upgrade, we recommend making a [backup](/docs/weaviate/configuration/backups.md) of your Weaviate Database. If a backup is not possible, you can explore other options, such as manually [exporting your data](/docs/weaviate/manage-collections/migrate.mdx) +Before proceeding with the upgrade, we recommend making a [backup](docs/deploy/config-guides/backups.md) of your Weaviate Database. If a backup is not possible, you can explore other options, such as manually [exporting your data](/docs/weaviate/manage-collections/migrate.mdx) ### 1. Delete StatefulSet @@ -193,6 +193,8 @@ helm upgrade weaviate weaviate/weaviate \ This should bring the cluster back to your specified pre-`1.25` version. +## Questions and feedback + import DocsFeedback from '/_includes/docs-feedback.mdx'; diff --git a/docs/weaviate/more-resources/migration/weaviate-1-30.md b/docs/deploy/migration/weaviate-1-30.md similarity index 98% rename from docs/weaviate/more-resources/migration/weaviate-1-30.md rename to docs/deploy/migration/weaviate-1-30.md index f939599d..56045215 100644 --- a/docs/weaviate/more-resources/migration/weaviate-1-30.md +++ b/docs/deploy/migration/weaviate-1-30.md @@ -1,7 +1,7 @@ --- title: BlockMax WAND migration guide sidebar_label: 1.30 (BlockMax WAND migration) -sidebar_position: 9 +sidebar_position: 1 image: og/docs/more-resources.jpg --- @@ -323,3 +323,10 @@ BlockMax WAND algorithm is available in `v1.29` as a **technical preview**. **We Enable BlockMax WAND by setting the environment variables `USE_BLOCKMAX_WAND` and `USE_INVERTED_SEARCHABLE` to `true`. Now, all new data added to Weaviate will use BlockMax WAND for BM25 and hybrid searches. However, preexisting data will continue to use the default WAND algorithm. + + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/deploy/production/Kubernetes/index.mdx b/docs/deploy/production/Kubernetes/index.mdx new file mode 100644 index 00000000..2f09b810 --- /dev/null +++ b/docs/deploy/production/Kubernetes/index.mdx @@ -0,0 +1,31 @@ +--- + +sidebar_label: "Kubernetes" + +--- + +# Kubernetes + +Weaviate leverages Kubernetes for scalable, resilient, and production-grade deployments. +This section provides guides, best practices, and tutorials for making the best of your Kubernetes deployments in production. + +## Deployment Options + +Weaviate can use Kubernetes in a variety of deployments for your enterprise environment: + +### Enterprise Cloud + +- Amazon EKS (Elastic Kubernetes Service) +- Google Kubernetes Engine (GKE) +- Azure Kubernetes Service (AKS) + +### Bring Your Own Cloud + +- Amazon EKS (Elastic Kubernetes Service) + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + + diff --git a/docs/deploy/production/Kubernetes/k8s-poc.md b/docs/deploy/production/Kubernetes/k8s-poc.md new file mode 100644 index 00000000..d1215e9c --- /dev/null +++ b/docs/deploy/production/Kubernetes/k8s-poc.md @@ -0,0 +1,160 @@ +# Building with Weaviate: Getting to Production + +## Introduction + +Are you ready to deploy and test Weaviate on a self-managed K8s (Kubernetes) cluster? This guide shows how to validate Weaviate’s capabilities in your enterprise environment. + +At the end of this guide, expect to have: + +- A configured Helm-based deployment and networking setup +- Basic scaling, persistent storage, and resource management +- TLS, RBAC, and security best practices implements +- Monitoring, logging, and backup strategies enabled + +### Prerequisites + +Before beginning, ensure that you have the following: + +#### Technical Knowledge + +- Basic Kubernetes and containerization conceptual knowledge +- Basic experience with Helm and `kubectl` + +:::note + +Check out the Academy course [“Run Weaviate on Kubernetes”](https://weaviate.io/developers/academy/deployment/k8s) if you need assistance. + +::: + +#### Required Tools + +- A running Kubernetes cluster with Weaviate installed +- `kubectl` installed +- Helm installed + +## Step 1: Configure your Helm Chart + +- Use the official [Weaviate Helm chart](https://github.com/weaviate/weaviate-helm) for your installation: + +``` + helm repo add weaviate https://weaviate.github.io/weaviate-helm + helm install my-weaviate weaviate/weaviate +``` + +- Customize the values to fit your enterprise requirements (e.g., resource allocation, storage settings). +- Deploy the chart and verify pod health. + +## Step 2: Network Security + +- Configure an ingress controller to securely expose Weaviate. +- Enable TLS with a certificate manager and enforce TLS encryption for all client-server communication. +- Assign a domain name for external access. +- Implement RBAC or admin lists to restrict user access. + +
    + An example of RBAC enabled on your Helm chart + +```yaml + authorization: + rbac: + enabled: true + root_users: + - admin_user1 + - admin_user2 +``` +
    + +
    + An example of admin lists implemented on your Helm chart (if not using RBAC) + +```yaml + admin_list: + enabled: true + users: + - admin_user1 + - admin_user2 + - api-key-user-admin + read_only_users: + - readonly_user1 + - readonly_user2 + - api-key-user-readOnly +``` +[Admin List Configuration](docs/deploy/config-guides/authorization.md#admin-list-kubernetes) + +
    + +:::tip +Using an admin list will allow you to define your admin or read-only user/API-key pairs across all Weaviate resources. Whereas RBAC allows you more granular permissions by defining roles and assigning them to users either via API keys or OIDC. +::: + +## Step 3: Scaling + +- Implement horizontal scaling to ensure high availability: + +```yaml +replicaCount: 3 +``` + +- Define CPU/memory limits and requests to optimize pod efficiency. + +
    + An example of defining CPU and memory limits and cores + +```yaml +resources: + requests: + cpu: "500m" + memory: "1Gi" + limits: + cpu: "2" + memory: "4Gi" +``` +
    + +## Step 4: Monitoring and Logging + +- Use Prometheus and Grafana to collect and analyze performance metrics. +- Implement alerting for issue resolution. + +
    + An example of enabling service monitoring + +```yaml +serviceMonitor: + enabled: true + interval: 30s + scrapeTimeout: 10s +``` +
    + + +## Step 5: Upgrades and Backups + +- Use the rolling update strategy used by Helm to minimize downtime. + +
    + An example of configuring the rolling update strategy. + +```yaml +updateStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 +``` +
    + +- Test new Weaviate versions before deploying into production. +- Implement disaster recovery procedures to ensure that data is restored quickly. + +### Conclusion + +Voila! You now have a deployment that is *somewhat* ready for production. Your next step will be to complete the self-assessment and identify any gaps. + +### Next Steps: [Production Readiness Self-Assessment](./production-readiness.md) + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/deploy/production/Kubernetes/production-readiness.md b/docs/deploy/production/Kubernetes/production-readiness.md new file mode 100644 index 00000000..7a7f828f --- /dev/null +++ b/docs/deploy/production/Kubernetes/production-readiness.md @@ -0,0 +1,93 @@ +--- +sidebar_label: Production Readiness Self-Assessment +--- + +# Kubernetes Production Readiness Self-Assessment + +Think you’re ready for production? Ensuring that your Weaviate cluster is production-ready requires careful planning, configuration, and ongoing maintenance. Ensuring that you have a stable, reliable deployment requires you to think of your *ending* at the *beginning.* This guide provides you with introspective questions to assess readiness and identify any potential gaps before moving your workloads into production. + +:::tip +If you *do* identify gaps within your deployment, be sure to reach out to your SE (solutions engineer) who can help steer you on the path to production success! +::: + + +### High Availability and Resilience + +- [ ] Are your clusters deployed across multiple availability zones (AZs) or regions to prevent downtime? +- [ ] Are you running Weaviate in a highly available setup with a 3-node minimum? +- [ ] Have you configured your schema to use a replication factor of 3 to ensure copies of data are available during node outage? +- [ ] Are replicas deployed across multiple nodes for redundancy? +- [ ] Is your control plane highly available? +- [ ] Is your application fault-tolerant *without* your control plane? +- [ ] Are there automatic node repair or self-healing mechanisms in place? +- [ ] Have failover scenarios been tested to validate resilience? +- [ ] Are you utilizing Weaviate’s backup capabilities for disaster recovery? + - [ ] How often are these mechanisms tested? + - [ ] Has the ability to recover from a node failure or database corruption been tested? +- [ ] Have you thought about the retention period of backups? + - [ ] How do you clean up any out-of-date backups? +- [ ] Are rolling updates performed to avoid downtime? +- [ ] Are canary deployments implemented to safely test new releases? +- [ ] Do you have development or test environments to safely test changes? + +### Data Ingestion and Query Performance + +- [ ] Is there a strategy for handling heavy ingestion loads? +- [ ] Has the percentage of resources for indexing vs querying applications been specified? +- [ ] Is there a defined strategy for data deduplication and cleanup before ingestion? +- [ ] How frequently is data added, updated, or deleted? + - [ ] Is data updated in place or mostly append-only + - [ ] How often do deletion operations trigger garbage collection? +- [ ] Have you implemented a scheduling strategy for large ingestion jobs? +- [ ] Have you tested query performance under load? + - [ ] Is query performance monitored using Prometheus or Grafana? +- [ ] Have replica shards been deployed for load balancing and failover support? + + +### Resource Management + +- [ ] Have you considered your data’s consumption pattern(s)? + - [ ] Has your memory allocation been right-sized to match workload demand? + - [ ] Has your storage/compute allocation also been right-sized to match workload demand? + - [ ] Is there a process to delete old or unused objects? +- [ ] Have multiple replicas been configured to balance read-heavy workloads? +- [ ] Has the proper storage class been selected for your needs? + - [ ] Does your storage class support volume expansion so that you can support growth over time? +- [ ] Is the data within your cluster properly backed up, including the persistent storage? +- [ ] Is the sharding strategy aligned with the size and access patterns of the dataset? +- [ ] Is `GOMEMLIMIT` properly configured for memory management? + - [ ] Is `GOMEMLIMIT` set based on available system memory to prevent excessive garbage collection pauses? +- [ ] Have you considered vector quantization techniques to reduce memory requirements? + +### Tenant State Management + +- [ ] Are you implementing multi-tenancy? + - [ ] Are there limits or quotas per tenant to avoid noisy neighbor issues? +- [ ] Is there a strategy for offloading inactive tenant data? + +### Security + +- [ ] Are the components of your cluster communicating via SSL/TLS and trusted certificates? +- [ ] Is the *“principle of least privilege”* being followed? +- [ ] Are your container security defaults set properly? +- [ ] Is access to your cluster strictly limited? +- [ ] Has [RBAC](/docs/weaviate/configuration/rbac/index.mdx) been implemented to restrict access? +- [ ] Have network policies been implemented to limit pod-to-pod communication? +- [ ] Are secrets secured with K8s Secrets or a vault solution? +- [ ] Do you have a process for when secrets are exposed, when access is lost to a key or certificate, and when secrets need to be rotated? + +### Monitoring and Observability + +- [ ] Is logging implemented? + - [ ] Are the collected logs stored centrally? +- [ ] Is metric collection enabled using Prometheus (or Alloy, DataDog, or another monitoring platform)? +- [ ] Are health and performance metrics being visualized in Grafana? +- [ ] Are alerts configured for events? + +Evaluate these key areas to build a highly available, resilient, and efficient deployment that will scale to meet your business needs. By ensuring that these self-assessment questions have been addressed, you can proactively identify potential risks and maximize the reliability of your deployment. + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/deploy/production/index.mdx b/docs/deploy/production/index.mdx new file mode 100644 index 00000000..73dce29a --- /dev/null +++ b/docs/deploy/production/index.mdx @@ -0,0 +1,44 @@ +--- + +sidebar_label: "Overview" +sidebar_position: 0 + +--- + +# Production Environments + +Deploying Weaviate in a production environment requires careful planning to ensure stability, security, and performance. +Currently, the only supported method of deploying Weaviate in a production environment is through Kubernetes. +Unlike development or testing environments, a production instance must be resilient, scalable, and optimized for real-world workloads. + +## Key Aspects of a Production Environments + +- **Scalability and Performance** + - Production workloads demand high availability and low latency. + - Auto-scaling strategies and resource monitoring must be in place. +- **Data Resilience and Backups** + - Data must be safeguarded against corruption and loss. + - Disaster recover strategies and automated backups must be implemented. +- **Security and Compliance** + - Data at rest and in transit should be encrypted. + - liance best practices must be followed. +- **Monitoring and Observability** + - Logs and metrics must be collected for troubleshooting and optimization. + - Monitoring tools should be used to track performance. + - Alerting mechanisms must be set up to detect any anaomolies or failures. +- **Networking and Access Control** + - Privileges should be limited using RBAC (role-based access control). + - Load balancing should be configured. + +### Weaviate Production Environments +Weaviate has different options for deploying in a production environment using: + +- **Serverless Cloud**: A Serverless Cloud instance is a robust cluster designed for production use. +- **Enterprise Cloud**: Enterprise Cloud is a fully-managed deployment of dedicated resources within a secure, highly available environment. +- **Bring Your Own Cloud (BYOC)**: Using your own cloud environment, you are able to have a fully managed deployed. + +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/deploy/tutorials/index.mdx b/docs/deploy/tutorials/index.mdx new file mode 100644 index 00000000..d0468b8a --- /dev/null +++ b/docs/deploy/tutorials/index.mdx @@ -0,0 +1,36 @@ +--- +title: Tutorials +description: Explore Weaviate tutorials for practical guidance on data management and queries. +sidebar_position: 0 +image: og/docs/tutorials.jpg +hide_table_of_contents: true +# tags: ['how to', 'schema'] +--- + +import BasicPrereqs from "/_includes/prerequisites-quickstart.md"; + + + +Tutorials aim to empower you to use Weaviate based on _your_ particular needs. +Here, you can learn how to: + +import CardsSection from "/src/components/CardsSection"; + +export const advancedFeaturesData = [ + { + title: "Set up Role-Based Access Control (RBAC)", + description: + "Configure roles, permissions, and user assignments for secure access control in Weaviate.", + link: "/docs/deploy/tutorials/rbac", + icon: "fas fa-user-shield", + }, +]; + +
    + +
    +## Questions and feedback + +import DocsFeedback from '/_includes/docs-feedback.mdx'; + + diff --git a/docs/weaviate/tutorials/rbac.mdx b/docs/deploy/tutorials/rbac.mdx similarity index 93% rename from docs/weaviate/tutorials/rbac.mdx rename to docs/deploy/tutorials/rbac.mdx index 7d828ea6..aeb7c95c 100644 --- a/docs/weaviate/tutorials/rbac.mdx +++ b/docs/deploy/tutorials/rbac.mdx @@ -1,7 +1,6 @@ --- title: Set up RBAC in Weaviate description: Learn how to set up RBAC (Role Based Access Control) in Weaviate -sidebar_position: 3 image: og/docs/tutorials.jpg # tags: ['basics'] --- @@ -80,7 +79,7 @@ We are going to create the following roles: Before starting this tutorial, ensure you have the following: - Docker for running a Weaviate instance locally. -- Your preferred Weaviate [client library](../client-libraries/index.mdx) installed. +- Your preferred Weaviate [client library](docs/weaviate/client-libraries/index.mdx) installed. ### Local instance - `root` user @@ -133,7 +132,7 @@ We will connect to Weaviate with `root-user`, and once we create a new role, we ### Step 1: Connect to Weaviate -Ensure you are connected to Weaviate with a user possessing sufficient permissions to manage roles. You can achieve this by either using the predefined `root` role during [Weaviate configuration](../configuration/rbac/configuration.md) or by granting a user the [`manage_roles` permission](../configuration/rbac/manage-roles.mdx#role-management-permissions). +Ensure you are connected to Weaviate with a user possessing sufficient permissions to manage roles. You can achieve this by either using the predefined `root` role during [Weaviate configuration](docs/deploy/config-guides/configuring-rbac.md) or by granting a user the [`manage_roles` permission](docs/weaviate/configuration/rbac/manage-roles.mdx#role-management-permissions). @@ -280,7 +279,9 @@ Now, you can assign the role `rw_role` to `custom-user`: ### Step 1: Connect to Weaviate -Ensure you are connected to Weaviate with a user possessing sufficient permissions to manage roles. You can achieve this by either using the predefined `root` role during [Weaviate configuration](../configuration/rbac/configuration.md) or by granting a user the [`manage_roles` permission](../configuration/rbac/manage-roles.mdx#role-management-permissions). +Ensure you are connected to Weaviate with a user possessing sufficient permissions to manage roles. +You can achieve this by either using the predefined `root` role during [Weaviate configuration](/docs/deploy/config-guides/configuring-rbac) +or by granting a user the [`manage_roles` permission](/docs/weaviate/configuration/rbac/manage-roles#role-management). @@ -427,7 +428,9 @@ Now, you can assign the role `viewer_role` to `custom-user`: ### Step 1: Connect to Weaviate -Ensure you are connected to Weaviate with a user possessing sufficient permissions to manage roles. You can achieve this by either using the predefined `root` role during [Weaviate configuration](../configuration/rbac/configuration.md) or by granting a user the [`manage_roles` permission](../configuration/rbac/manage-roles.mdx#role-management-permissions). +Ensure you are connected to Weaviate with a user possessing sufficient permissions to manage roles. +You can achieve this by either using the predefined `root` role during [Weaviate configuration](docs/deploy/config-guides/configuring-rbac.md) +or by granting a user the [`manage_roles` permission](docs/weaviate/configuration/rbac/manage-roles.mdx#role-management-permissions). @@ -580,9 +583,9 @@ It walks you through connecting to Weaviate using a user with role management ca ## Additional resources -- [RBAC: Configuration](../configuration/rbac/configuration.md) -- [RBAC: Manage roles](../configuration/rbac/manage-roles.mdx) -- [RBAC: Manage users](../configuration/rbac/manage-users.mdx) +- [RBAC: Configuration](docs/deploy/config-guides/configuring-rbac.md) +- [RBAC: Manage roles](docs/weaviate/configuration/rbac/manage-roles.mdx) +- [RBAC: Manage users](docs/weaviate/configuration/rbac/manage-users.mdx) ## Questions and feedback diff --git a/docs/weaviate/api/grpc.md b/docs/weaviate/api/grpc.md index 9f0d48fb..40f6dc4e 100644 --- a/docs/weaviate/api/grpc.md +++ b/docs/weaviate/api/grpc.md @@ -29,7 +29,7 @@ This directory contains the following files: As an example, the snippet below maps `50051` as the host port so that it can be accessed from outside the container. The `50051` port is mapped to the `50051` port inside the container for gRPC calls, and the `8080` port is mapped to the `8080` port inside the container for REST calls. :::info -We suggest using the default port `50051` for gRPC calls. It can be modified through the `GRPC_PORT` [environment variable](../config-refs/env-vars/index.md). +We suggest using the default port `50051` for gRPC calls. It can be modified through the `GRPC_PORT` [environment variable](docs/deploy/config-guides/env-vars/index.md). Note that [Weaviate Cloud](https://console.weaviate.cloud/) uses port `443` for gRPC. ::: diff --git a/docs/weaviate/best-practices/index.md b/docs/weaviate/best-practices/index.md index 047bbece..58850fcd 100644 --- a/docs/weaviate/best-practices/index.md +++ b/docs/weaviate/best-practices/index.md @@ -97,7 +97,8 @@ These thresholds can be adjusted to better fit your use case. For example, if yo Set `DISK_USE_WARNING_PERCENTAGE` and `DISK_USE_READONLY_PERCENTAGE` to adjust the disk usage thresholds, and `MEMORY_WARNING_PERCENTAGE` and `MEMORY_READONLY_PERCENTAGE` to adjust the memory usage thresholds. :::tip Further resources -- [References: Environment variables](../config-refs/env-vars/index.md#general) +- [References: Environment variables](/docs/deploy/config-guides/env-vars/index.md#general) () + ::: ### Plan memory allocation diff --git a/docs/weaviate/concepts/cluster.md b/docs/weaviate/concepts/cluster.md index b10b703b..17837042 100644 --- a/docs/weaviate/concepts/cluster.md +++ b/docs/weaviate/concepts/cluster.md @@ -72,7 +72,7 @@ Example - assume the following scenario: A class is comprised of 4 shards and ta By default, Weaviate nodes in a cluster use a gossip-like protocol through [Hashicorp's Memberlist](https://github.com/hashicorp/memberlist) to communicate node state and failure scenarios. -Weaviate - especially when running as a cluster - is optimized to run on Kubernetes. The [Weaviate Helm chart](/docs/weaviate/installation/kubernetes.md#weaviate-helm-chart) makes use of a `StatefulSet` and a headless `Service` that automatically configures node discovery. All you have to do is specify the desired node count. +Weaviate - especially when running as a cluster - is optimized to run on Kubernetes. The [Weaviate Helm chart](docs/deploy/installation-guides/k8s-installation.md#weaviate-helm-chart) makes use of a `StatefulSet` and a headless `Service` that automatically configures node discovery. All you have to do is specify the desired node count.
    FQDN for node discovery @@ -83,7 +83,7 @@ This was an experimental feature. Use with caution. ::: -There can be a situation where IP-address based node discovery is not optimal. In such cases, you can set `RAFT_ENABLE_FQDN_RESOLVER` and `RAFT_FQDN_RESOLVER_TLD` [environment variables](../config-refs/env-vars/index.md#multi-node-instances) to enable fully qualified domain name (FQDN) based node discovery. +There can be a situation where IP-address based node discovery is not optimal. In such cases, you can set `RAFT_ENABLE_FQDN_RESOLVER` and `RAFT_FQDN_RESOLVER_TLD` [environment variables](docs/deploy/config-guides/env-vars/index.md#multi-node-instances) to enable fully qualified domain name (FQDN) based node discovery. If this feature is enabled, Weaviate uses the FQDN resolver to resolve the node name to the node IP address for metadata (e.g., Raft) communication. @@ -124,7 +124,7 @@ Shards were assigned to 'live' nodes in a round-robin fashion starting with a ra ## Consistency and current limitations -* Starting with `v1.25.0`, Weaviate adopts the [Raft consensus algorithm](https://raft.github.io/) which is a log-based algorithm coordinated by an elected leader. This brings an additional benefit in that concurrent schema changes are now supported.
    If you are a Kubernetes user, see the [`1.25 migration guide`](/docs/weaviate/more-resources/migration/weaviate-1-25.md) before you upgrade. To upgrade, you have to delete your existing StatefulSet. +* Starting with `v1.25.0`, Weaviate adopts the [Raft consensus algorithm](https://raft.github.io/) which is a log-based algorithm coordinated by an elected leader. This brings an additional benefit in that concurrent schema changes are now supported.
    If you are a Kubernetes user, see the [`1.25 migration guide`](docs/deploy/migration/weaviate-1-25.md) before you upgrade. To upgrade, you have to delete your existing StatefulSet. * As of `v1.8.0`, the process of broadcasting schema changes across the cluster uses a form of two-phase transaction that as of now cannot tolerate node failures during the lifetime of the transaction. * As of `v1.8.0`, dynamically scaling a cluster is not fully supported yet. New nodes can be added to an existing cluster, however it does not affect the ownership of shards. Existing nodes can not yet be removed if data is present, as shards are not yet being moved to other nodes prior to a removal of a node. diff --git a/docs/weaviate/concepts/data.md b/docs/weaviate/concepts/data.md index 2c68df98..a3aacb32 100644 --- a/docs/weaviate/concepts/data.md +++ b/docs/weaviate/concepts/data.md @@ -466,7 +466,7 @@ Cross-references like these are not supported: ### Monitoring metrics -To group tenants together for monitoring, set [`PROMETHEUS_MONITORING_GROUP = true`](../config-refs/env-vars/index.md) in your system configuration file. +To group tenants together for monitoring, set [`PROMETHEUS_MONITORING_GROUP = true`](/docs/deploy/config-guides/env-vars/index.md) in your system configuration file. ### Number of tenants per node diff --git a/docs/weaviate/concepts/filtering.md b/docs/weaviate/concepts/filtering.md index 588d74d0..1920dff6 100644 --- a/docs/weaviate/concepts/filtering.md +++ b/docs/weaviate/concepts/filtering.md @@ -82,7 +82,7 @@ A roaring bitmap index for `text` properties is available from `1.19` and up, an If you are using Weaviate version `< 1.18.0`, you can take advantage of roaring bitmaps by migrating to `1.18.0` or higher, and going through a one-time process to create the new index. Once your Weaviate instance creates the Roaring Bitmap index, it will operate in the background to speed up your work. -This behavior is set through the REINDEX_SET_TO_ROARINGSET_AT_STARTUP [environment variable](../config-refs/env-vars/index.md). If you do not wish for reindexing to occur, you can set this to `false` prior to upgrading. +This behavior is set through the REINDEX_SET_TO_ROARINGSET_AT_STARTUP [environment variable](docs/deploy/config-guides/env-vars/index.md). If you do not wish for reindexing to occur, you can set this to `false` prior to upgrading. :::info Read more To learn more about Weaviate's roaring bitmaps implementation, see the [in-line documentation](https://pkg.go.dev/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/roaringset). diff --git a/docs/weaviate/concepts/indexing/inverted-index.md b/docs/weaviate/concepts/indexing/inverted-index.md index b8e61e43..6b9d279d 100644 --- a/docs/weaviate/concepts/indexing/inverted-index.md +++ b/docs/weaviate/concepts/indexing/inverted-index.md @@ -27,7 +27,7 @@ As always, we recommend upgrading to the latest version of Weaviate to benefit f The BlockMax WAND algorithm is a variant of the WAND algorithm that is used to speed up BM25 and hybrid searches. It organizes the inverted index in blocks to enable skipping over blocks that are not relevant to the query. This can significantly reduce the number of documents that need to be scored, improving search performance. -If you are experiencing slow BM25 (or hybrid) searches and use a Weaviate version prior to `v1.30`, try migrating to a newer version that uses the BlockMax WAND algorithm to see if it improves performance. If you need to migrate existing data from a previous version of Weaviate, follow the [v1.30 migration guide](../../more-resources/migration/weaviate-1-30.md). +If you are experiencing slow BM25 (or hybrid) searches and use a Weaviate version prior to `v1.30`, try migrating to a newer version that uses the BlockMax WAND algorithm to see if it improves performance. If you need to migrate existing data from a previous version of Weaviate, follow the [v1.30 migration guide](docs/deploy/migration/weaviate-1-30.md). :::note Scoring changes with BlockMax WAND diff --git a/docs/weaviate/concepts/indexing/vector-index.md b/docs/weaviate/concepts/indexing/vector-index.md index 2c6f9660..d1a68c02 100644 --- a/docs/weaviate/concepts/indexing/vector-index.md +++ b/docs/weaviate/concepts/indexing/vector-index.md @@ -136,7 +136,7 @@ The resulting search list has these characteristics. - A maximum length of 25 objects ("dynamicEfMax": 25). - An actual size of 5 to 25 objects. -If you use the [`docker-compose.yml` file from Weaviate](../../installation/docker-compose.md) to run your local instance, the `QUERY_DEFAULTS_LIMIT` environment variable sets a reasonable default query limit. To prevent out of memory errors,`QUERY_DEFAULTS_LIMIT` is significantly lower than `QUERY_MAXIMUM_RESULTS`. +If you use the [`docker-compose.yml` file from Weaviate](docs/deploy/installation-guides/docker-installation.md) to run your local instance, the `QUERY_DEFAULTS_LIMIT` environment variable sets a reasonable default query limit. To prevent out of memory errors,`QUERY_DEFAULTS_LIMIT` is significantly lower than `QUERY_MAXIMUM_RESULTS`. To change the default limit, edit the value for `QUERY_DEFAULTS_LIMIT` when you configure your Weaviate instance. @@ -162,7 +162,7 @@ When asynchronous indexing is enabled, all vector indexing operations go through This means that the object store can be updated quickly to finish performing user requests while the vector index updates in the background. Asynchronous indexing is especially useful for importing large amounts of data. -This means that there will be a short delay between object creation and the object being available for vector search using the HNSW index. The number of objects in the queue can be monitored per node [as shown here](../../config-refs/nodes.md). +This means that there will be a short delay between object creation and the object being available for vector search using the HNSW index. The number of objects in the queue can be monitored per node [as shown here](/docs/deploy/config-guides/nodes.md). :::info Changes in `v1.28` In Weaviate `v1.22` to `v1.27`, the async indexing feature only affected batch import operations, using an in-memory queue. diff --git a/docs/weaviate/concepts/replication-architecture/cluster-architecture.md b/docs/weaviate/concepts/replication-architecture/cluster-architecture.md index 27f7ed91..21e8e972 100644 --- a/docs/weaviate/concepts/replication-architecture/cluster-architecture.md +++ b/docs/weaviate/concepts/replication-architecture/cluster-architecture.md @@ -14,7 +14,7 @@ In Weaviate, metadata replication and data replication are separate. For the met By default, Weaviate nodes in a cluster use a gossip-like protocol through [Hashicorp's Memberlist](https://github.com/hashicorp/memberlist) to communicate node state and failure scenarios. -Weaviate is optimized to run on Kubernetes, especially when operating as a cluster. The [Weaviate Helm chart](/docs/weaviate/installation/kubernetes.md#weaviate-helm-chart) makes use of a `StatefulSet` and a headless `Service` that automatically configures node discovery. +Weaviate is optimized to run on Kubernetes, especially when operating as a cluster. The [Weaviate Helm chart](docs/deploy/installation-guides/k8s-installation.md#weaviate-helm-chart) makes use of a `StatefulSet` and a headless `Service` that automatically configures node discovery.
    FQDN for node discovery @@ -25,7 +25,7 @@ This was an experimental feature. Use with caution. ::: -There can be a situation where IP-address based node discovery is not optimal. In such cases, you can set `RAFT_ENABLE_FQDN_RESOLVER` and `RAFT_FQDN_RESOLVER_TLD` [environment variables](../../config-refs/env-vars/index.md#multi-node-instances) to enable [fully qualified domain name (FQDN)](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) based node discovery. +There can be a situation where IP-address based node discovery is not optimal. In such cases, you can set `RAFT_ENABLE_FQDN_RESOLVER` and `RAFT_FQDN_RESOLVER_TLD` [environment variables](docs/deploy/config-guides/env-vars/index.md#multi-node-instances) to enable [fully qualified domain name (FQDN)](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) based node discovery. If this feature is enabled, Weaviate uses the FQDN resolver to resolve the node name to the node IP address for metadata (e.g., Raft) communication. diff --git a/docs/weaviate/concepts/replication-architecture/consistency.md b/docs/weaviate/concepts/replication-architecture/consistency.md index 8f20f99c..b7f4d633 100644 --- a/docs/weaviate/concepts/replication-architecture/consistency.md +++ b/docs/weaviate/concepts/replication-architecture/consistency.md @@ -59,7 +59,7 @@ A clean (without fails) execution has two phases: Some queries require the collection definition. Prior to the introduction of this feature, every such query led to the local (requesting) node to fetch the collection definition from the leader node. This meant that the definition was strongly consistent, but it could lead to additional traffic and load. -Where available, the `COLLECTION_RETRIEVAL_STRATEGY` [environment variable](../../config-refs/env-vars/index.md#multi-node-instances) can be set to `LeaderOnly`, `LocalOnly`, or `LeaderOnMismatch`. +Where available, the `COLLECTION_RETRIEVAL_STRATEGY` [environment variable](docs/deploy/config-guides/env-vars/index.md#multi-node-instances) can be set to `LeaderOnly`, `LocalOnly`, or `LeaderOnMismatch`. - `LeaderOnly` (default): Always requests the definition from the leader node. This is the most consistent behavior but can lead to higher intra-cluster traffic. - `LocalOnly`: Always use the local definition; leading to eventually consistent behavior while reducing intra-cluster traffic. @@ -195,7 +195,7 @@ Repair-on-read works well with one or two isolated repairs. Async replication is Async replication supplements the repair-on-read mechanism. If a node becomes inconsistent between sync checks, the repair-on-read mechanism catches the problem at read time. -To activate async replication, set `asyncEnabled` to true in the [`replicationConfig` section of your collection definition](../../manage-collections/multi-node-setup.mdx#replication-settings). Visit the [How-to: Replication](/docs/weaviate/configuration/replication#async-replication-settings) page to learn more about the available async replication settings. +To activate async replication, set `asyncEnabled` to true in the [`replicationConfig` section of your collection definition](../../manage-collections/multi-node-setup.mdx#replication-settings). Visit the [How-to: Replication](docs/deploy/config-guides/replication.md#async-replication-settings) page to learn more about the available async replication settings. #### Memory and performance considerations for async replication diff --git a/docs/weaviate/concepts/replication-architecture/index.md b/docs/weaviate/concepts/replication-architecture/index.md index 0ab463f1..e7a03745 100644 --- a/docs/weaviate/concepts/replication-architecture/index.md +++ b/docs/weaviate/concepts/replication-architecture/index.md @@ -147,7 +147,7 @@ Prior to Weaviate `v1.25`, each cluster metadata change was recorded via a distr This is a synchronous process, which means that the cluster metadata change is only committed when all nodes have acknowledged the change. In this architecture, any node downtime would temporarily prevent metadata operations. Additionally, only one such operation could be processed at a time. -If you are using Weaviate `v1.24` or earlier, you can [upgrade to `v1.25`](../../more-resources/migration/weaviate-1-25.md) to benefit from the Raft consensus algorithm for cluster metadata changes. +If you are using Weaviate `v1.24` or earlier, you can [upgrade to `v1.25`](docs/deploy/migration/weaviate-1-25.md) to benefit from the Raft consensus algorithm for cluster metadata changes.
    @@ -163,7 +163,7 @@ Read more about how replication works in Weaviate in [Philosophy](./philosophy.m ## How do I use replication in Weaviate? -See [how to configure replication](../../configuration/replication.md). You can enable replication in the collection definition. In queries, you can [specify the desired consistency level](../../search/basics.md#replication). +See [how to configure replication](docs/deploy/config-guides/replication.md). You can enable replication in the collection definition. In queries, you can [specify the desired consistency level](../../search/basics.md#replication). ## Roadmap @@ -172,7 +172,7 @@ See [how to configure replication](../../configuration/replication.md). You can ## Related pages -- [Configuration: Replication](../../configuration/replication.md) +- [Configuration: Replication](docs/deploy/config-guides/replication.md) ## Questions and feedback diff --git a/docs/weaviate/concepts/resources.md b/docs/weaviate/concepts/resources.md index 0ef5730e..768e0069 100644 --- a/docs/weaviate/concepts/resources.md +++ b/docs/weaviate/concepts/resources.md @@ -10,7 +10,7 @@ Weaviate scales well for large projects. Smaller projects, less than 1M objects, ## Limit available resources -You can set [environment variables](../config-refs/env-vars/index.md) to manage Weaviate's resource usage, as to prevent Weaviate from using all available resources. The following environment variables are available: +You can set [environment variables](docs/deploy/config-guides/env-vars/index.md) to manage Weaviate's resource usage, as to prevent Weaviate from using all available resources. The following environment variables are available: - `LIMIT_RESOURCES`: When set to true, Weaviate automatically limits its resource usage. It sets memory usage to 80% of the total memory and uses all but one CPU core. It overrides any `GOMEMLIMIT` values but respects `GOMAXPROCS` settings. @@ -96,7 +96,7 @@ The memory calculation that includes `maxConnections` describes the system state In rare situations - typically on large machines with very high import speeds - Weaviate can allocate memory faster than the garbage collector can free it. When this happens, the system kernel can trigger an `out of memory kill (OOM-Kill)`. This is a known issue that Weaviate is actively working on. ### Data import -To avoid out-of-memory issues during imports, set `LIMIT_RESOURCES` to `True` or configure the `GOMEMLIMIT` environment variable. For details, see [Environment variables](../config-refs/env-vars/index.md). +To avoid out-of-memory issues during imports, set `LIMIT_RESOURCES` to `True` or configure the `GOMEMLIMIT` environment variable. For details, see [Environment variables](docs/deploy/config-guides/env-vars/index.md). ## Strategies to reduce memory usage @@ -144,7 +144,7 @@ Weaviate is optimized to work with Solid-State Disks (SSDs). However, spinning h ## File system -For optimal performance and reliability, avoid using `NFS` or similar file systems for the Weaviate persistent volume ([`PERSISTENCE_DATA_PATH`](../config-refs/env-vars/index.md)). +For optimal performance and reliability, avoid using `NFS` or similar file systems for the Weaviate persistent volume ([`PERSISTENCE_DATA_PATH`](docs/deploy/config-guides/env-vars/index.md)). Instead, use file systems like `Ext4` or `XFS` in combination with SAN storage (e.g. `EBS`) to ensure the best performance. diff --git a/docs/weaviate/concepts/search/vector-search.md b/docs/weaviate/concepts/search/vector-search.md index 20583b35..61bd542d 100644 --- a/docs/weaviate/concepts/search/vector-search.md +++ b/docs/weaviate/concepts/search/vector-search.md @@ -267,7 +267,7 @@ If you search a vector database containing vectors for colors "Red", "Crimson" a As a result, Weaviate provides multiple ways to limit the search results: - **Limit**: Specify the maximum number of results to return. - - If not provided, defaults to system-defined [`QUERY_DEFAULTS_LIMIT`](../../config-refs/env-vars/index.md#general) of 10. + - If not provided, defaults to system-defined [`QUERY_DEFAULTS_LIMIT`](docs/deploy/config-guides/env-vars/index.md#general) of 10. - **AutoCut**: Limit results based on discontinuities in result metrics such as vector distance or search score. - **Threshold**: Specify a minimum similarity score (e.g. maximum cosine distance) for the results. - **Apply filters**: Use [filters](../filtering.md) to exclude results based on other criteria, such as metadata or properties. diff --git a/docs/weaviate/config-refs/index.mdx b/docs/weaviate/config-refs/index.mdx index f314e200..7d33ec77 100644 --- a/docs/weaviate/config-refs/index.mdx +++ b/docs/weaviate/config-refs/index.mdx @@ -44,56 +44,9 @@ export const mainReferencesData = [ link: "/docs/weaviate/config-refs/distances", icon: "fas fa-ruler", }, - { - title: "Environment variables", - description: - "Configure Weaviate through environment variables and runtime settings.", - link: "/docs/weaviate/config-refs/env-vars", - icon: "fas fa-cogs", - }, -]; -export const otherReferencesData = [ - { - title: "Status", - description: "Monitor the status and health of your Weaviate instance.", - link: "/docs/weaviate/config-refs/status", - icon: "fas fa-heartbeat", - }, - { - title: "Telemetry", - description: - "Understand how Weaviate telemetry works and how to configure it.", - link: "/docs/weaviate/config-refs/telemetry", - icon: "fas fa-chart-line", - }, - { - title: "Cluster metadata", - description: "Access and manage metadata about your Weaviate cluster.", - link: "/docs/weaviate/config-refs/meta", - icon: "fas fa-info-circle", - }, - { - title: "Cluster node data", - description: "View and manage data about individual nodes in your cluster.", - link: "/docs/weaviate/config-refs/nodes", - icon: "fas fa-server", - }, - { - title: "OIDC Configuration", - description: - "Configure OpenID Connect for secure authentication with Weaviate.", - link: "/docs/weaviate/config-refs/oidc", - icon: "fas fa-lock", - }, ];

    - -## Other references - -
    - -
    diff --git a/docs/weaviate/config-refs/schema/index.md b/docs/weaviate/config-refs/schema/index.md index 9448629b..f9408f57 100644 --- a/docs/weaviate/config-refs/schema/index.md +++ b/docs/weaviate/config-refs/schema/index.md @@ -159,7 +159,7 @@ We are working on a re-indexing API to allow you to re-index the data after addi To ensure optimal performance, Weaviate **limits the number of collections per node**. Each collection adds overhead in terms of indexing, definition management, and storage. This limit aims to ensure Weaviate remains performant. - **Default limit**: `1000` collections. -- **Modify the limit**: Use the [`MAXIMUM_ALLOWED_COLLECTIONS_COUNT`](../../config-refs/env-vars/index.md) environment variable to adjust the collection count limit. +- **Modify the limit**: Use the [`MAXIMUM_ALLOWED_COLLECTIONS_COUNT`](docs/deploy/config-guides/env-vars/index.md) environment variable to adjust the collection count limit. :::note If your instance already exceeds the limit, Weaviate will not allow the creation of any new collections. Existing collections will not be deleted. @@ -385,7 +385,7 @@ These parameters are explained below: -[Replication](../../configuration/replication.md) configurations can be set using the definition, through the `replicationConfig` parameter. +[Replication](docs/deploy/config-guides/replication.md) configurations can be set using the definition, through the `replicationConfig` parameter. The `factor` parameter sets the number of copies of to be stored for objects in this collection. @@ -607,7 +607,7 @@ The `kagome_kr` tokenizer is not loaded by default to save resources. To use it, ### Limit the number of `gse` and `Kagome` tokenizers The `gse` and `Kagome` tokenizers can be resource intensive and affect Weaviate's performance. -You can limit the combined number of `gse` and `Kagome` tokenizers running at the same time using the [`TOKENIZER_CONCURRENCY_COUNT` environment variable](../env-vars/index.md). +You can limit the combined number of `gse` and `Kagome` tokenizers running at the same time using the [`TOKENIZER_CONCURRENCY_COUNT` environment variable](docs/deploy/config-guides/env-vars/index.md). ### Inverted index types diff --git a/docs/weaviate/config-refs/schema/vector-index.md b/docs/weaviate/config-refs/schema/vector-index.md index 2dc7f9f7..195e4ffd 100644 --- a/docs/weaviate/config-refs/schema/vector-index.md +++ b/docs/weaviate/config-refs/schema/vector-index.md @@ -54,7 +54,7 @@ Tombstones are records that mark deleted objects. In an HNSW index, tombstones a As the index grows in size, the cleanup process may take longer to complete and require more resources. For very large indexes, this may cause performance issues. -To control the number of tombstones deleted per cleanup cycle and prevent performance issues, set the [`TOMBSTONE_DELETION_MAX_PER_CYCLE` and `TOMBSTONE_DELETION_MIN_PER_CYCLE` environment variables](../env-vars/index.md#general). +To control the number of tombstones deleted per cleanup cycle and prevent performance issues, set the [`TOMBSTONE_DELETION_MAX_PER_CYCLE` and `TOMBSTONE_DELETION_MIN_PER_CYCLE` environment variables](docs/deploy/config-guides/env-vars/index.md#general). - Set `TOMBSTONE_DELETION_MIN_PER_CYCLE` to prevent occurrences of unnecessary cleanup cycles. - Set `TOMBSTONE_DELETION_MAX_PER_CYCLE` to prevent the cleanup process from taking too long and consuming too many resources. @@ -219,7 +219,7 @@ services:
    -To get the index status, check the [node status](/docs/weaviate/config-refs/nodes) endpoint. +To get the index status, check the [node status](docs/deploy/config-guides/nodes.md) endpoint.
    Node status example usage diff --git a/docs/weaviate/configuration/_enterprise-usage-collector.md b/docs/weaviate/configuration/_enterprise-usage-collector.md index fd407ed3..deb3b143 100644 --- a/docs/weaviate/configuration/_enterprise-usage-collector.md +++ b/docs/weaviate/configuration/_enterprise-usage-collector.md @@ -51,7 +51,7 @@ services: ## Using the Docker Compose configurator -You can also use the Docker compose [configurator](/docs/weaviate/installation/docker-compose.md#configurator). Make sure to select `Enabled` for the Enterprise Usage Collector option. +You can also use the Docker compose [configurator](docs/deploy/installation-guides/docker-installation.md#configurator). Make sure to select `Enabled` for the Enterprise Usage Collector option. ## Collector proxy on Kubernetes using Helm diff --git a/docs/weaviate/configuration/authentication.md b/docs/weaviate/configuration/authentication.md index 7a06bbd9..e47da4bc 100644 --- a/docs/weaviate/configuration/authentication.md +++ b/docs/weaviate/configuration/authentication.md @@ -9,7 +9,7 @@ image: og/docs/configuration.jpg Authentication and authorization are closely related concepts, and sometimes abbreviated as `AuthN` and `AuthZ`. Authentication (`AuthN`) is the process of verifying the identity of a user, while authorization (`AuthZ`) is the process of determining what permissions the user has. ::: -Weaviate controls access through user authentication via API keys or OpenID Connect (OIDC), with an option for anonymous access. Users can then be assigned different [authorization](./authorization.md) levels, as shown in the diagram below. +Weaviate controls access through user authentication via API keys or OpenID Connect (OIDC), with an option for anonymous access. Users can then be assigned different [authorization](docs/deploy/config-guides/authorization.md) levels, as shown in the diagram below. ```mermaid flowchart LR @@ -70,279 +70,6 @@ The way to configure authentication differs by your deployment method, depending For Weaviate Cloud (WCD) instances, authentication is pre-configured with OIDC and API key access. You can [authenticate against Weaviate](../connections/connect-cloud.mdx) with your WCD credentials using OIDC, or [with API keys](/docs/cloud/platform/manage-api-keys). ::: -## API Key Authentication - -API key authentication is a simple and effective way to authenticate users. Each user is assigned a unique API key, which is used to authenticate the user. - -### API keys: Database users - -When [creating database users programatically](./rbac/manage-users.mdx#create-a-user), each user is assigned a distinct API key at creation time. These API keys can also be [regenerated (rotated)](./rbac/manage-users.mdx#rotate-user-api-key). - -### API keys: Docker - -API key authentication can be configured using environment variables. In Docker Compose, set them in the configuration file (`docker-compose.yml`) such as in the following example: - -```yaml -services: - weaviate: - ... - environment: - ... - # Disable anonymous access. - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'false' - - # Enables API key authentication. - AUTHENTICATION_APIKEY_ENABLED: 'true' - - # List one or more keys in plaintext separated by commas. Each key corresponds to a specific user identity below. - AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'user-a-key,user-b-key' - - # List one or more user identities, separated by commas. Each identity corresponds to a specific key above. - AUTHENTICATION_APIKEY_USERS: 'user-a,user-b' -``` - -This configuration: -- Disables anonymous access -- Enables API key authentication -- Defines plaintext API keys in `AUTHENTICATION_APIKEY_ALLOWED_KEYS` -- Associates users with the API keys in `AUTHENTICATION_APIKEY_USERS` - -These users can now be assigned permissions based on the authorization settings. - -import DynamicUserManagement from '/_includes/configuration/dynamic-user-management.mdx'; - - - -:::note - -Note that you can either: -- Set one user for all API keys, or -- Define one user per API key (the number of users must match the number of API keys) - -Make sure all listed users are also configured in the authorization settings. - -::: - -### API keys: Kubernetes - -For Kubernetes deployments using Helm, API key authentication can be configured in the `values.yaml` file under the `authentication` section. Here's an example configuration: - -```yaml -authentication: - anonymous_access: - # Disable anonymous access. - enabled: false - - apikey: - # Enables API key authentication. - enabled: true - - # List one or more keys in plaintext separated by commas. Each key corresponds to a specific user identity below. - allowed_keys: - - user-a-key - - user-b-key - - # List one or more user identities, separated by commas. Each identity corresponds to a specific key above. - users: - - user-a - - user-b -``` - -This configuration: -- Disables anonymous access -- Enables API key authentication -- Defines plaintext API keys in `allowed_keys` -- Associates users with the API keys in `users` - -:::warning Environment Variables Take Precedence -If you configure API keys using environment variables, those settings will take precedence over the values in `values.yaml`. To use the Helm values configuration, ensure you have not set the corresponding environment variables. -::: - -For enhanced security in production environments, you can store API keys in Kubernetes secrets and reference them using environment variables instead of storing them as plaintext in the Helm values. - -## OIDC Authentication - -OIDC authentication requires the resource (Weaviate) to validate tokens issued by an identity provider. The identity provider authenticates the user and issues tokens, which are then validated by Weaviate. - -In an example setup, a Weaviate instance acts as the resource, Weaviate Cloud (WCD) acts as the identity provider, and the Weaviate client acts on behalf of the user. - -Any "OpenID Connect" compatible token issuer that implements OpenID Connect Discovery is compatible with Weaviate. - -This document discusses how to configure Weaviate as the resource. - -
    - - More about OIDC - - -With [OpenID Connect](https://openid.net/connect/) (based on OAuth2), an -external identity provider and token issuer ('token issuer' hereafter) is responsible for managing users. - -OIDC authentication requires obtaining a valid token from the token issuer so that it can be sent in the header of any request to Weaviate. This applies to both REST and GraphQL requests. - -When Weaviate receives a token (JSON Web Token or JWT), it verifies -that it was indeed signed by the configured token issuer. If the signature is -correct, all contents of the token are trusted, which authenticates the user based on the information in the token. - -
    - -:::tip TIP: OIDC and RBAC - -The [user management API](./rbac/manage-users.mdx#oidc-user-permissions-management) allows you to assign custom roles and permissions to OIDC users via [Role-Based Access Control (RBAC)](./rbac/index.mdx). - -::: - -### OIDC: Docker - -To configure Weaviate for OIDC-based authentication, add the following environment variables to your configuration file. - -An example `docker-compose.yml` file looks like this: - -```yaml -services: - weaviate: - ... - environment: - ... - # enabled (optional - defaults to false) turns OIDC auth on. All other fields in - # this section will only be validated if enabled is set to true. - AUTHENTICATION_OIDC_ENABLED: 'true' - - # issuer (required) tells weaviate how to discover the token issuer. This - # endpoint must implement the OpenID Connect Discovery spec, so that weaviate - # can retrieve the issuer's public key. - # - # The example URL below uses the path structure commonly found with keycloak - # where an example realm 'my-weaviate-usecase' was created. The exact - # path structure depends on the token issuer. See the token issuer's documentation - # about which endpoint implements OIDC Discovery. - AUTHENTICATION_OIDC_ISSUER: 'http://my-token-issuer/auth/realms/my-weaviate-usecase' - - # client_id (required unless skip_client_id_check is set to true) tells - # Weaviate to check for a particular OAuth 2.0 client_id in the audience claim. - # This is to prevent that a token which was signed by the correct issuer - # but never intended to be used with Weaviate can be used for authentication. - # - # For more information on what clients are in OAuth 2.0, see - # https://tools.ietf.org/html/rfc6749#section-1.1 - AUTHENTICATION_OIDC_CLIENT_ID: 'my-weaviate-client' - - # username_claim (required) tells Weaviate which claim in the token to use for extracting - # the username. The username will be passed to the authorization plugin. - AUTHENTICATION_OIDC_USERNAME_CLAIM: 'email' - - # skip_client_id_check (optional, defaults to false) skips the client_id - # validation in the audience claim as outlined in the section above. - # Not recommended to set this option as it reduces security, only set this - # if your token issuer is unable to provide a correct audience claim - AUTHENTICATION_OIDC_SKIP_CLIENT_ID_CHECK: 'false' - - # scope (optional) these will be used by clients as default scopes for authentication - AUTHENTICATION_OIDC_SCOPES: '' -``` - -:::info OIDC and Azure -As of November 2022, we were aware of some differences in Microsoft Azure's OIDC implementation compared to others. If you are using Azure and experiencing difficulties, [this external blog post](https://xsreality.medium.com/making-azure-ad-oidc-compliant-5734b70c43ff) may be useful. -::: - -### OIDC: Kubernetes - -For Kubernetes deployments using Helm, OIDC authentication can be configured in the `values.yaml` file under the `authentication` section. Here's an example configuration: - -```yaml -authentication: - anonymous_access: - # Disable anonymous access. - enabled: false - oidc: - # enabled (optional - defaults to false) turns OIDC auth on. All other fields in - # this section will only be validated if enabled is set to true. - enabled: true - - # issuer (required) tells weaviate how to discover the token issuer. This - # endpoint must implement the OpenID Connect Discovery spec, so that weaviate - # can retrieve the issuer's public key. - # - # The example URL below uses the path structure commonly found with keycloak - # where an example realm 'my-weaviate-usecase' was created. The exact - # path structure depends on the token issuer. See the token issuer's documentation - # about which endpoint implements OIDC Discovery. - issuer: 'http://my-token-issuer/auth/realms/my-weaviate-usecase' - - # client_id (required unless skip_client_id_check is set to true) tells - # Weaviate to check for a particular OAuth 2.0 client_id in the audience claim. - # This is to prevent that a token which was signed by the correct issuer - # but never intended to be used with Weaviate can be used for authentication. - # - # For more information on what clients are in OAuth 2.0, see - # https://tools.ietf.org/html/rfc6749#section-1.1 - client_id: 'my-weaviate-client' - - # username_claim (required) tells Weaviate which claim in the token to use for extracting - # the username. The username will be passed to the authorization plugin. - username_claim: 'email' - - # skip_client_id_check (optional, defaults to false) skips the client_id - # validation in the audience claim as outlined in the section above. - # Not recommended to set this option as it reduces security, only set this - # if your token issuer is unable to provide a correct audience claim - skip_client_id_check: 'false' - - # scope (optional) these will be used by clients as default scopes for authentication - scopes: '' - - # groups_claim: '' -``` - -### Note: Configuring the OIDC token issuer - -Configuring the OIDC token issuer is outside the scope of this document, but here are a few options as a starting point: - -- For simple use-cases such as for a single user, you can use Weaviate Cloud (WCD) as the OIDC token issuer. To do so: - - Make sure you have a WCD account (you can [sign up here](https://console.weaviate.cloud/)). - - In the Docker Compose file (e.g. `docker-compose.yml`), specify: - - `https://auth.wcs.api.weaviate.io/auth/realms/SeMI` as the issuer (in `AUTHENTICATION_OIDC_ISSUER`), - - `wcs` as the client id (in `AUTHENTICATION_OIDC_CLIENT_ID`), and - - enable the adminlist (`AUTHORIZATION_ADMINLIST_ENABLED: 'true'`) and add your WCD account email as the user (in `AUTHORIZATION_ADMINLIST_USERS`) . - - `email` as the username claim (in `AUTHENTICATION_OIDC_USERNAME_CLAIM`). - -- If you need a more customizable setup you can use commercial OIDC providers like [Okta](https://www.okta.com/). -- As another alternative, you can run your own OIDC token issuer server, which may be the most complex but also configurable solution. Popular open-source solutions include Java-based [Keycloak](https://www.keycloak.org/) and Golang-based [dex](https://github.com/dexidp/dex). - -:::info -By default, Weaviate validates that the token includes a specified client id in the audience claim. If your token issuer does not support this feature, you can turn it off as outlined in the configuration section below. -::: - -## Anonymous Access - -Weaviate can be configured to accept anonymous requests. This is strongly discouraged except for development or evaluation purposes. - -Users that send requests without explicit authentication are authenticated as `user: anonymous`. - -You can use the authorization plugin to specify which permissions to apply to this `anonymous` user. If anonymous access is disabled altogether, any request without an allowed authentication scheme returns `401 Unauthorized`. - -### Anonymous access: Docker - -To enable anonymous access in Docker Compose, add the following environment variable to your configuration file: - -```yaml -services: - weaviate: - ... - environment: - ... - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' -``` - -### Anonymous access: Kubernetes - -To enable anonymous access in Kubernetes, add the following configuration to your `values.yaml` file: - -```yaml -authentication: - anonymous_access: - enabled: true -``` ## Client-side Usage @@ -491,8 +218,8 @@ curl https://localhost:8080/v1/objects -H "Authorization: Bearer ${WEAVIATE_API_ ## Further resources -- [Configuration: Authorization and RBAC](./authorization.md) -- [References: Environment variables / Authentication and Authorization](../config-refs/env-vars/index.md#authentication-and-authorization) +- [Configuration: Authorization and RBAC](docs/deploy/config-guides/authorization.md) +- [References: Environment variables / Authentication and Authorization](docs/deploy/config-guides/env-vars/index.md#authentication-and-authorization) ## Questions and feedback diff --git a/docs/weaviate/configuration/index.md b/docs/weaviate/configuration/index.md new file mode 100644 index 00000000..13ec6e8b --- /dev/null +++ b/docs/weaviate/configuration/index.md @@ -0,0 +1,22 @@ +--- +title: How to configure Weaviate +sidebar_position: 0 +image: og/docs/configuration.jpg +# tags: ['configuration'] +--- + +This section shows you how to configure Weaviate. + +- Extend Weaviate's functionality by adding vectorizers and other [modules](./modules.md) +- Configure how Weaviate stores and indexes data + + - [schema](docs/weaviate/manage-collections/index.mdx) + - [data types](../config-refs/datatypes.md) + - [distance metrics](../config-refs/distances.md) + +- Manage performance and cost with [vector index properties](/docs/weaviate/config-refs/schema/vector-index) +- [Back up](docs/deploy/config-guides/backups.md) your Weaviate instance +- Control access + - [authentication](./authentication.md) + - [authorization](docs/deploy/config-guides/authorization.md) +- [Monitor](docs/deploy/config-guides/monitoring.md) your Weaviate instance diff --git a/docs/weaviate/configuration/index.mdx b/docs/weaviate/configuration/index.mdx index 31895ec6..55ab4ebd 100644 --- a/docs/weaviate/configuration/index.mdx +++ b/docs/weaviate/configuration/index.mdx @@ -50,7 +50,7 @@ export const configOpsData = [ title: "Authorization", description: "Understand Weaviate's authorization schemes (RBAC, Admin List) to control user permissions.", - link: "/docs/weaviate/configuration/authorization", + link: "/docs/deploy/config-guides/authorization.md", icon: "fas fa-shield-alt", }, { @@ -71,7 +71,7 @@ export const configOpsData = [ title: "Monitoring", description: "Enable and configure Prometheus-compatible metrics for observing instance health.", - link: "/docs/weaviate/configuration/monitoring", + link: "/docs/deploy/config-guides/monitoring", icon: "fas fa-chart-line", }, { diff --git a/docs/weaviate/configuration/modules.md b/docs/weaviate/configuration/modules.md index 3e4ec23d..f007c1ce 100644 --- a/docs/weaviate/configuration/modules.md +++ b/docs/weaviate/configuration/modules.md @@ -15,7 +15,7 @@ At the instance (i.e. Weaviate cluster) level, you can: - Configure the default vectorizer module - Configure module-specific variables (e.g. API keys), where applicable -This can be done by setting the appropriate [environment variables](../config-refs/env-vars/index.md) as shown below. +This can be done by setting the appropriate [environment variables](docs/deploy/config-guides/env-vars/index.md) as shown below. :::tip What about WCD? Weaviate Cloud (WCD) instances come with modules pre-configured. See [this page](/docs/cloud/manage-clusters/status#enabled-modules) for details. diff --git a/docs/weaviate/configuration/monitoring.md b/docs/weaviate/configuration/monitoring.md deleted file mode 100644 index f1cf0229..00000000 --- a/docs/weaviate/configuration/monitoring.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -title: Monitoring -sidebar_position: 50 -image: og/docs/configuration.jpg -# tags: ['configuration', 'operations', 'monitoring', 'observability'] ---- - -Weaviate can expose Prometheus-compatible metrics for monitoring. A standard -Prometheus/Grafana setup can be used to visualize metrics on various -dashboards. - -Metrics can be used to measure request latencies, import -speed, time spent on vector vs object storage, memory usage, application usage, -and more. - -## Configure Monitoring - -### Enable within Weaviate - -To tell Weaviate to collect metrics and expose them in a Prometheus-compatible -format, all that's required is to set the following environment variable: - -```sh -PROMETHEUS_MONITORING_ENABLED=true -``` - -By default, Weaviate will expose the metrics at `:2112/metrics`. You -can optionally change the port to a custom port using the following environment -variable: - -```sh -PROMETHEUS_MONITORING_PORT=3456 -``` - -### Scrape metrics from Weaviate - -Metrics are typically scraped into a time-series database, such as Prometheus. -How you consume metrics depends on your setup and environment. - -The [Weaviate examples repo contains a fully pre-configured setup using -Prometheus, Grafana and some example -dashboards](https://github.com/weaviate/weaviate-examples/tree/main/monitoring-prometheus-grafana). -You can start up a full-setup including monitoring and dashboards with a single -command. In this setup the following components are used: - -- Docker Compose is used to provide a fully-configured setup that can be - started with a single command. -- Weaviate is configured to expose Prometheus metrics as outlined in the - section above. -- A Prometheus instance is started with the setup and configured to scrape - metrics from Weaviate every 15s. -- A Grafana instance is started with the setup and configured to use the - Prometheus instance as a metrics provider. Additionally, it runs a dashboard - provider that contains a few sample dashboards. - -### Multi-tenancy - -When using multi-tenancy, we suggest setting the `PROMETHEUS_MONITORING_GROUP` [environment variable](../config-refs/env-vars/index.md) as `true` so that data across all tenants are grouped together for monitoring. - -## Obtainable Metrics - -The list of metrics that are obtainable through Weaviate's metric system is -constantly being expanded. The complete list is in the [`prometheus.go`](https://github.com/weaviate/weaviate/blob/main/usecases/monitoring/prometheus.go) source code file. - -This page describes some noteworthy metrics and their uses. - -Typically metrics are quite granular, as they can always be aggregated later -on. For example if the granularity is "shard", you could aggregate all "shard" -metrics of the same "class" to obtain a class metrics, or aggregate all metrics -to obtain the metric for the entire Weaviate instance. - -| Metric | Description | Labels | Type | -| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ----------- | -| `async_operations_running` | Number of currently running async operations. The operation itself is defined through the `operation` label. | `operation`, `class_name`, `shard_name`, `path` | `Gauge` | -| `batch_delete_durations_ms` | Duration of a batch delete in ms. The `operation` label further defines what operation as part of the batch delete is being measured. Granularity is a shard of a class | `class_name`, `shard_name` | `Histogram` | -| `batch_durations_ms` | Duration of a single batch operation in ms. The `operation` label further defines what operation as part of the batch (e.g. object, inverted, vector) is being used. Granularity is a shard of a class. | `operation`, `class_name`, `shard_name` | `Histogram` | -| `index_queue_delete_duration_ms` | Duration of deleting one or more vectors from the index queue and the underlying index. | `class_name`, `shard_name`, `target_vector` | `Summary` | -| `index_queue_paused` | Whether the index queue is paused. | `class_name`, `shard_name`, `target_vector` | `Gauge` | -| `index_queue_preload_count` | Number of vectors preloaded to the index queue. | `class_name`, `shard_name`, `target_vector` | `Gauge` | -| `index_queue_preload_duration_ms` | Duration of preloading un-indexed vectors to the index queue. | `class_name`, `shard_name`, `target_vector` | `Summary` | -| `index_queue_push_duration_ms` | Duration of pushing one or more vectors to the index queue. | `class_name`, `shard_name`, `target_vector` | `Summary` | -| `index_queue_search_duration_ms` | Duration of searching for vectors in the index queue and the underlying index. | `class_name`, `shard_name`, `target_vector` | `Summary` | -| `index_queue_size` | Number of vectors in the index queue. | `class_name`, `shard_name`, `target_vector` | `Gauge` | -| `index_queue_stale_count` | Number of times the index queue has been marked as stale. | `class_name`, `shard_name`, `target_vector` | `Counter` | -| `index_queue_vectors_dequeued` | Number of vectors sent to the workers per tick. | `class_name`, `shard_name`, `target_vector` | `Gauge` | -| `index_queue_wait_duration_ms` | Duration of waiting for the workers to finish. | `class_name`, `shard_name`, `target_vector` | `Summary` | -| `lsm_active_segments` | Number of currently present segments per shard. Granularity is shard of a class. Grouped by `strategy`. | `strategy`, `class_name`, `shard_name`, `path` | `Gauge` | -| `lsm_bloom_filter_duration_ms` | Duration of a bloom filter operation per shard in ms. Granularity is shard of a class. Grouped by `strategy`. | `operation`, `strategy`, `class_name`, `shard_name` | `Histogram` | -| `lsm_segment_count` | Number of segments by level | `strategy`, `class_name`, `shard_name`, `path`, `level` | `Gauge` | -| `lsm_segment_objects` | Number of entries per LSM segment by level. Granularity is shard of a class. Grouped by `strategy` and `level`. | `operation`, `strategy`, `class_name`, `shard_name`, `path`, `level` | `Gauge` | -| `lsm_segment_size` | Size of LSM segment by level and unit. | `strategy`, `class_name`, `shard_name`, `path`, `level`, `unit` | `Gauge` | -| `object_count` | Numbers of objects present. Granularity is a shard of a class | `class_name`, `shard_name` | `Gauge` | -| `objects_durations_ms` | Duration of an individual object operation, such as `put`, `delete`, etc. as indicated by the `operation` label, also as part of a batch. The `step` label adds additional precisions to each `operation`. Granularity is a shard of a class. | `class_name`, `shard_name` | `Histogram` | -| `requests_total` | Metric that tracks all user requests to determine if it was successful or failed. | `api`, `query_type`, `class_name` | `GaugeVec` | -| `startup_diskio_throughput` | Disk I/O throughput in bytes/s at startup operations, such as reading back the HNSW index or recovering LSM segments. The operation itself is defined by the `operation` label. | `operation`, `step`, `class_name`, `shard_name` | `Histogram` | -| `startup_durations_ms` | Duration of individual startup operations in ms. The operation itself is defined through the `operation` label. | `operation`, `class_name`, `shard_name` | `Histogram` | -| `vector_index_durations_ms` | Duration of regular vector index operation, such as insert or delete. The operation itself is defined through the `operation` label. The `step` label adds more granularity to each operation. | `operation`, `step`, `class_name`, `shard_name` | `Histogram` | -| `vector_index_maintenance_durations_ms` | Duration of a sync or async vector index maintenance operation. The operation itself is defined through the `operation` label. | `opeartion`, `class_name`, `shard_name` | `Histogram` | -| `vector_index_operations` | Total number of mutating operations on the vector index. The operation itself is defined by the `operation` label. | `operation`, `class_name`, `shard_name` | `Gauge` | -| `vector_index_size` | The total capacity of the vector index. Typically larger than the number of vectors imported as it grows proactively. | `class_name`, `shard_name` | `Gauge` | -| `vector_index_tombstone_cleaned` | Total number of deleted and removed vectors after repair operations. | `class_name`, `shard_name` | `Counter` | -| `vector_index_tombstone_cleanup_threads` | Number of currently active threads for repairing/cleaning up the vector index after deletes have occurred. | `class_name`, `shard_name` | `Gauge` | -| `vector_index_tombstones` | Number of currently active tombstones in the vector index. Will go up on each incoming delete and go down after a completed repair operation. | `class_name`, `shard_name` | `Gauge` | -| `weaviate_runtime_config_hash` | Hash value of the currently active runtime configuration, useful for tracking when new configurations take effect. | `sha256` | `GaugeVec` | -| `weaviate_runtime_config_last_load_success` | Indicates whether the last loading attempt was successful (`1` for success, `0` for failure). | | `Gauge` | - -Extending Weaviate with new metrics is very easy. To suggest a new metric, see the [contributor guide](/docs/contributor-guide). - -### Versioning - -Be aware that metrics do not follow the semantic versioning guidelines of other Weaviate features. Weaviate's main APIs are stable and breaking changes are extremely rare. Metrics, however, have shorter feature lifecycles. It can sometimes be necessary to introduce an incompatible change or entirely remove a metric, for example, because the cost of observing a specific metric in production has grown too high. As a result, it is possible that a Weaviate minor release contains a breaking change for the Monitoring system. If so, it will be clearly highlighted in the release notes. - -## Sample Dashboards - -Weaviate does not ship with any dashboards by default, but here is a list of -dashboards being used by the various Weaviate teams, both during development, -and when helping users. These do not come with any support, but may still be -helpful. Treat them as inspiration to design your own dashboards which fit -your uses perfectly: - -| Dashboard | Purpose | Preview | -| ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | -| [Importing Data Into Weaviate](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/importing.json) | Visualize speed of import operations (including its components, such as object store, inverted index, and vector index). | ![Importing Data into Weaviate](./img/weaviate-sample-dashboard-importing.png 'Importing Data Into Weaviate') | -| [Object Operations](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/objects.json) | Visualize speed of whole object operations, such as GET, PUT, etc. | ![Objects](./img/weaviate-sample-dashboard-objects.png 'Objects') | -| [Vector Index](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/vectorindex.json) | Visualize the current state, as well as operations on the HNSW vector index | ![Vector Index](./img/weaviate-sample-dashboard-vector.png 'Vector Index') | -| [LSM Stores](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/lsm.json) | Get insights into the internals (including segments) of the various LSM stores within Weaviate. | ![LSM Store](./img/weaviate-sample-dashboard-lsm.png 'LSM Store') | -| [Startup](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/startup.json) | Visualize the startup process, including recovery operations | ![Startup](./img/weaviate-sample-dashboard-startup.png 'Vector Index') | -| [Usage](https://github.com/weaviate/weaviate/blob/master/tools/dev/grafana/dashboards/usage.json) | Obtain usage metrics, such as number of objects imported, etc. | ![Usage](./img/weaviate-sample-dashboard-usage.png 'Usage') | -| [Aysnc index queue](https://github.com/weaviate/weaviate/blob/main/tools/dev/grafana/dashboards/index_queue.json) | Observe index queue activity | ![Async index queue](./img/weaviate-sample-dashboard-async-queue.png 'Async index queue') | - -## `nodes` API Endpoint - -To get collection details programmatically, use the [`nodes`](../config-refs/nodes.md) REST endpoint. - -import APIOutputs from '/\_includes/rest/node-endpoint-info.mdx'; - - - -## Questions and feedback - -import DocsFeedback from '/\_includes/docs-feedback.mdx'; - - diff --git a/docs/weaviate/configuration/rbac/index.mdx b/docs/weaviate/configuration/rbac/index.mdx index b953360e..a09a1d7e 100644 --- a/docs/weaviate/configuration/rbac/index.mdx +++ b/docs/weaviate/configuration/rbac/index.mdx @@ -12,7 +12,7 @@ import SkipLink from '/src/components/SkipValidationLink' Role-based access control (RBAC) is generally available in Weaviate from version `v1.29`. ::: -Weaviate provides differentiated access through [authorization](/docs/weaviate/configuration/authorization) levels based on the [authenticated](/docs/weaviate/configuration/authentication) user identity. +Weaviate provides differentiated access through [authorization](docs/deploy/config-guides/authorization.md) levels based on the [authenticated](docs/deploy/config-guides/authentication.md) user identity. If **role-based access control (RBAC)** is enabled, access can be further restricted based on the roles of users. The diagram below illustrates the RBAC model in Weaviate, where access is governed by defining roles and assigning them specific permissions. These permissions determine which actions users can perform on designated resource types. @@ -100,11 +100,11 @@ Weaviate comes with a set of predefined roles. These roles are: - `root`: The root role has **full access** to all resources in Weaviate. - `viewer`: The viewer role has **read-only access** to all resources in Weaviate. -The `root` role can be assigned to a user through the Weaviate configuration file using the [`AUTHORIZATION_RBAC_ROOT_USERS`](/docs/weaviate/config-refs/env-vars#rbac-authorization) environment variable. A predefined role cannot be modified. The user can, however, be assigned additional roles through the Weaviate API. +The `root` role can be assigned to a user through the Weaviate configuration file using the [`AUTHORIZATION_RBAC_ROOT_USERS`](docs/deploy/config-guides/env-vars/index.md#rbac-authorization) environment variable. A predefined role cannot be modified. The user can, however, be assigned additional roles through the Weaviate API. All roles can also be assigned through the Weaviate API, including the predefined role. The predefined roles cannot be modified, but they can be assigned to or revoked from users. -Refer to the [RBAC: Configuration](/docs/weaviate/configuration/rbac/configuration) page for more information on how to assign predefined roles to users. +Refer to the [RBAC: Configuration](docs/deploy/config-guides/configuring-rbac.md) page for more information on how to assign predefined roles to users. ### Custom roles @@ -112,7 +112,7 @@ Any authenticated user that is not assigned a predefined role has no roles or pe These users' permissions can be modified through Weaviate by those with the appropriate permissions for **managing roles**. This allows for the creation of custom roles, which can be assigned to users as needed. -Role management can be performed with a [predefined `root` role](/docs/weaviate/configuration/rbac/configuration) or a custom role with [`manage_roles` permissions](/docs/weaviate/configuration/rbac/manage-roles#role-management-permissions). +Role management can be performed with a [predefined `root` role](docs/deploy/config-guides/configuring-rbac.md) or a custom role with [`manage_roles` permissions](/docs/weaviate/configuration/rbac/manage-roles#role-management-permissions). :::caution Role Management Permissions Be careful when assigning permissions to roles that manage roles. These permissions can be used to escalate privileges by assigning additional roles to users. Only assign these permissions to trusted users. @@ -354,10 +354,10 @@ The [user management](./manage-users.mdx) API can be used to create, delete and ## Further resources -- [RBAC: Configuration](./configuration.md) +- [RBAC: Configuration](docs/deploy/config-guides/configuring-rbac.md) - [RBAC: Manage roles](./manage-roles.mdx) - [RBAC: Manage users](./manage-users.mdx) -- [RBAC: Tutorial](../../tutorials/rbac.mdx) +- [RBAC: Tutorial](docs/deploy/tutorials/rbac.mdx) ## Questions and feedback diff --git a/docs/weaviate/configuration/rbac/manage-roles.mdx b/docs/weaviate/configuration/rbac/manage-roles.mdx index fc86285b..027d23e3 100644 --- a/docs/weaviate/configuration/rbac/manage-roles.mdx +++ b/docs/weaviate/configuration/rbac/manage-roles.mdx @@ -31,7 +31,7 @@ import ConfigureRbac from '/_includes/configuration/configure-rbac.mdx'; Role management requires appropriate `role` resource permissions that can be obtained through: -- A predefined `root` role when [configuring RBAC](/docs/weaviate/configuration/rbac/configuration). +- A predefined `root` role when [configuring RBAC](docs/deploy/config-guides/configuring-rbac.md). - A role with [`Role Management`](#role-management-permissions) permissions granted. @@ -638,7 +638,7 @@ Visit the [Manage users](./manage-users.mdx) page to learn more about assigning ## Further resources - [RBAC: Overview](./index.mdx) -- [RBAC: Configuration](./configuration.md) +- [RBAC: Configuration](docs/deploy/config-guides/configuring-rbac.md) - [RBAC: Manage users](./manage-users.mdx) ## Questions and feedback diff --git a/docs/weaviate/configuration/rbac/manage-users.mdx b/docs/weaviate/configuration/rbac/manage-users.mdx index fe719a9b..7bc5597e 100644 --- a/docs/weaviate/configuration/rbac/manage-users.mdx +++ b/docs/weaviate/configuration/rbac/manage-users.mdx @@ -350,7 +350,7 @@ viewer ## OIDC users: Permissions management {#oidc-user-permissions-management} -When using [OIDC](../../config-refs/oidc.md), an identity provider authenticates the user and issues tokens, which are then validated by Weaviate. These users can be assigned roles with custom permissions using RBAC. +When using [OIDC](docs/deploy/config-guides/oidc.md), an identity provider authenticates the user and issues tokens, which are then validated by Weaviate. These users can be assigned roles with custom permissions using RBAC. ### Assign a role to an OIDC user @@ -480,7 +480,7 @@ viewer ## Further resources - [RBAC: Overview](./index.mdx) -- [RBAC: Configuration](./configuration.md) +- [RBAC: Configuration](docs/deploy/config-guides/configuring-rbac.md) - [RBAC: Manage roles](./manage-roles.mdx) ## Questions and feedback diff --git a/docs/weaviate/index.mdx b/docs/weaviate/index.mdx index a066ee67..e916c738 100644 --- a/docs/weaviate/index.mdx +++ b/docs/weaviate/index.mdx @@ -124,7 +124,7 @@ export const deploymentCardsData = [ ], button: { text: "Run Weaviate with Docker", - link: "/docs/weaviate/installation/docker-compose", + link: "/docs/deploy/installation-guides/docker-installation", }, }, { @@ -146,7 +146,7 @@ export const deploymentCardsData = [ ], button: { text: "Run Weaviate with Kubernetes", - link: "/docs/weaviate/installation/kubernetes", + link: "/docs/deploy/installation-guides/k8s-installation", }, }, { diff --git a/docs/weaviate/installation/index.md b/docs/weaviate/installation/index.md index 8c697cdf..ad5815ab 100644 --- a/docs/weaviate/installation/index.md +++ b/docs/weaviate/installation/index.md @@ -7,21 +7,21 @@ image: og/docs/installation.jpg Weaviate is available as a hosted service, [Weaviate Cloud (WCD)](https://console.weaviate.cloud/), or as a self managed instance. If you manage your own instance, you can host it locally or with a cloud provider. Self-managed instances use the same Weaviate Database as WCD. -If you are upgrading from a previous version of Weaviate, see the [Migration Guide](../more-resources/migration/index.md) for any changes that may affect your installation. +If you are upgrading from a previous version of Weaviate, see the [Migration Guide](docs/deploy/migration/index.md) for any changes that may affect your installation. ## Installation methods To install and configure Weaviate, see the following: - **[Weaviate Cloud](../../cloud/quickstart.mdx)**: Managed services for development and production environments. -- **[Docker Compose](./docker-compose.md)**: Docker containers are well suited for development and testing. -- **[Kubernetes](./kubernetes.md)**: Kubernetes is ideal for scalable, production deployments. +- **[Docker Compose](docs/deploy/installation-guides/docker-installation.md)**: Docker containers are well suited for development and testing. +- **[Kubernetes](docs/deploy/installation-guides/k8s-installation.md)**: Kubernetes is ideal for scalable, production deployments. - **[Snowpark Container Services](./spcs-integration.mdx)** Deploy Weaviate in Snowflake's Snowpark environment. - **[Embedded Weaviate](./embedded.md)**: Experimental. Embedded Weaviate is a client based tool. ## Configuration files -Docker Compose and Kubernetes use yaml files to configure Weaviate instances. Docker uses the [`docker-compose.yml`](./docker-compose.md) file. Kubernetes relies on [Helm charts](./kubernetes.md#weaviate-helm-chart) and the `values.yaml` file. The Weaviate documentation also calls these files `configuration yaml files`. +Docker Compose and Kubernetes use yaml files to configure Weaviate instances. Docker uses the [`docker-compose.yml`](docs/deploy/installation-guides/docker-installation.md) file. Kubernetes relies on [Helm charts](docs/deploy/installation-guides/k8s-installation.md#weaviate-helm-chart) and the `values.yaml` file. The Weaviate documentation also calls these files `configuration yaml files`. If you are self-hosting, consider experimenting on a small scale with Docker and then transferring your configuration to Kubernetes Helm charts when you are more familiar with Weaviate. diff --git a/docs/weaviate/manage-collections/multi-node-setup.mdx b/docs/weaviate/manage-collections/multi-node-setup.mdx index 6321be86..08226252 100644 --- a/docs/weaviate/manage-collections/multi-node-setup.mdx +++ b/docs/weaviate/manage-collections/multi-node-setup.mdx @@ -21,7 +21,7 @@ import RaftRFChangeWarning from "/_includes/1-25-replication-factor.mdx"; -Configure replication settings, such as [async replication](/docs/weaviate/configuration/replication#async-replication-settings) and [deletion resolution strategy](../concepts/replication-architecture/consistency.md#deletion-resolution-strategies). +Configure replication settings, such as [async replication](docs/deploy/config-guides/replication.md#async-replication-settings) and [deletion resolution strategy](../concepts/replication-architecture/consistency.md#deletion-resolution-strategies). @@ -96,7 +96,7 @@ http://localhost:8080/v1/schema Additional information -To use replication factors greater than one, use a [multi-node deployment](../installation/docker-compose.md#multi-node-configuration). +To use replication factors greater than one, use a [multi-node deployment](docs/deploy/installation-guides/docker-installation.md#multi-node-configuration). For details on the configuration parameters, see the following: diff --git a/docs/weaviate/manage-collections/vector-config.mdx b/docs/weaviate/manage-collections/vector-config.mdx index 620fb0e0..874aa350 100644 --- a/docs/weaviate/manage-collections/vector-config.mdx +++ b/docs/weaviate/manage-collections/vector-config.mdx @@ -23,7 +23,7 @@ Specify a `vectorizer` for a collection.
    Additional information -Collection level settings override default values and general configuration parameters such as [environment variables](../config-refs/env-vars/index.md). +Collection level settings override default values and general configuration parameters such as [environment variables](docs/deploy/config-guides/env-vars/index.md). - [Available model integrations](../model-providers/index.md) - [Vectorizer configuration references](/docs/weaviate/config-refs/schema#vectorizer) diff --git a/docs/weaviate/manage-objects/delete.mdx b/docs/weaviate/manage-objects/delete.mdx index f70591aa..8d83680e 100644 --- a/docs/weaviate/manage-objects/delete.mdx +++ b/docs/weaviate/manage-objects/delete.mdx @@ -187,7 +187,7 @@ To delete objects that match a set of criteria, specify the collection and a [`w Additional information -- There is a configurable [maximum limit (QUERY_MAXIMUM_RESULTS)](../config-refs/env-vars/index.md#general) on the number of objects that can be deleted in a single query (default 10,000). To delete more objects than the limit, re-run the query. +- There is a configurable [maximum limit (QUERY_MAXIMUM_RESULTS)](docs/deploy/config-guides/env-vars/index.md#general) on the number of objects that can be deleted in a single query (default 10,000). To delete more objects than the limit, re-run the query.
    @@ -296,7 +296,7 @@ There is an upper limit (`QUERY_MAXIMUM_RESULTS`) to how many objects can be del Objects are deleted in the same order that they would be fetched, by order of UUID. To delete more objects than the limit, run the same query multiple times until no objects are matched anymore. -The default `QUERY_MAXIMUM_RESULTS` value is 10,000. This may be configurable, e.g. in [the environment variables](../config-refs/env-vars/index.md). +The default `QUERY_MAXIMUM_RESULTS` value is 10,000. This may be configurable, e.g. in [the environment variables](docs/deploy/config-guides/env-vars/index.md).
    diff --git a/docs/weaviate/model-providers/anthropic/generative.md b/docs/weaviate/model-providers/anthropic/generative.md index 32a63464..889a407f 100644 --- a/docs/weaviate/model-providers/anthropic/generative.md +++ b/docs/weaviate/model-providers/anthropic/generative.md @@ -40,7 +40,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/anyscale/generative.md b/docs/weaviate/model-providers/anyscale/generative.md index 709932e9..09cc2579 100644 --- a/docs/weaviate/model-providers/anyscale/generative.md +++ b/docs/weaviate/model-providers/anyscale/generative.md @@ -40,7 +40,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/aws/embeddings.md b/docs/weaviate/model-providers/aws/embeddings.md index 47bf89b0..d075021e 100644 --- a/docs/weaviate/model-providers/aws/embeddings.md +++ b/docs/weaviate/model-providers/aws/embeddings.md @@ -43,7 +43,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/aws/generative.md b/docs/weaviate/model-providers/aws/generative.md index 621cad52..fb889450 100644 --- a/docs/weaviate/model-providers/aws/generative.md +++ b/docs/weaviate/model-providers/aws/generative.md @@ -41,7 +41,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/cohere/embeddings-multimodal.md b/docs/weaviate/model-providers/cohere/embeddings-multimodal.md index 16037165..b2119dce 100644 --- a/docs/weaviate/model-providers/cohere/embeddings-multimodal.md +++ b/docs/weaviate/model-providers/cohere/embeddings-multimodal.md @@ -43,7 +43,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/cohere/embeddings.md b/docs/weaviate/model-providers/cohere/embeddings.md index 230c250d..0b9f6f0e 100644 --- a/docs/weaviate/model-providers/cohere/embeddings.md +++ b/docs/weaviate/model-providers/cohere/embeddings.md @@ -46,7 +46,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/cohere/generative.md b/docs/weaviate/model-providers/cohere/generative.md index c69b6f06..d0b182ab 100644 --- a/docs/weaviate/model-providers/cohere/generative.md +++ b/docs/weaviate/model-providers/cohere/generative.md @@ -41,7 +41,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/cohere/reranker.md b/docs/weaviate/model-providers/cohere/reranker.md index f214b14a..732825cd 100644 --- a/docs/weaviate/model-providers/cohere/reranker.md +++ b/docs/weaviate/model-providers/cohere/reranker.md @@ -41,7 +41,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/databricks/embeddings.md b/docs/weaviate/model-providers/databricks/embeddings.md index e03f6213..cfcd5f3a 100644 --- a/docs/weaviate/model-providers/databricks/embeddings.md +++ b/docs/weaviate/model-providers/databricks/embeddings.md @@ -44,7 +44,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/databricks/generative.md b/docs/weaviate/model-providers/databricks/generative.md index a6d17923..4988b639 100644 --- a/docs/weaviate/model-providers/databricks/generative.md +++ b/docs/weaviate/model-providers/databricks/generative.md @@ -42,7 +42,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/friendliai/generative.md b/docs/weaviate/model-providers/friendliai/generative.md index 70e14460..87590981 100644 --- a/docs/weaviate/model-providers/friendliai/generative.md +++ b/docs/weaviate/model-providers/friendliai/generative.md @@ -43,7 +43,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/google/_includes/use_google_auth_instructions.mdx b/docs/weaviate/model-providers/google/_includes/use_google_auth_instructions.mdx index 3826f225..cdca2caf 100644 --- a/docs/weaviate/model-providers/google/_includes/use_google_auth_instructions.mdx +++ b/docs/weaviate/model-providers/google/_includes/use_google_auth_instructions.mdx @@ -10,7 +10,7 @@ You can save your Google Vertex AI credentials and have Weaviate generate the ne To do so: -- Set `USE_GOOGLE_AUTH` [environment variable](/docs/weaviate/config-refs/env-vars#module-specific) to `true`. +- Set `USE_GOOGLE_AUTH` [environment variable](docs/deploy/config-guides/env-vars/index.md#module-specific) to `true`. - Have the credentials available in one of the following locations. Once appropriate credentials are found, Weaviate uses them to generate an access token and authenticates itself against Vertex AI. Upon token expiry, Weaviate generates a replacement access token. diff --git a/docs/weaviate/model-providers/google/embeddings-multimodal.md b/docs/weaviate/model-providers/google/embeddings-multimodal.md index 9760e39b..6a635d55 100644 --- a/docs/weaviate/model-providers/google/embeddings-multimodal.md +++ b/docs/weaviate/model-providers/google/embeddings-multimodal.md @@ -48,7 +48,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/google/embeddings.md b/docs/weaviate/model-providers/google/embeddings.md index a93c1607..9fa2e66a 100644 --- a/docs/weaviate/model-providers/google/embeddings.md +++ b/docs/weaviate/model-providers/google/embeddings.md @@ -51,7 +51,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/google/generative.md b/docs/weaviate/model-providers/google/generative.md index f06a4c37..befb2f94 100644 --- a/docs/weaviate/model-providers/google/generative.md +++ b/docs/weaviate/model-providers/google/generative.md @@ -49,7 +49,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/gpt4all/embeddings.md b/docs/weaviate/model-providers/gpt4all/embeddings.md index 03708419..bc2e5591 100644 --- a/docs/weaviate/model-providers/gpt4all/embeddings.md +++ b/docs/weaviate/model-providers/gpt4all/embeddings.md @@ -44,7 +44,7 @@ This integration is not available for Weaviate Cloud (WCD) serverless instances,
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    @@ -60,7 +60,7 @@ The following example shows how to configure the GPT4All integration in Weaviate #### Docker Option 1: Use a pre-configured `docker-compose.yml` file -Follow the instructions on the [Weaviate Docker installation configurator](../../installation/docker-compose.md#configurator) to download a pre-configured `docker-compose.yml` file with a selected model +Follow the instructions on the [Weaviate Docker installation configurator](docs/deploy/installation-guides/docker-installation.md#configurator) to download a pre-configured `docker-compose.yml` file with a selected model
    #### Docker Option 2: Add the configuration manually diff --git a/docs/weaviate/model-providers/huggingface/embeddings.md b/docs/weaviate/model-providers/huggingface/embeddings.md index 827fd696..ccc2bff6 100644 --- a/docs/weaviate/model-providers/huggingface/embeddings.md +++ b/docs/weaviate/model-providers/huggingface/embeddings.md @@ -42,7 +42,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/imagebind/embeddings-multimodal.md b/docs/weaviate/model-providers/imagebind/embeddings-multimodal.md index de4c655e..0e47763b 100644 --- a/docs/weaviate/model-providers/imagebind/embeddings-multimodal.md +++ b/docs/weaviate/model-providers/imagebind/embeddings-multimodal.md @@ -39,7 +39,7 @@ This integration is not available for Weaviate Cloud (WCD) serverless instances, #### Enable the integration module -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate. #### Configure the integration @@ -53,7 +53,7 @@ The following example shows how to configure the ImageBind integration in Weavia #### Docker Option 1: Use a pre-configured `docker-compose.yml` file -Follow the instructions on the [Weaviate Docker installation configurator](../../installation/docker-compose.md#configurator) to download a pre-configured `docker-compose.yml` file with a selected model +Follow the instructions on the [Weaviate Docker installation configurator](docs/deploy/installation-guides/docker-installation.md#configurator) to download a pre-configured `docker-compose.yml` file with a selected model
    #### Docker Option 2: Add the configuration manually diff --git a/docs/weaviate/model-providers/jinaai/embeddings-colbert.md b/docs/weaviate/model-providers/jinaai/embeddings-colbert.md index 0e66d70a..697707b8 100644 --- a/docs/weaviate/model-providers/jinaai/embeddings-colbert.md +++ b/docs/weaviate/model-providers/jinaai/embeddings-colbert.md @@ -44,7 +44,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/jinaai/embeddings-multimodal.md b/docs/weaviate/model-providers/jinaai/embeddings-multimodal.md index c1810955..6743fdd9 100644 --- a/docs/weaviate/model-providers/jinaai/embeddings-multimodal.md +++ b/docs/weaviate/model-providers/jinaai/embeddings-multimodal.md @@ -42,7 +42,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/jinaai/embeddings.md b/docs/weaviate/model-providers/jinaai/embeddings.md index db50c10f..2afa6228 100644 --- a/docs/weaviate/model-providers/jinaai/embeddings.md +++ b/docs/weaviate/model-providers/jinaai/embeddings.md @@ -42,7 +42,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/jinaai/reranker.md b/docs/weaviate/model-providers/jinaai/reranker.md index 9f358bda..8ee9b451 100644 --- a/docs/weaviate/model-providers/jinaai/reranker.md +++ b/docs/weaviate/model-providers/jinaai/reranker.md @@ -42,7 +42,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/kubeai/embeddings.md b/docs/weaviate/model-providers/kubeai/embeddings.md index f37f5711..64ecfc42 100644 --- a/docs/weaviate/model-providers/kubeai/embeddings.md +++ b/docs/weaviate/model-providers/kubeai/embeddings.md @@ -44,7 +44,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/kubeai/generative.md b/docs/weaviate/model-providers/kubeai/generative.md index 4c6352f1..985f0730 100644 --- a/docs/weaviate/model-providers/kubeai/generative.md +++ b/docs/weaviate/model-providers/kubeai/generative.md @@ -44,7 +44,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/mistral/embeddings.md b/docs/weaviate/model-providers/mistral/embeddings.md index 258642ef..7108af85 100644 --- a/docs/weaviate/model-providers/mistral/embeddings.md +++ b/docs/weaviate/model-providers/mistral/embeddings.md @@ -42,7 +42,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/mistral/generative.md b/docs/weaviate/model-providers/mistral/generative.md index f2a1111c..3c51c42b 100644 --- a/docs/weaviate/model-providers/mistral/generative.md +++ b/docs/weaviate/model-providers/mistral/generative.md @@ -41,7 +41,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/nvidia/embeddings-multimodal.md b/docs/weaviate/model-providers/nvidia/embeddings-multimodal.md index 4d546a05..053261d1 100644 --- a/docs/weaviate/model-providers/nvidia/embeddings-multimodal.md +++ b/docs/weaviate/model-providers/nvidia/embeddings-multimodal.md @@ -43,7 +43,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/nvidia/embeddings.md b/docs/weaviate/model-providers/nvidia/embeddings.md index 48d40715..6764ade6 100644 --- a/docs/weaviate/model-providers/nvidia/embeddings.md +++ b/docs/weaviate/model-providers/nvidia/embeddings.md @@ -45,7 +45,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/nvidia/generative.md b/docs/weaviate/model-providers/nvidia/generative.md index bede2466..00bd6ff4 100644 --- a/docs/weaviate/model-providers/nvidia/generative.md +++ b/docs/weaviate/model-providers/nvidia/generative.md @@ -42,7 +42,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/nvidia/reranker.md b/docs/weaviate/model-providers/nvidia/reranker.md index b7c2723b..5fcd2b79 100644 --- a/docs/weaviate/model-providers/nvidia/reranker.md +++ b/docs/weaviate/model-providers/nvidia/reranker.md @@ -43,7 +43,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/octoai/embeddings.md b/docs/weaviate/model-providers/octoai/embeddings.md index 3616c919..c4def7a2 100644 --- a/docs/weaviate/model-providers/octoai/embeddings.md +++ b/docs/weaviate/model-providers/octoai/embeddings.md @@ -46,7 +46,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/octoai/generative.md b/docs/weaviate/model-providers/octoai/generative.md index 6cfb99b1..7558faa5 100644 --- a/docs/weaviate/model-providers/octoai/generative.md +++ b/docs/weaviate/model-providers/octoai/generative.md @@ -47,7 +47,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/ollama/embeddings.md b/docs/weaviate/model-providers/ollama/embeddings.md index 1a2d6daf..4d9a1c57 100644 --- a/docs/weaviate/model-providers/ollama/embeddings.md +++ b/docs/weaviate/model-providers/ollama/embeddings.md @@ -51,7 +51,7 @@ For use cases such as this, consider using a self-hosted Weaviate instance, or a
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/ollama/generative.md b/docs/weaviate/model-providers/ollama/generative.md index 49883819..da705574 100644 --- a/docs/weaviate/model-providers/ollama/generative.md +++ b/docs/weaviate/model-providers/ollama/generative.md @@ -51,7 +51,7 @@ For use cases such as this, consider using a self-hosted Weaviate instance, or a
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/openai-azure/embeddings.md b/docs/weaviate/model-providers/openai-azure/embeddings.md index 779e633f..e4873883 100644 --- a/docs/weaviate/model-providers/openai-azure/embeddings.md +++ b/docs/weaviate/model-providers/openai-azure/embeddings.md @@ -43,7 +43,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/openai-azure/generative.md b/docs/weaviate/model-providers/openai-azure/generative.md index d5a25d79..1e75f187 100644 --- a/docs/weaviate/model-providers/openai-azure/generative.md +++ b/docs/weaviate/model-providers/openai-azure/generative.md @@ -41,7 +41,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/openai/embeddings.md b/docs/weaviate/model-providers/openai/embeddings.md index f5a325dc..041fb42a 100644 --- a/docs/weaviate/model-providers/openai/embeddings.md +++ b/docs/weaviate/model-providers/openai/embeddings.md @@ -46,7 +46,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/openai/generative.md b/docs/weaviate/model-providers/openai/generative.md index 96a232ef..6f63fce8 100644 --- a/docs/weaviate/model-providers/openai/generative.md +++ b/docs/weaviate/model-providers/openai/generative.md @@ -44,7 +44,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/transformers/embeddings-multimodal.md b/docs/weaviate/model-providers/transformers/embeddings-multimodal.md index e4bae13b..fd033c03 100644 --- a/docs/weaviate/model-providers/transformers/embeddings-multimodal.md +++ b/docs/weaviate/model-providers/transformers/embeddings-multimodal.md @@ -40,7 +40,7 @@ This integration is not available for Weaviate Cloud (WCD) serverless instances, #### Enable the integration module -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate. #### Configure the integration @@ -54,7 +54,7 @@ The following example shows how to configure the CLIP integration in Weaviate: #### Docker Option 1: Use a pre-configured `docker-compose.yml` file -Follow the instructions on the [Weaviate Docker installation configurator](../../installation/docker-compose.md#configurator) to download a pre-configured `docker-compose.yml` file with a selected model +Follow the instructions on the [Weaviate Docker installation configurator](docs/deploy/installation-guides/docker-installation.md#configurator) to download a pre-configured `docker-compose.yml` file with a selected model
    #### Docker Option 2: Add the configuration manually diff --git a/docs/weaviate/model-providers/transformers/embeddings.md b/docs/weaviate/model-providers/transformers/embeddings.md index bb34ee1f..ccfa40fb 100644 --- a/docs/weaviate/model-providers/transformers/embeddings.md +++ b/docs/weaviate/model-providers/transformers/embeddings.md @@ -40,7 +40,7 @@ This integration is not available for Weaviate Cloud (WCD) serverless instances, #### Enable the integration module -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate. #### Configure the integration @@ -54,7 +54,7 @@ The following example shows how to configure the Hugging Face Transformers integ #### Docker Option 1: Use a pre-configured `docker-compose.yml` file -Follow the instructions on the [Weaviate Docker installation configurator](../../installation/docker-compose.md#configurator) to download a pre-configured `docker-compose.yml` file with a selected model +Follow the instructions on the [Weaviate Docker installation configurator](docs/deploy/installation-guides/docker-installation.md#configurator) to download a pre-configured `docker-compose.yml` file with a selected model
    #### Docker Option 2: Add the configuration manually diff --git a/docs/weaviate/model-providers/transformers/reranker.md b/docs/weaviate/model-providers/transformers/reranker.md index 22730da3..9f43d111 100644 --- a/docs/weaviate/model-providers/transformers/reranker.md +++ b/docs/weaviate/model-providers/transformers/reranker.md @@ -40,7 +40,7 @@ This integration is not available for Weaviate Cloud (WCD) serverless instances, #### Enable the integration module -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate. #### Configure the integration @@ -54,7 +54,7 @@ The following example shows how to configure the Hugging Face Transformers integ #### Docker Option 1: Use a pre-configured `docker-compose.yml` file -Follow the instructions on the [Weaviate Docker installation configurator](../../installation/docker-compose.md#configurator) to download a pre-configured `docker-compose.yml` file with a selected model +Follow the instructions on the [Weaviate Docker installation configurator](docs/deploy/installation-guides/docker-installation.md#configurator) to download a pre-configured `docker-compose.yml` file with a selected model
    #### Docker Option 2: Add the configuration manually diff --git a/docs/weaviate/model-providers/voyageai/embeddings-multimodal.md b/docs/weaviate/model-providers/voyageai/embeddings-multimodal.md index a6b21f61..dd994ee4 100644 --- a/docs/weaviate/model-providers/voyageai/embeddings-multimodal.md +++ b/docs/weaviate/model-providers/voyageai/embeddings-multimodal.md @@ -45,7 +45,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/voyageai/embeddings.md b/docs/weaviate/model-providers/voyageai/embeddings.md index ec178092..08ceb57f 100644 --- a/docs/weaviate/model-providers/voyageai/embeddings.md +++ b/docs/weaviate/model-providers/voyageai/embeddings.md @@ -43,7 +43,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/voyageai/reranker.md b/docs/weaviate/model-providers/voyageai/reranker.md index 12db608f..8bdedc5a 100644 --- a/docs/weaviate/model-providers/voyageai/reranker.md +++ b/docs/weaviate/model-providers/voyageai/reranker.md @@ -41,7 +41,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/model-providers/xai/generative.md b/docs/weaviate/model-providers/xai/generative.md index 114607a8..db46a676 100644 --- a/docs/weaviate/model-providers/xai/generative.md +++ b/docs/weaviate/model-providers/xai/generative.md @@ -42,7 +42,7 @@ This integration is enabled by default on Weaviate Cloud (WCD) serverless instan
    For self-hosted users -- Check the [cluster metadata](../../config-refs/meta.md) to verify if the module is enabled. +- Check the [cluster metadata](docs/deploy/config-guides/meta.md) to verify if the module is enabled. - Follow the [how-to configure modules](../../configuration/modules.md) guide to enable the module in Weaviate.
    diff --git a/docs/weaviate/modules/custom-modules.md b/docs/weaviate/modules/custom-modules.md index dbfcb6e3..50d5d49a 100644 --- a/docs/weaviate/modules/custom-modules.md +++ b/docs/weaviate/modules/custom-modules.md @@ -95,7 +95,7 @@ Because you are not touching the Go Weaviate interface code, you don't have the _Note that Weaviate APIs are not guaranteed to be stable. Even on a non-breaking Weaviate release, 'internal' APIS could always change._ To use a new inference model (part 2) with an existing Weaviate interface (part 1), you could reuse all the Go-code from the existing module and simply point it to a different inference container. As an example, here's how to use a custom inference module using the `text2vec-transformers` Go-code: -1. In a valid `docker-compose.yml` that's configured to use transformers (e.g. for example configure one via the [configuration configurator](/docs/weaviate/installation/docker-compose.md#configurator)), you will find an env var like this: `TRANSFORMERS_INFERENCE_API: 'http://t2v-transformers:8080'`, you can point that to any app you like. You should keep the variable name `TRANSFORMERS_INFERENCE_API`. +1. In a valid `docker-compose.yml` that's configured to use transformers (e.g. for example configure one via the [configuration configurator](docs/deploy/installation-guides/docker-installation.md#configurator)), you will find an env var like this: `TRANSFORMERS_INFERENCE_API: 'http://t2v-transformers:8080'`, you can point that to any app you like. You should keep the variable name `TRANSFORMERS_INFERENCE_API`. 2. Build a small HTTP API wrapper around your model, it should at the minimum have the endpoints listed below (which is in this example entirely specific to the `text2vec-transformers` module and fully in its control): 1. `GET /.well-known/live` -> respond `204` when the app is alive 2. `GET /.well-known/ready` -> respond `204` when the app is ready to serve traffic diff --git a/docs/weaviate/modules/img2vec-neural.md b/docs/weaviate/modules/img2vec-neural.md index edb0e8d5..dbdb881f 100644 --- a/docs/weaviate/modules/img2vec-neural.md +++ b/docs/weaviate/modules/img2vec-neural.md @@ -32,7 +32,7 @@ This module is not available on Weaviate Cloud. To use `img2vec-neural`, you must enable it in your Docker Compose file (e.g. `docker-compose.yml`). :::tip Use the configuration tool -While you can do so manually, we recommend using the [Weaviate configuration tool](/docs/weaviate/installation/docker-compose.md#configurator) to generate the `Docker Compose` file. +While you can do so manually, we recommend using the [Weaviate configuration tool](docs/deploy/installation-guides/docker-installation.md#configurator) to generate the `Docker Compose` file. ::: #### Parameters diff --git a/docs/weaviate/modules/index.md b/docs/weaviate/modules/index.md index 61fbf8c0..946166e8 100644 --- a/docs/weaviate/modules/index.md +++ b/docs/weaviate/modules/index.md @@ -68,7 +68,7 @@ For example: - A module name must be url-safe, meaning it must not contain any characters which would require url-encoding. - A module name is not case-sensitive. `text2vec-bert` would be the same module as `text2vec-BERT`. - Module information is accessible through the `v1/modules//` RESTful endpoint. -- General module information (which modules are attached, version, etc.) is accessible through Weaviate's [`v1/meta` endpoint](../config-refs/meta.md). +- General module information (which modules are attached, version, etc.) is accessible through Weaviate's [`v1/meta` endpoint](docs/deploy/config-guides/meta.md). - Modules can add `additional` properties in the RESTful API and [`_additional` properties in the GraphQL API](../api/graphql/additional-properties.md). - A module can add [filters](../api/graphql/filters.md) in GraphQL queries. - Which vectorizer and other modules are applied to which data collection is configured in the [schema](../manage-collections/vector-config.mdx#specify-a-vectorizer). @@ -88,9 +88,9 @@ This type of provider is ideal for production environments. This is because stor Additionally, multi-node Weaviate clusters _require_ the use of an external provider. Storing a multi-node backup on internally on a single node presents several issues, like significantly reducing the durability and availability of the backup, and is not supported. The supported external backup providers are: -- [S3](/docs/weaviate/configuration/backups.md#s3-aws-or-s3-compatible) -- [GCS](/docs/weaviate/configuration/backups.md#gcs-google-cloud-storage) -- [Azure](/docs/weaviate/configuration/backups.md#azure-storage) +- [S3](docs/deploy/config-guides/backups.md#s3-aws-or-s3-compatible) +- [GCS](docs/deploy/config-guides/backups.md#gcs-google-cloud-storage) +- [Azure](docs/deploy/config-guides/backups.md#azure-storage) Thanks to the extensibility of the module system, new providers can be readily added. If you are interested in an external provider other than the ones listed above, feel free to reach out via our [forum](https://forum.weaviate.io/), or open an issue on [GitHub](https://github.com/weaviate/weaviate). @@ -98,7 +98,7 @@ Thanks to the extensibility of the module system, new providers can be readily a Internal providers coordinate the storage and retrieval of backed-up Weaviate data within a Weaviate instance. This type of provider is intended for developmental or experimental use, and is not recommended for production. Internal Providers are not compatible for multi-node backups, which require the use of an external provider. -As of Weaviate `v1.16`, the only supported internal backup provider is the [filesystem](/docs/weaviate/configuration/backups.md#filesystem) provider. +As of Weaviate `v1.16`, the only supported internal backup provider is the [filesystem](docs/deploy/config-guides/backups.md#filesystem) provider. ## Offloading Modules diff --git a/docs/weaviate/modules/text2vec-contextionary.md b/docs/weaviate/modules/text2vec-contextionary.md index 4aa28d06..0f83827e 100644 --- a/docs/weaviate/modules/text2vec-contextionary.md +++ b/docs/weaviate/modules/text2vec-contextionary.md @@ -32,7 +32,7 @@ This module is not available on Weaviate Cloud. To use `text2vec-contextionary`, you must enable it in your Docker Compose file (e.g. `docker-compose.yml`). :::tip Use the configuration tool -While you can do so manually, we recommend using the [Weaviate configuration tool](/docs/weaviate/installation/docker-compose.md#configurator) to generate the `Docker Compose` file. +While you can do so manually, we recommend using the [Weaviate configuration tool](docs/deploy/installation-guides/docker-installation.md#configurator) to generate the `Docker Compose` file. ::: #### Parameters diff --git a/docs/weaviate/more-resources/faq.md b/docs/weaviate/more-resources/faq.md index 1009c998..30fdb4a0 100644 --- a/docs/weaviate/more-resources/faq.md +++ b/docs/weaviate/more-resources/faq.md @@ -73,7 +73,7 @@ image: og/docs/more-resources.jpg Answer > Role-based access control (RBAC) can be enabled when configuring Weaviate via the `AUTHORIZATION_RBAC_ENABLED` environment variable. -> For more info visit the [RBAC: Configuration](/docs/weaviate/configuration/rbac/configuration) guide. +> For more info visit the [RBAC: Configuration](docs/deploy/config-guides/configuring-rbac.md) guide.
    @@ -473,7 +473,7 @@ More concretely: If you had to pick between a machine that has 16 GB of RAM and
    Answer -You can do this by sending a `SIGQUIT` signal to the process. This will print a stack trace to the console. The logging level and debugging variables can be set with `LOG_LEVEL` and `DEBUG` [environment variables](../config-refs/env-vars/index.md). +You can do this by sending a `SIGQUIT` signal to the process. This will print a stack trace to the console. The logging level and debugging variables can be set with `LOG_LEVEL` and `DEBUG` [environment variables](docs/deploy/config-guides/env-vars/index.md). Read more on SIGQUIT [here](https://en.wikipedia.org/wiki/Signal_(IPC)#SIGQUIT) and this [StackOverflow answer](https://stackoverflow.com/questions/19094099/how-to-dump-goroutine-stacktraces/35290196#35290196). diff --git a/docs/weaviate/more-resources/index.md b/docs/weaviate/more-resources/index.md index fdd50eb6..71905fd1 100644 --- a/docs/weaviate/more-resources/index.md +++ b/docs/weaviate/more-resources/index.md @@ -17,7 +17,7 @@ These pages can help with common questions: - [Example datasets](./example-datasets.md) - [Example use cases](./example-use-cases.md) - [Index types and performance](./performance.md) -- [Migration Guide](./migration/index.md) +- [Migration Guide](docs/deploy/migration/index.md) ## (Even) more resources diff --git a/docs/weaviate/more-resources/migration/_category_.json b/docs/weaviate/more-resources/migration/_category_.json deleted file mode 100644 index d2921f4f..00000000 --- a/docs/weaviate/more-resources/migration/_category_.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "label": "Migration guide", - "position": 8 -} diff --git a/docs/weaviate/recipes/generative_search_nvidia.md b/docs/weaviate/recipes/generative_search_nvidia.md index 0c530059..522bbe84 100644 --- a/docs/weaviate/recipes/generative_search_nvidia.md +++ b/docs/weaviate/recipes/generative_search_nvidia.md @@ -19,7 +19,7 @@ In this demo, we will use an embedding and generative model on NVIDIA to generat 1. Weaviate cluster 1. You can create a 14-day free sandbox on [WCD](https://console.weaviate.cloud/) 2. [Embedded Weaviate](https://weaviate.io/developers/weaviate/installation/embedded) - 3. [Local deployment](https://weaviate.io/developers/weaviate/installation/docker-compose#starter-docker-compose-file) + 3. [Local deployment](docs/deploy/installation-guides/docker-installation.md#starter-docker-compose-file) 4. [Other options](https://weaviate.io/developers/weaviate/installation) 2. NVIDIA NIM API key. Grab one [here](https://build.nvidia.com/models). diff --git a/docs/weaviate/recipes/hybrid_search_mistral.md b/docs/weaviate/recipes/hybrid_search_mistral.md index 7b6b92da..d4eb1f91 100644 --- a/docs/weaviate/recipes/hybrid_search_mistral.md +++ b/docs/weaviate/recipes/hybrid_search_mistral.md @@ -19,7 +19,7 @@ This recipe will show you how to run hybrid search with embeddings from Mistral. 1. Weaviate cluster 1. You can create a 14-day free sandbox on [WCD](https://console.weaviate.cloud/) 2. [Embedded Weaviate](https://weaviate.io/developers/weaviate/installation/embedded) - 3. [Local deployment](https://weaviate.io/developers/weaviate/installation/docker-compose#starter-docker-compose-file) + 3. [Local deployment](docs/deploy/installation-guides/docker-installation.md#starter-docker-compose-file) 4. [Other options](https://weaviate.io/developers/weaviate/installation) 2. Mistral API key. Grab one [here](https://docs.mistral.ai/api/). diff --git a/docs/weaviate/release-notes/index.md b/docs/weaviate/release-notes/index.md index daf948ea..93a2fdd2 100644 --- a/docs/weaviate/release-notes/index.md +++ b/docs/weaviate/release-notes/index.md @@ -42,7 +42,7 @@ export const pythonCardsData = [ This page lists [developer release notes](https://github.com/weaviate/weaviate/releases) for Weaviate Database. - To see the GitHub release notes for a particular version, click on the version number. -- For more information on upgrading Weaviate, see the [Migration Guide](../more-resources/migration/index.md). +- For more information on upgrading Weaviate, see the [Migration Guide](docs/deploy/migration/index.md). ### Weaviate Database and client releases diff --git a/docs/weaviate/starter-guides/generative.md b/docs/weaviate/starter-guides/generative.md index c4db0ce4..e8876d28 100644 --- a/docs/weaviate/starter-guides/generative.md +++ b/docs/weaviate/starter-guides/generative.md @@ -388,7 +388,7 @@ services: ENABLE_MODULES: 'text2vec-cohere,text2vec-huggingface,text2vec-openai,text2vec-google,generative-cohere,generative-openai,generative-googles' ``` -Check the specific documentation for your deployment method ([Docker](../installation/docker-compose.md), [Kubernetes](../installation/kubernetes.md), [Embedded Weaviate](../installation/embedded.md)) for more information on how to configure it. +Check the specific documentation for your deployment method ([Docker](docs/deploy/installation-guides/docker-installation.md), [Kubernetes](docs/deploy/installation-guides/k8s-installation.md), [Embedded Weaviate](../installation/embedded.md)) for more information on how to configure it.
    diff --git a/docs/weaviate/starter-guides/managing-collections/collections-scaling-limits.mdx b/docs/weaviate/starter-guides/managing-collections/collections-scaling-limits.mdx index 1fbd0a87..2f48843c 100644 --- a/docs/weaviate/starter-guides/managing-collections/collections-scaling-limits.mdx +++ b/docs/weaviate/starter-guides/managing-collections/collections-scaling-limits.mdx @@ -11,7 +11,7 @@ image: og/docs/more-resources.jpg :::caution Have you hit the collections count limit? If you see an error that the collection count limit has been reached, it means you **can't create any more collections**. This limit aims to ensure Weaviate remains performant. If your instance already exceeds the limit, Weaviate will not allow the creation of any new collections. Existing collections will not be deleted.

    -**Instead of simply raising the limit, consider rethinking your architecture**. If you really need to change the limit, use the [`MAXIMUM_ALLOWED_COLLECTIONS_COUNT`](../../config-refs/env-vars/index.md) environment variable. +**Instead of simply raising the limit, consider rethinking your architecture**. If you really need to change the limit, use the [`MAXIMUM_ALLOWED_COLLECTIONS_COUNT`](docs/deploy/config-guides/env-vars/index.md) environment variable. ::: This guide offers an overview of the available architectural choices of using **multi-tenancy** or defining a **dedicated collection** for each subset of data. @@ -139,7 +139,7 @@ Use multi-tenancy when you need to support a large number of tenants and priorit #### Challenges -- **Access control complexity**: [Fine-grained access control](../../configuration/authorization.md) must be implemented to ensure data isolation between tenants. +- **Access control complexity**: [Fine-grained access control](docs/deploy/config-guides/authorization.md) must be implemented to ensure data isolation between tenants. - **Uniform collection definition**: All tenants must share the same collection schema and configuration. :::tip Reduce storage costs diff --git a/docs/weaviate/starter-guides/which-weaviate.md b/docs/weaviate/starter-guides/which-weaviate.md index f49cf780..1ebf33ac 100644 --- a/docs/weaviate/starter-guides/which-weaviate.md +++ b/docs/weaviate/starter-guides/which-weaviate.md @@ -16,9 +16,9 @@ This page helps you to find the right combination for your project. Weaviate can be deployed in the following ways: - [Embedded Weaviate](../installation/embedded.md) -- [Docker-Compose](../installation/docker-compose.md) +- [Docker-Compose](docs/deploy/installation-guides/docker-installation.md) - [Weaviate Cloud (WCD)](../installation/weaviate-cloud-services.md) -- [Self-managed Kubernetes](../installation/kubernetes.md) +- [Self-managed Kubernetes](docs/deploy/installation-guides/k8s-installation.md) - [Hybrid SaaS](https://weaviate.io/pricing) ## Vectorization options @@ -52,7 +52,7 @@ The [Quickstart guide](/docs/weaviate/quickstart) uses a WCD sandbox and an API For development, we recommend using -- [Weaviate Cloud (WCD)](https://console.weaviate.cloud/) or [Docker Compose](/docs/weaviate/installation/docker-compose). +- [Weaviate Cloud (WCD)](https://console.weaviate.cloud/) or [Docker Compose](docs/deploy/installation-guides/docker-installation.md). - A vectorization strategy that matches your production vectorization strategy. #### Docker-Compose vs. Weaviate Cloud (WCD) @@ -82,7 +82,7 @@ Keep in mind two other factors, which are cost, and their footprint. For production deployments, consider one of these hosting models: - [Weaviate Cloud (WCD)](/docs/cloud) -- [Self-managed Kubernetes](/docs/weaviate/installation/kubernetes) +- [Self-managed Kubernetes](/docs/deploy/installation-guides/k8s-installation.md) - [Hybrid SaaS](/docs/cloud) All of these options are scalable. Kubernetes and Hybrid SaaS offer the most configuration flexibility. diff --git a/docs/weaviate/tutorials/import.md b/docs/weaviate/tutorials/import.md index 172cd002..5af21041 100644 --- a/docs/weaviate/tutorials/import.md +++ b/docs/weaviate/tutorials/import.md @@ -132,7 +132,7 @@ When importing large datasets, it may be worth planning out an optimized import 1. To use multiple CPUs efficiently, enable sharding when you import data. For the fastest imports, enable sharding even on a single node. 1. Use [parallelization](https://www.computerhope.com/jargon/p/parallelization.htm); if the CPUs are not maxed out, just add another import process. 1. Use `htop` when importing to see if all CPUs are maxed out. -1. To avoid out-of-memory issues during imports, set `LIMIT_RESOURCES` to `True` or configure the `GOMEMLIMIT` environment variable. For details, see [Environment variables](../config-refs/env-vars/index.md). +1. To avoid out-of-memory issues during imports, set `LIMIT_RESOURCES` to `True` or configure the `GOMEMLIMIT` environment variable. For details, see [Environment variables](docs/deploy/config-guides/env-vars/index.md). 1. For Kubernetes, a few large machines are faster than many small machines (due to network latency). Our rules of thumb are: diff --git a/docs/weaviate/tutorials/index.mdx b/docs/weaviate/tutorials/index.mdx index 3d45ec6a..5a2f88a4 100644 --- a/docs/weaviate/tutorials/index.mdx +++ b/docs/weaviate/tutorials/index.mdx @@ -21,7 +21,7 @@ export const advancedFeaturesData = [ title: "Set up Role-Based Access Control (RBAC)", description: "Configure roles, permissions, and user assignments for secure access control in Weaviate.", - link: "/docs/weaviate/tutorials/rbac", + link: "/docs/deploy/tutorials/rbac", icon: "fas fa-user-shield", }, { diff --git a/secondaryNavbar.js b/secondaryNavbar.js index 61c0e53e..117c2844 100644 --- a/secondaryNavbar.js +++ b/secondaryNavbar.js @@ -56,7 +56,7 @@ const secondaryNavbarItems = { }, ], }, - /* + deploy: { title: "Deploy", icon: "fa fa-database", @@ -64,10 +64,18 @@ const secondaryNavbarItems = { link: "/docs/deploy", links: [ { label: "Get Started", link: "/docs/deploy", sidebar: "deploySidebar" }, - { label: "AWS", link: "/docs/deploy/aws", sidebar: "deployAwsSidebar" }, - ], + { label: "Configuration Guides", link: "/docs/deploy/config-guides", sidebar: "deployConfigSidebar"}, + /*{ label: "Kubernetes", link: "/docs/deploy/k8s", sidebar: "deployK8sSidebar" },*/ + { label: "Production Guides", link: "/docs/deploy/production", sidebar: "deployProductionSidebar"}, + { label: "Tutorials", link: "/docs/deploy/tutorials/rbac", sidebar: "deployTutorialSidebar"}, + /*{ label: "AWS", link: "/docs/deploy/aws", sidebar: "deployAwsSidebar" },*/ + /*{ label: "Scaling Strategies", link: "/docs/deploy/scaling-strategies", sidebar: "deployScalingSidebar"},*/ + /*{ label: "Monitoring and Observability", link: "/docs/deploy/monitoring-obs", sidebar: "deployObservabilitySidebar"},*/ + { label: "FAQs", link: "/docs/deploy/faqs", sidebar: "deployFaqsSidebar"}, + { label: "Migration", link: "/docs/deploy/migration", sidebar: "deployMigrationSidebar"}, + ] }, - */ + agents: { title: "Weaviate Agents", icon: "fa fa-robot", @@ -119,6 +127,7 @@ const secondaryNavbarItems = { integrations: { title: "Integrations", icon: "fa fa-puzzle-piece", + isSmall: true, description: "For hyperscalers, data platforms, LLM frameworks, etc.", link: "/docs/integrations", links: [ diff --git a/sidebars.js b/sidebars.js index fa4355ef..224fb64f 100644 --- a/sidebars.js +++ b/sidebars.js @@ -26,8 +26,6 @@ const sidebars = { }, items: [ "weaviate/installation/weaviate-cloud-services", - "weaviate/installation/docker-compose", - "weaviate/installation/kubernetes", "weaviate/installation/embedded", "weaviate/installation/spcs-integration", ], @@ -428,9 +426,7 @@ const sidebars = { }, "weaviate/configuration/tenant-offloading", "weaviate/configuration/modules", - "weaviate/configuration/backups", "weaviate/configuration/authentication", - "weaviate/configuration/authorization", { type: "category", label: "RBAC", @@ -439,14 +435,10 @@ const sidebars = { id: "weaviate/configuration/rbac/index", }, items: [ - "weaviate/configuration/rbac/configuration", "weaviate/configuration/rbac/manage-roles", "weaviate/configuration/rbac/manage-users", ], }, - "weaviate/configuration/persistence", - "weaviate/configuration/monitoring", - "weaviate/configuration/replication", ], }, { @@ -518,7 +510,6 @@ const sidebars = { id: "weaviate/tutorials/index", }, items: [ - "weaviate/tutorials/rbac", "weaviate/tutorials/multi-vector-embeddings", //"weaviate/tutorials/import", "weaviate/tutorials/cross-references", @@ -564,41 +555,6 @@ const sidebars = { id: "weaviate/config-refs/distances", className: "sidebar-item", }, - { - type: "category", - label: "Environment variables", - className: "sidebar-item", - link: { - type: "doc", - id: "weaviate/config-refs/env-vars/index", - }, - items: ["weaviate/config-refs/env-vars/runtime-config"], - }, - { - type: "doc", - id: "weaviate/config-refs/status", - className: "sidebar-item", - }, - { - type: "doc", - id: "weaviate/config-refs/telemetry", - className: "sidebar-item", - }, - { - type: "doc", - id: "weaviate/config-refs/meta", - className: "sidebar-item", - }, - { - type: "doc", - id: "weaviate/config-refs/nodes", - className: "sidebar-item", - }, - { - type: "doc", - id: "weaviate/config-refs/oidc", - className: "sidebar-item", - }, ], }, { @@ -731,7 +687,7 @@ const sidebars = { ], }, ], - othersSidebar: [ + othersSidebar: [ { type: "category", label: "Releases", @@ -740,19 +696,6 @@ const sidebars = { id: "weaviate/release-notes/index", }, items: [ - { - type: "category", - label: "Migration", - link: { - type: "doc", - id: "weaviate/more-resources/migration/index", - }, - items: [ - "weaviate/more-resources/migration/weaviate-1-30", - "weaviate/more-resources/migration/weaviate-1-25", - "weaviate/more-resources/migration/archive", - ], - }, ], }, { @@ -786,12 +729,174 @@ const sidebars = { "weaviate/more-resources/glossary", "weaviate/more-resources/example-datasets", ], - deploySidebar: ["deploy/index"], - deployAwsSidebar: [ + deploySidebar: [ + { + type: "doc", + id: "deploy/index", // document ID + label: "Overview", // sidebar label + }, + { + type: "category", + label: "Installation Guides", + collapsible: true, + collapsed: true, + items: [ + "deploy/installation-guides/k8s-installation", + "deploy/installation-guides/docker-installation", + "deploy/installation-guides/gcp-installation", + "deploy/installation-guides/aws-installation", + "deploy/installation-guides/aws-cli", + ], + }, + ], +deployConfigSidebar: [ + { + type: "category", + label: "Configuration Guides", + className: "sidebar-main-category", + collapsible: false, + link: { + type: "doc", + id: "deploy/config-guides/index", + }, + items: [ + { + type: "doc", + id: "deploy/config-guides/backups", + className: "sidebar-item", + }, + { + type: "doc", + id: "deploy/config-guides/horizontal-scaling", + className: "sidebar-item", + }, + { + type: "doc", + id: "deploy/config-guides/meta", + className: "sidebar-item", + }, + { + type: "doc", + id: "deploy/config-guides/monitoring", + className: "sidebar-item", + }, + { + type: "doc", + id: "deploy/config-guides/nodes", + className: "sidebar-item", + }, + { + type: "doc", + id: "deploy/config-guides/persistence", + className: "sidebar-item", + }, + { + type: "doc", + id: "deploy/config-guides/replication", + className: "sidebar-item", + }, + { + type: "doc", + id: "deploy/config-guides/status", + className: "sidebar-item", + }, + { + type: "doc", + id: "deploy/config-guides/telemetry", + className: "sidebar-item", + }, + { + type: "html", + value: "
    ", + }, + { + type: "category", + label: "Authorization and Authentication", + className: "sidebar-item", + link: { + type: "doc", + id: "deploy/config-guides/authentication", + }, + items: [ + "deploy/config-guides/authentication", + "deploy/config-guides/authorization", + "deploy/config-guides/oidc", + "deploy/config-guides/configuring-rbac", + ], + }, + { + type: "html", + value: "
    ", + }, + { + type: "category", + label: "Replication", + className: "sidebar-item", + items: [ + "deploy/config-guides/replication", + "deploy/config-guides/async-rep", + ], + }, + { + type: "html", + value: "
    ", + }, + { + type: "category", + label: "Environment Variables", + className: "sidebar-item", + items: [ + "deploy/config-guides/env-vars/index", + "deploy/config-guides/env-vars/runtime-config", + ], + }, + ], + }, +], + /*deployAwsSidebar: [ { type: "autogenerated", dirName: "deploy/aws", }, + ],*/ + /*deployK8sSidebar: [ + { + type: "autogenerated", + dirName: "deploy/k8s", + }, + ],*/ + deployProductionSidebar: [ + { + type: "autogenerated", + dirName: "deploy/production", + + }, + ], +deployTutorialSidebar: [ + { + type: "category", + label: "Tutorials", + className: "sidebar-main-category", + collapsible: false, + link: { + type: "doc", + id: "deploy/tutorials/rbac", + }, + items: [], + }, +], + deployFaqsSidebar: [ + { + type: "autogenerated", + dirName: "deploy/faqs", + + }, + ], + deployMigrationSidebar: [ + { + type: "autogenerated", + dirName: "deploy/migration" + }, ], agentsSidebar: [ "agents/index", diff --git a/yarn.lock b/yarn.lock index 6915f50c..5533dba0 100644 --- a/yarn.lock +++ b/yarn.lock @@ -1857,6 +1857,21 @@ react-helmet-async "npm:@slorber/react-helmet-async@*" react-loadable "npm:@docusaurus/react-loadable@6.0.0" +"@docusaurus/plugin-client-redirects@^3.7.0": + version "3.7.0" + resolved "https://registry.yarnpkg.com/@docusaurus/plugin-client-redirects/-/plugin-client-redirects-3.7.0.tgz#b5cf92529768c457c01ad350bfc50862c6149463" + integrity sha512-6B4XAtE5ZVKOyhPgpgMkb7LwCkN+Hgd4vOnlbwR8nCdTQhLjz8MHbGlwwvZ/cay2SPNRX5KssqKAlcHVZP2m8g== + dependencies: + "@docusaurus/core" "3.7.0" + "@docusaurus/logger" "3.7.0" + "@docusaurus/utils" "3.7.0" + "@docusaurus/utils-common" "3.7.0" + "@docusaurus/utils-validation" "3.7.0" + eta "^2.2.0" + fs-extra "^11.1.1" + lodash "^4.17.21" + tslib "^2.6.0" + "@docusaurus/plugin-content-blog@3.7.0": version "3.7.0" resolved "https://registry.yarnpkg.com/@docusaurus/plugin-content-blog/-/plugin-content-blog-3.7.0.tgz#7bd69de87a1f3adb652e1473ef5b7ccc9468f47e"