Added OpenSearch2.19.1 as the vector_database support

[pyyuhao]

What problem does this PR solve?

This PR adds the support for latest OpenSearch2.19.1 as the store engine & search engine option for RAGFlow.

Main Benefit

1. OpenSearch2.19.1 is licensed under the [Apache v2.0 License] which is much better than Elasticsearch
2. For search, OpenSearch2.19.1 supports full-text search、vector_search、hybrid_search those are similar with Elasticsearch on schema
3. For store, OpenSearch2.19.1 stores text、vector those are quite simliar with Elasticsearch on schema

Changes

• Support opensearch_python_connetor. I make a lot of adaptions since the schema and api/method between ES and Opensearch differs in many ways(especially the knn_search has a significant gap) : rag/utils/opensearch_coon.py
• Support static config adaptions by changing: conf/service_conf.yaml、api/settings.py、rag/settings.py
• Supprt some store&search schema changes between OpenSearch and ES: conf/os_mapping.json
• Support OpenSearch python sdk : pyproject.toml
• Support docker config for OpenSearch2.19.1 : docker/.env、docker/docker-compose-base.yml、docker/service_conf.yaml.template

How to use

• I didn't change the priority that ES as the default doc/search engine. Only if in docker/.env , we set DOC_ENGINE=${DOC_ENGINE:-opensearch}, it will work.

Others

Our team tested a lot of docs in our environment by using OpenSearch as the vector database ,it works very well.
All the conifg for OpenSearch is necessary.

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

[yingfeng]

Thanks for the contribution. OpenSearch can be added as a doc engine alternative. However, maintaining OpenSearch is not an easy work, after this PR is merged, OpenSearch might not be able to work after several weeks.

[asiroliu]

the default DOC_ENGINE should be elasticsearch

[pyyuhao]

@asiroliu I'v changed it into elasticsearch by default in my commit. It's a config mistake I forgot to change back from my local environment

[pyyuhao]

Thanks for the contribution. OpenSearch can be added as a doc engine alternative. However, maintaining OpenSearch is not an easy work, after this PR is merged, OpenSearch might not be able to work after several weeks.

Thanks for you reply, and I'v changed some litte problems mentioned above. I am an search-engine engineer focusing on the stuff about OpenSearch/Elasticsearch for years, and also write some plugins for Opnsearch.I will still pay much attention on ES/OS continuously. During these two years, We gave more attention on RAG stuff

[pyyuhao]

@asiroliu @yingfeng @KevinHuSh ,hi: I've made some commits mainly about fomat and comment. Please review again, thanks a lot

[yingfeng]

It can not pass CI, the container of elasticsearch can not be started. See the CI logs https://github.com/infiniflow/ragflow/actions/runs/14533575135/job/40777952188

[pyyuhao]

It can not pass CI, the container of elasticsearch can not be started. See the CI logs https://github.com/infiniflow/ragflow/actions/runs/14533575135/job/40777952188

It worked well now at my local environment, I will check the code again and create a virtual machine to verify it.
Maybe because I use opensearch on 9200 port which is the same as elasticsearch(on most cases these are the same), is there a rule about this that will block the test job for es?
I will check these in a few days

Have a nice weekend