Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Opensearch vector store bugs #1589

Open
tstephen1 opened this issue Oct 23, 2024 · 4 comments · May be fixed by #1725
Open

Opensearch vector store bugs #1589

tstephen1 opened this issue Oct 23, 2024 · 4 comments · May be fixed by #1725
Assignees
@ilayaperumalg
Labels
opensearch vector store
Milestone
1.0.0-M4

Comments

@tstephen1
Copy link

Following the walkthrough in spring ai documenation for opensearch vector store on aws, the following properties are incorrectly located:

spring.opensearch.aws.host
spring.opensearch.aws.service-name
spring.opensearch.aws.access-key
spring.opensearch.aws.secret-key
spring.opensearch.aws.region

these only work if located on
spring.ai.vectorstore.opensearch.*

else causes error:
Caused by: java.lang.NullPointerException: Access key ID cannot be blank.

Also similarity search for index with the following json mapping gives:
Similarity search results with error:
org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [search_phase_execution_exception] all shards failed

for mapping json:
mappingJson: |
"properties": {
"embedding": {
"type": "knn_vector",
"dimension": 1536
}
}
@garc33
Copy link

garc33 commented Oct 23, 2024

I had exactly the same issue. I fixed it by disabling the json indentation in Jackson ObjectMapper.

@tstephen1
Copy link
Author

tstephen1 commented Oct 24, 2024 •
edited
Loading

I had exactly the same issue. I fixed it by disabling the json indentation in Jackson ObjectMapper.

I tried this but it didnt work -

spring:
jackson:
serialization:
indent_output: false

Could you give an example?

@apappascs apappascs mentioned this issue Oct 25, 2024
Merged
@tzolov
Copy link
Contributor

tzolov commented Oct 25, 2024

@garc33 @tstephen1 does the updated docs helps your resolve the issue?
https://docs.spring.io/spring-ai/reference/api/vectordbs/opensearch.html (refresh the page).

@tzolov tzolov added this to the 1.0.0-M4 milestone Oct 25, 2024
@tstephen1
Copy link
Author

@tzolov @garc33 - the updated docs change, adding documents to aws openseach works fine. But similarity search still causes error:

org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [search_phase_execution_exception] all shards failed

Updating the object mapper with indent output to false as suggested above :
objectMapper.configure(SerializationFeature.INDENT_OUTPUT, false);

generates this error when using ollama embedding model:

2024-10-25T14:27:04.111+01:00 DEBUG 11139 --- h.i.i.PoolingHttpClientConnectionManager : ep-0000000001 connection released [route: {}->http://localhost:11434][total available: 1; route allocated: 1 of 5; total allocated: 1 of 25]
2024-10-25T14:27:04.212+01:00 DEBUG 11139 --- o.s.w.s.m.m.a.HttpEntityMethodProcessor : Using 'text/plain', given [/] and supported [text/plain, /, application/json, application/*+json]
2024-10-25T14:27:04.214+01:00 DEBUG 11139 --- o.s.w.s.m.m.a.HttpEntityMethodProcessor : Writing ["Failed to save text: Error while extracting response for type [org.springframework.ai.ollama.api.Oll (truncated)..."]

@ilayaperumalg ilayaperumalg self-assigned this Nov 4, 2024
ilayaperumalg added a commit to ilayaperumalg/spring-ai that referenced this issue Nov 12, 2024
@ilayaperumalg
spring-projectsGH-1589 Fix OpenSearch client indices mapping
6ab9fad 
 - Fix the embedding dimension configuration for opensearch client indices mapping
    - The dimension config is obtained by the underlying embedding model's dimension
ilayaperumalg added a commit to ilayaperumalg/spring-ai that referenced this issue Nov 12, 2024
@ilayaperumalg
spring-projectsGH-1589 Fix OpenSearch client indices mapping
c66dfbf 
 - Fix the embedding dimension configuration for opensearch client indices mapping
    - The dimension config is obtained by the underlying embedding model's dimension

Resolves spring-projects#1589
@ilayaperumalg ilayaperumalg linked a pull request Nov 12, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ilayaperumalg ilayaperumalg

Labels
opensearch vector store
Projects
None yet
Milestone
1.0.0-M4
Development

Successfully merging a pull request may close this issue.

GH-1589 Fix OpenSearch client indices mapping ilayaperumalg/spring-ai
4 participants
@ilayaperumalg @tzolov @garc33 @tstephen1