Open AI Agent with your own data

[eeadie]

Hi,

I was looking into using an OpenAIAgent and was following this example:

OpenAIAssistant_ChartMaker

Is it possible to initialise an OpenAIAssistantAgent with an AzureSearchChatDataSource? I have previously used this with the IChatCompletionService like below:

IChatCompletionService chatCompletionService = Kernel.GetRequiredService<IChatCompletionService>();

 AzureOpenAIPromptExecutionSettings openAIPromptExecutionSettings = new AzureOpenAIPromptExecutionSettings
{
	AzureChatDataSource = new AzureSearchChatDataSource
	{
		Endpoint = new Uri(settings.SearchEndpoint),
		Authentication = DataSourceAuthentication.FromApiKey(settings.SearchApiKey),
		IndexName = settings.IndexName
	}                    
};

IAsyncEnumerable<StreamingChatMessageContent> myResult = chatCompletionService.GetStreamingChatMessageContentsAsync(chatHistory,
                                                                                                                    executionSettings: openAIPromptExecutionSettings,
                                                                                                                    kernel: Kernel,
                                                                                                                    cancellationToken: cancellationToken);

Thanks,

Euan

[crickman]

Hi @eeadie - Your are correct that this would be problematic with OpenAIAssistantAgent as it is not based upon IChatCompletionService...rather is it targeting the Open AI Assistant API.

For the Assistant API, a file-search tools is available that works against its VectorStore targets.

I am aware that work is being done on the Microsoft/Azure side to develop the Assistant API so that it might be more integrated into Azure specific services. In the meantime, you might consider creating a function or plugin that exposes this behavior so that the assistant might call it as a function-tool; although, doing this in streaming-mode may present extra challenges.

Note: I have worked with an internal customer on streaming results from a function as IAsyncEnumerable so I know it is possible, but not trivial (care must be taken to properly manage the asyncronous result at every level of processing and automatic function-calling may interfere with this as the entire result is ultimately stored in a single KernelContent message).

Another option to consider might be to utilize the new SK Vector Store connector architecture to target Azure AI Search and create a plugin/function around this.

https://github.com/microsoft/semantic-kernel/tree/main/dotnet/src/Connectors/Connectors.Memory.AzureAISearch