Integrating Elasticsearch and OpenAI in an ASP.NET Core Web API

In this article, we will walk through the process of creating an ASP.NET Core Web API that integrates Elasticsearch and OpenAI. The goal is to build a system that can answer questions based on data stored in Elasticsearch, using OpenAI to generate responses grounded in the retrieved data. This approach is powerful for applications such as customer support, virtual assistants, and recommendation systems.

1. Setting Up the ASP.NET Core Web API Project

a) Creating the Project

First, let’s create a new ASP.NET Core Web API project using the .NET CLI:

dotnet new webapi -n ElasticsearchOpenAIWebAPI

This command creates a new Web API project named ElasticsearchOpenAIWebAPI.

b) Installing Required Packages

Next, install the NEST and OpenAI packages to enable interaction with Elasticsearch and OpenAI:

dotnet add package NEST
dotnet add package OpenAI-API

NEST is the official library for communicating with Elasticsearch in C#. The OpenAI-API package will be used to interact with the OpenAI API.

2. Configuring appsettings.json

API keys and the Elasticsearch URL are typically stored in the appsettings.json file for easy access:

{
  "Elasticsearch": {
    "Uri": "https://0179425de9034d8596bdb1b192f8229b.us-central1.gcp.cloud.es.io:443",
    "ApiKey": "your_elasticsearch_api_key"
  },
  "OpenAI": {
    "ApiKey": "your_openai_api_key"
  }
}

This configuration file contains connection information for both Elasticsearch and the OpenAI API, which will be loaded into the application during startup.

3. Creating Configuration Classes

Let’s create a class AppSettings.cs to map the settings from appsettings.json:

public class AppSettings
{
    public ElasticsearchSettings Elasticsearch { get; set; }
    public OpenAISettings OpenAI { get; set; }
}

public class ElasticsearchSettings
{
    public string Uri { get; set; }
    public string ApiKey { get; set; }
}

public class OpenAISettings
{
    public string ApiKey { get; set; }
}

This class structure will allow us to access the configuration settings in our services.

4. Setting Up the Elasticsearch Service Using NEST

a) Creating the Elasticsearch Service

The ElasticsearchService class will use the NEST client to interact with Elasticsearch:

using Microsoft.Extensions.Options;
using Nest;

public class ElasticsearchService
{
    private readonly ElasticClient _client;

    public ElasticsearchService(IOptions<AppSettings> settings)
    {
        var connectionSettings = new ConnectionSettings(new Uri(settings.Value.Elasticsearch.Uri))
            .ApiKeyAuthentication(settings.Value.Elasticsearch.ApiKey);
        _client = new ElasticClient(connectionSettings);
    }

    public async Task<List<SearchResult>> SearchAsync(string query)
    {
        var searchResponse = await _client.SearchAsync<SearchResult>(s => s
            .Index("search-olx")
            .Query(q => q
                .MultiMatch(mm => mm
                    .Query(query)
                    .Fields(f => f
                        .Field(p => p.BodyContent)
                        .Field(p => p.Title)
                    )
                )
            )
            .Size(3)
        );

        return searchResponse.Documents.ToList();
    }
}

b) Detailed Explanation

• ConnectionSettings: This class defines the connection settings for Elasticsearch. The URI and API key are retrieved from the appsettings.json.

• ElasticClient: The ElasticClient is the main client used to interact with Elasticsearch, configured with the ConnectionSettings.

• SearchAsync: This method executes a search query on Elasticsearch. It uses a MultiMatch query to search across multiple fields (body_content and title). The method returns the top 3 relevant documents.

📌 Summary Note:

• ElasticsearchService: A service class that interacts with an Elasticsearch instance using the NEST client.
• ElasticClient: Main client used to perform Elasticsearch operations, configured via ConnectionSettings with the URI and API key.
• SearchAsync Method: An asynchronous method that performs a search on the “search-olx” index, using a MultiMatch query to search across the BodyContent and Title fields, returning the top 3 results.
• Purpose: Enables efficient search and retrieval of relevant documents from Elasticsearch based on user queries.

c) SearchResult Model

Create a SearchResult model to map the search results:

public class SearchResult
{
    public string BodyContent { get; set; }
    public string Title { get; set; }
}

This model maps the fields of the documents returned by Elasticsearch.

5. Setting Up the OpenAI Service

a) Creating the OpenAI Service

The OpenAIService class will interact with the OpenAI API to generate responses based on the provided context:

using OpenAI_API;
using OpenAI_API.Chat;
using Microsoft.Extensions.Options;

public class OpenAIService
{
    private readonly OpenAIAPI _api;

    public OpenAIService(IOptions<AppSettings> settings)
    {
        _api = new OpenAIAPI(settings.Value.OpenAI.ApiKey);
    }

    public async Task<string> GetCompletionAsync(string context, string question)
    {
        var chat = _api.Chat.CreateConversation();
        chat.AppendSystemMessage($@"
            You are an assistant for question-answering tasks.
            Answer questions truthfully and factually using only the context presented.
            If you don't know the answer, just say that you don't know, don't make up an answer.
            Use markdown format for code examples.
            You are correct, factual, precise, and reliable.
        ");
        chat.AppendUserInput($"Context: {context}");
        chat.AppendUserInput(question);

        var result = await chat.GetResponseFromChatbotAsync();
        return result;
    }
}

b) Detailed Explanation

• OpenAIAPI: This class is used to interact with the OpenAI API. The API key is provided via configuration.

• GetCompletionAsync: This method creates a conversation with the OpenAI model, where the context is passed as part of the user input. OpenAI generates a response based on the context and the user’s question.

📌 Summary Note:

• OpenAIService: A service class that interacts with the OpenAI API to generate responses based on provided context.
• OpenAIAPI: Main client used to communicate with OpenAI, initialized with an API key from configuration settings.
• GetCompletionAsync Method: An asynchronous method that creates a conversation with the OpenAI model. It provides:
  • System Instructions: Detailed guidelines for how the AI should behave:
    • Be Truthful and Factual: The AI is instructed to answer questions using only the provided context and not to fabricate any information.
    • Context-Only Answers: The AI should rely exclusively on the context provided to generate responses, ensuring accuracy and relevance.
    • Honesty in Uncertainty: If the AI doesn’t know the answer, it is directed to admit that rather than guessing or creating information.
    • Code Formatting: If the response involves code, it should be formatted using Markdown for clarity.
    • Tone and Style: The AI should be correct, factual, precise, and reliable in its responses.
  • User Inputs: The actual context (from previous searches or provided data) and the specific question from the user.
  • The method generates and returns a response based on this context and question.
• Prompt Explanation: The prompt carefully instructs the AI on the nature of its response, emphasizing accuracy, the use of given context, and appropriate formatting. This ensures that the AI’s output is not only relevant but also trustworthy and clear, particularly in technical scenarios involving code.
• Purpose: Enables dynamic and context-aware response generation using OpenAI, tailored to specific user queries and the provided context, while ensuring the responses adhere to the specified guidelines.

6. Creating the Web API Controller

Create a controller to expose the endpoint that combines Elasticsearch and OpenAI:

using Microsoft.AspNetCore.Mvc;

[ApiController]
[Route("api/[controller]")]
public class SearchController : ControllerBase
{
    private readonly ElasticsearchService _elasticsearchService;
    private readonly OpenAIService _openAIService;

    public SearchController(ElasticsearchService elasticsearchService, OpenAIService openAIService)
    {
        _elasticsearchService = elasticsearchService;
        _openAIService = openAIService;
    }

    [HttpGet("ask")]
    public async Task<IActionResult> Ask([FromQuery] string question)
    {
        var results = await _elasticsearchService.SearchAsync(question);
        var context = string.Join("\n", results.Select(r => r.BodyContent));

        var completion = await _openAIService.GetCompletionAsync(context, question);

        return Ok(completion);
    }
}

a) Detailed Explanation

• Ask: This is the main endpoint that receives the user’s question via a query string (?question=...). It first queries Elasticsearch using the ElasticsearchService, then passes the resulting context to OpenAIService, which generates a response based on the retrieved data.

• HttpGet(“ask”): This attribute defines an HTTP GET endpoint that can be accessed at api/search/ask.

7. Configuring Program.cs

In Program.cs, register the services with the dependency injection container:

using Microsoft.Extensions.DependencyInjection;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.Configure<AppSettings>(builder.Configuration);
builder.Services.AddSingleton<ElasticsearchService>();
builder.Services.AddSingleton<OpenAIService>();

builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.Run();

a) Detailed Explanation

• Configure: Maps the configuration settings from appsettings.json to the AppSettings class.

• AddSingleton() and AddSingleton(): Registers the services as singletons, ensuring a single instance of each service is used throughout the application’s lifecycle.

• MapControllers(): Configures routing to direct HTTP requests to the appropriate controllers.

8. Running the Application

To run the application, use the following command:

dotnet run

You can now access the endpoint GET /api/search/ask?question=your_question. This endpoint retrieves relevant information from Elasticsearch and uses OpenAI to generate a response based on the context.

9. Understanding the Application Workflow

a) API Workflow

1. Receiving the Question: The user sends a GET request to the /api/search/ask endpoint, passing a question as a query string.

2. Elasticsearch Search:

   • The question is sent to the ElasticsearchService.
   • This service searches the specified indices (in this case, search-olx) using a MultiMatch query on the body_content and title fields.
   • The most relevant results (up to 3) are returned and converted into a list of SearchResult objects.

3. Context Generation:

   • The content of the retrieved documents is combined into a single string (context).
   • This string represents the context that will be used to answer the question.

4. Generating a Response with OpenAI:

   • The combined context is passed to the OpenAIService along with the original question.
   • The OpenAIService creates the conversation with OpenAI and provides the context as part of the input. OpenAI then generates a response based on the context and the user’s question.

5. Returning the Response:

   • The response generated by OpenAI is returned to the user in the HTTP response body.

b) Benefits of the Architecture

• Modularity: The code is well-modularized, with responsibilities separated between services, controllers, and configurations. This makes it easier to maintain and extend the application in the future.

• Scalability: Both Elasticsearch and OpenAI are scalable services. This solution can handle large volumes of data and requests as long as the server resources are appropriately scaled.

• Security: API keys are stored securely in the appsettings.json file. For added security in a production environment, consider using user secrets or a secrets management service.

• Flexibility: The API can be easily adapted to different Elasticsearch indices or question types by modifying the search logic or the instructions passed to OpenAI.

c) Performance Considerations

• Limiting Results: The code limits the number of Elasticsearch results to 3. Depending on the application, you can adjust this limit to improve performance or provide more context to OpenAI.

• Response Time: Since the process involves calls to both Elasticsearch and OpenAI, the API’s response time may vary. For critical applications, you might consider implementing caching, optimizing Elasticsearch queries, or even preprocessing data.

d) Potential Extensions

• Enhanced Search Capabilities: You can improve the Elasticsearch queries by adding filters, aggregations, or more relevant fields depending on your use case.

• Context Refinement: The method of constructing the context could be refined. For example, you might add more metadata to the context, such as titles, dates, or other useful information.

• Adding Authentication: If you’re building a public or protected API, consider adding authentication and authorization using JWT or OAuth.

• Monitoring and Logging: Implementing robust logging and performance monitoring will help you identify bottlenecks and improve the API’s efficiency.

• Database Integration: Depending on your needs, you could integrate the application with relational or NoSQL databases to store additional information or perform complex data operations not ideal for Elasticsearch.

10. Conclusion

In this article, we explored how to integrate Elasticsearch and OpenAI within an ASP.NET Core Web API. This setup allows you to create a powerful system capable of retrieving data from Elasticsearch and generating context-aware responses using OpenAI.

The example provided offers a solid starting point for building more complex applications that require advanced data retrieval and natural language processing capabilities. Whether you’re developing customer support tools, virtual assistants, or recommendation systems, this approach provides the modularity, scalability, and flexibility needed to grow and adapt as your requirements evolve.

With a clear understanding of the application workflow, performance considerations, and potential extensions, you’re well-equipped to further develop and customize your API to meet specific use cases. This combination of Elasticsearch and OpenAI in an ASP.NET Core Web API opens up a wide range of possibilities for creating intelligent and responsive applications.