Stochastic Coder

In the world of conversational AI, delivering a seamless user experience hinges on the ability to maintain meaningful interactions and context across chat sessions. Chat history is a critical component in achieving this, as it preserves the memory of past exchanges, enabling contextually relevant and personalized responses. While maintaining chat history is straightforward for a single session on a single device, challenges arise when users want to resume conversations after a period of time or continue seamlessly across multiple devices.

Fortunately, with Azure Cosmos DB, storing and managing chat history at any scale becomes effortless. In this article, we’ll explore how Azure Cosmos DB, integrated with Azure OpenAI Service and Semantic Kernel, provides a robust, scalable, and efficient solution to ensure continuity and context in conversational AI applications.

This article provides an in-depth exploration of the chat history functionality and its associated code from the article Enhance Customer Support with Semantic Kernel and Azure OpenAI.

In this article

• Overview of Azure Cosmos DB
• Benefits of Using Azure Cosmos DB for Chat History
• How to Use Azure Cosmos DB for Chat History with Semantic Kernel
  • Setting Up an Azure Cosmos DB Container for Chat History Management
  • Chat History Data Management in Azure Cosmos DB
  • Synchronizing Messages with Chat History Data
• Chat History Testing with Azure Cosmos DB
• Conclusion

Overview of Azure Cosmos DB

Azure Cosmos DB is a fully managed NoSQL, relational, and vector database service designed for high performance, reliability, and scalability. It supports multiple data models, including key-value, document, graph, and column-family, and offers seamless integration for modern applications with diverse data handling needs. Cosmos DB delivers single-digit millisecond response times and automatic, instant scalability to handle workloads of any size.

Benefits of Using Azure Cosmos DB for Chat History

Azure Cosmos DB provides reliable, secure, and efficient storage for chat history, making it the ideal solution for Azure OpenAI applications thanks to its scalability, performance, and seamless integration with the Azure ecosystem.

Key Benefits:

• Scalability: Easily handles growing chat volumes with horizontal scaling, supporting millions of records without performance loss.
• Flexible Schema: Effortlessly stores text, metadata (timestamps, user IDs, session IDs), and context vectors with its schema flexibility.
• Low Latency: With its global distribution capabilities, Cosmos DB ensures low latency for data read and write operations. This is critical in real-time conversational AI, where users expect instant responses.
• Rich Query Capabilities: Enables advanced queries, such as filtering by timestamps, retrieving conversations by sessions (or users), or semantic searches with embeddings.
• Security and Compliance: Offers encryption at rest, private endpoints, and RBAC to protect user data and meet compliance standards.

How to Use Azure Cosmos DB for Chat History with Semantic Kernel

Setting Up an Azure Cosmos DB Container for Chat History Management

Within the project repository, Bicep scripts are used to deploy the Azure Cosmos DB service, including the database and container. The example below defines an Azure Cosmos DB database named chatdatabase with a container called chathistory for storing chat sessions and messages. It includes a 24-hour time-to-live (TTL) policy for automatic data deletion and an indexing policy focused solely on the sessionId path. If you prefer to create your Azure Cosmos DB instance directly through the Azure portal, please follow this QuickStart.

infra/app/database.bicep

// Define the database for the chat application
var database = {
  name: 'chatdatabase' 
}

// Define the containers for the database
var containers = [
  {
    name: 'chathistory' // Container for storing chat sessions and messages (chat history)
    partitionKeyPaths: [
      '/id' 
    ]
    ttlValue: 86400 // Time-to-live (TTL) for automatic deletion of data after 24 hours (86400 seconds)
    indexingPolicy: {
      automatic: true // Automatically index new data
      indexingMode: 'consistent' // Ensure data is indexed immediately
      includedPaths: [
        {
          path: '/sessionId/?' 
        }
      ]
      excludedPaths: [
        {
          path: '/*' // Exclude all other paths from indexing
        }
      ]
    }
    vectorEmbeddingPolicy: {
      vectorEmbeddings: [] // Placeholder for future vector embedding configuration
    }
  }

Chat History Data Management in Azure Cosmos DB

In this solution, chat history is organized by sessions (identified by sessionId), which can represent either a web session or a chat conversation, depending on your implementation requirements. The corresponding code for managing chat history can be found in: src/ChatAPI/Data/ChatHistoryData.cs.

To begin, we need to define the ChatMessage class, which will serve as the schema for storing chat history data in Azure Cosmos DB. This class will outline the structure of each chat message, including essential properties such as the sessionId, timestamp, message, and role. By defining this schema, we can ensure that the data is organized consistently and can be efficiently stored and queried within the Azure Cosmos DB container

public class ChatMessage
{
    public string id { get; set; }
    public string sessionid { get; set; }
    public string message { get; set; }
    public string role { get; set; } // "User" or "Assistant"
    public DateTime Timestamp { get; set; }
}

To populate the Semantic Kernel Chat History with session-based data from Azure Cosmos DB, we will use the InitializeChatHistoryFromCosmosDBAsync method from ChatHistoryData. This method asynchronously initializes the ChatHistory object by retrieving messages from Cosmos DB using the provided sessionId. It calls GetMessagesBySessionIdAsync to fetch all associated messages, checks each message’s role to identify whether it’s from the “user” or “assistant,” and then adds the message to the appropriate collection in the ChatHistory object. This ensures the chat history is organized correctly with both user and assistant messages.

public async Task InitializeChatHistoryFromCosmosDBAsync(ChatHistory chatHistory, string sessionId)
    {
        var messages = await GetMessagesBySessionIdAsync(sessionId);

        foreach (var message in messages)
        {
            if (message.role == "user")
            {
                chatHistory.AddUserMessage(message.message);
            }
            else if (message.role == "assistant")
            {
                chatHistory.AddAssistantMessage(message.message);
            }
        }
    }

private async Task<List<ChatMessage>> GetMessagesBySessionIdAsync(string sessionId)
    {
        var container = _cosmosClient.GetContainer(_databaseName, _containerName);
        var query = container.GetItemQueryIterator<ChatMessage>(
            new QueryDefinition("SELECT * FROM c WHERE c.sessionid = @sessionId")
            .WithParameter("@sessionId", sessionId)
        );

        var messages = new List<ChatMessage>();

        while (query.HasMoreResults)
        {
            var response = await query.ReadNextAsync();
            messages.AddRange(response);
        }

        return messages;
    }

The AddMessageToHistoryAndSaveAsync method asynchronously adds a new message to the chat history and saves it to Azure Cosmos DB based on the specified session ID, role, and content. It checks the role of the message—either “user” or “assistant”—and accordingly adds the message to the appropriate collection within the Semantic Kernel chatHistory object (user messages or assistant messages).

  public async Task AddMessageToHistoryAndSaveAsync(ChatHistory chatHistory, string sessionId, string role, string content)
    {
        if (role == "user")
        {
            chatHistory.AddUserMessage(content);
            await AddUserMessageAsync(sessionId, content);
        }
        else if (role == "assistant")
        {
            chatHistory.AddAssistantMessage(content);
            await AddAssistantMessageAsync(sessionId, content);
        }

    }

For clarity AddAssistantMessageAsync and AddUserMessageAsync are two separate methods, whereas it may make sense to consolidate the logic into a single method.

Synchronizing Messages with Chat History Data

The .NET web API integrates the ChatHistoryData object with ChatService to efficiently manage chat history. In the GetResponseAsync method, _chatHistoryData retrieves and initializes session-based chat history from Azure Cosmos DB when the Semantic Kernel chatHistory object contains only the system message (_chatHistory.Count == 1). Semantic Kernel serves as an orchestrator, seamlessly retrieving relevant chat history from Cosmos DB and passing it to the Azure OpenAI model. This integration ensures the system dynamically adapts to user interactions while maintaining a consistent and contextually aware conversational flow.

The AddMessageToHistoryAndSaveAsync method ensures synchronization between user prompts (“user messages”), AI responses (“assistant messages”), the Semantic Kernel chat history, and Azure Cosmos DB storage. This guarantees that the chat history remains consistent across in-memory and persistent storage, maintaining context for AI responses.

src/ChatAPI/Services/ChatService.cs

 public async Task<string> GetResponseAsync( string question, string sessionId)
    {
         _logger.LogInformation("Chat History Count {count}",chatHistory.Count );

        if(_chatHistory.Count ==1)
        {
            _logger.LogInformation("Init Chat History");
            await _chatHistoryData.InitializeChatHistoryFromCosmosDBAsync(_chatHistory,sessionId);
        }

        await _chatHistoryData.AddMessageToHistoryAndSaveAsync(_chatHistory,sessionId,"user",question);

        IChatCompletionService chatCompletion = kernel.GetRequiredService<IChatCompletionService>();
        OpenAIPromptExecutionSettings openAIPromptExecutionSettings = new() 
        {
            FunctionChoiceBehavior = FunctionChoiceBehavior.Auto()
        };

        
        ChatMessageContent  response  = await  chatCompletion.GetChatMessageContentAsync(
        _chatHistory,
        executionSettings: openAIPromptExecutionSettings,
        kernel: kernel);

        string resp = string.Join(" ",response.Items);
        _logger.LogInformation("Response {response}",resp );
        await _chatHistoryData.AddMessageToHistoryAndSaveAsync(_chatHistory,sessionId,"assistant",resp);

        return JsonSerializer.Serialize(new { resp });
 
    }

Chat History Testing with Azure Cosmos DB

To see this in action, follow the steps outlined in Enhance Customer Support with Semantic Kernel and Azure OpenAI to deploy the solution. Once deployed to Azure and the web application is running, test the chat interface. During your interactions, your messages and chat history will be managed for your session and stored in Azure Cosmos DB.

Visualize the messages from the Chat History container within the Chat database by utilizing the Azure Cosmos DB Data Explorer. As show below:

Conclusion

Maintaining chat history is essential for enhancing the user experience in conversational AI, allowing for contextually relevant and personalized interactions. By leveraging Azure Cosmos DB in conjunction with Azure OpenAI Service and Semantic Kernel, developers can efficiently store and manage chat history at scale, ensuring seamless conversations across time and devices. This integration offers a powerful solution for building scalable and robust AI-driven applications that provide continuity and context, ultimately improving user satisfaction and engagement. The insights and code explored in this article serve as a foundation for creating effective, context-aware conversational experiences.