[WIP] Use Agent Framework with local and cloud models #33102
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a multi-agent AI workflow system for the trip planner sample, adds multi-language support for AI-generated itineraries, and updates AI-related dependencies. The implementation uses Microsoft Agents Framework to orchestrate a 4-agent workflow (Travel Planner → Researcher → Itinerary Planner → conditional Translator) and includes both cloud-based (OpenAI) and local (Apple Intelligence) model support.
Key Changes
- Multi-agent workflow architecture: Replaced direct IChatClient usage with a 4-agent workflow system using Microsoft.Agents.AI packages for more modular itinerary generation
- Multi-language support: Added LanguagePreferenceService and UI controls to generate itineraries in 10 different languages with conditional translation
- Apple Intelligence integration: Implemented NonFunctionInvokingChatClient wrapper to prevent double tool invocation when using on-device models with Agent Framework
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| src/AI/src/Essentials.AI/PublicAPI/*/PublicAPI.Unshipped.txt | Updated/removed API signatures for AddPlatformChatClient and AddAppleIntelligenceChatClient methods |
| src/AI/src/Essentials.AI/Extensions.cs | Removed all extension method implementations, leaving empty class with XML documentation |
| src/AI/samples/Essentials.AI.Sample/Services/NonFunctionInvokingChatClient.cs | New wrapper to prevent Agent Framework from double-invoking tools on clients that handle invocation internally |
| src/AI/samples/Essentials.AI.Sample/Services/LanguagePreferenceService.cs | New service tracking user's preferred language for AI responses with 10 supported languages |
| src/AI/samples/Essentials.AI.Sample/Services/ItineraryWorkflowExtensions.cs | New 4-agent workflow registration with specialized executors for parsing, research, planning, and translation |
| src/AI/samples/Essentials.AI.Sample/Services/ItineraryService.cs | Refactored to use workflow-based itinerary generation instead of direct chat client interactions |
| src/AI/samples/Essentials.AI.Sample/Services/LandmarkDataService.cs | Renamed GetLandmarkNames to GetDestinationNames for clarity |
| src/AI/samples/Essentials.AI.Sample/ViewModels/LandmarksViewModel.cs | Added language preference integration and AvailableLanguages property |
| src/AI/samples/Essentials.AI.Sample/Pages/LandmarksPage.xaml | Added floating language selection button in top-right corner |
| src/AI/samples/Essentials.AI.Sample/Pages/LandmarksPage.xaml.cs | Added OnLanguageButtonClicked event handler for language selection |
| src/AI/samples/Essentials.AI.Sample/Pages/TripPlanningPage.xaml | Added Padding="0" to back button for consistency |
| src/AI/samples/Essentials.AI.Sample/MauiProgram.cs | Refactored AI service registration for keyed services, added workflow registration, changed UseCloudAI default to true, and set debug logging to Trace level |
| src/AI/samples/Essentials.AI.Sample/Essentials.AI.Sample.csproj | Added Microsoft.Agents.AI packages (v1.0.0-preview.251204.1) and updated OpenAI-related packages |
| eng/Versions.props | Bumped Microsoft.Extensions.AI packages from 10.0.0 to 10.0.1 |
| src/AI/samples/Essentials.AI.Sample/Services/FindPointsOfInterestTool.cs | Deleted - functionality moved into ItineraryPlannerExecutor |
| src/AI/samples/Essentials.AI.Sample/Services/ToolLookup.cs | Namespace changed from Maui.Controls.Sample.Services.Tools to Maui.Controls.Sample.Services |
| abstract Microsoft.Maui.Essentials.AI.ChatClientBase.GetStreamingResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Collections.Generic.IAsyncEnumerable<Microsoft.Extensions.AI.ChatResponseUpdate!>! | ||
| override Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient.GetResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task<Microsoft.Extensions.AI.ChatResponse!>! | ||
| override Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient.GetStreamingResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Collections.Generic.IAsyncEnumerable<Microsoft.Extensions.AI.ChatResponseUpdate!>! | ||
| static Microsoft.Maui.Essentials.AI.Extensions.AddAppleIntelligenceChatClient(this Microsoft.Extensions.DependencyInjection.IServiceCollection! services, Microsoft.Extensions.DependencyInjection.ServiceLifetime lifetime = Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton) -> Microsoft.Extensions.DependencyInjection.IServiceCollection! |
There was a problem hiding this comment.
The AddPlatformChatClient method that was removed from this PublicAPI file needs to be implemented somewhere or this removal is incorrect. If this method is still a public API on this platform, it should remain in the PublicAPI.Unshipped.txt file and the corresponding implementation should exist in the codebase.
| abstract Microsoft.Maui.Essentials.AI.ChatClientBase.GetResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task<Microsoft.Extensions.AI.ChatResponse!>! | ||
| abstract Microsoft.Maui.Essentials.AI.ChatClientBase.GetStreamingResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Collections.Generic.IAsyncEnumerable<Microsoft.Extensions.AI.ChatResponseUpdate!>! | ||
| static Microsoft.Maui.Essentials.AI.Extensions.AddPlatformChatClient(this Microsoft.Extensions.DependencyInjection.IServiceCollection! services, Microsoft.Extensions.DependencyInjection.ServiceLifetime lifetime = Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton) -> Microsoft.Extensions.DependencyInjection.IServiceCollection! | ||
| static Microsoft.Maui.Essentials.AI.Extensions.AddPlatformChatClient(this Microsoft.Extensions.DependencyInjection.IServiceCollection! services, Microsoft.Extensions.DependencyInjection.ServiceLifetime lifetime = Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton, bool useAgentFrameworkWrapper = false) -> Microsoft.Extensions.DependencyInjection.IServiceCollection! |
There was a problem hiding this comment.
The AddPlatformChatClient method declared in PublicAPI.Unshipped.txt does not exist in Extensions.cs. The Extensions class is currently empty. You need to either remove this API declaration if the method is no longer needed, or implement the method in Extensions.cs to match the public API signature.
| static Microsoft.Maui.Essentials.AI.Extensions.AddPlatformChatClient(this Microsoft.Extensions.DependencyInjection.IServiceCollection! services, Microsoft.Extensions.DependencyInjection.ServiceLifetime lifetime = Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton, bool useAgentFrameworkWrapper = false) -> Microsoft.Extensions.DependencyInjection.IServiceCollection! |
|
|
||
| logger.LogTrace("[TravelPlannerExecutor] Raw response: {Response}", response.Text); | ||
|
|
||
| var result = JsonSerializer.Deserialize<TravelPlanResult>(response.Text, s_jsonOptions)!; |
There was a problem hiding this comment.
The null-forgiving operator (!) is used on JsonSerializer.Deserialize without null checking. If deserialization fails or returns null, this will cause a NullReferenceException. Add proper null handling to gracefully handle deserialization failures.
| var result = JsonSerializer.Deserialize<TravelPlanResult>(response.Text, s_jsonOptions)!; | |
| var result = JsonSerializer.Deserialize<TravelPlanResult>(response.Text, s_jsonOptions); | |
| if (result == null) | |
| { | |
| logger.LogError("[TravelPlannerExecutor] Failed to deserialize response to TravelPlanResult. Response text: {ResponseText}", response.Text); | |
| throw new InvalidOperationException("Failed to parse travel plan result from agent response."); | |
| } |
| // Register the Apple Intelligence client as IChatClient to allow direct use | ||
| builder.Services.AddSingleton<IChatClient>(sp => sp.GetRequiredService<AppleIntelligenceChatClient>()); | ||
|
|
||
| // Register the Agent Framework wrapper as the default IChatClient | ||
| // This prevents double tool invocation when using Microsoft Agent Framework | ||
| // TODO: workaround for https://github.com/dotnet/extensions/pull/7126 | ||
| builder.Services.AddKeyedSingleton<IChatClient>("local-model", (sp, _) => | ||
| { | ||
| builder.Services.AddPlatformChatClient(); | ||
| } | ||
| var appleClient = sp.GetRequiredService<AppleIntelligenceChatClient>(); | ||
| var loggerFactory = sp.GetRequiredService<ILoggerFactory>(); | ||
| return new NonFunctionInvokingChatClient(appleClient, loggerFactory, sp); | ||
| }); |
There was a problem hiding this comment.
The service registration registers IChatClient twice as a singleton with different implementations. Line 74 registers it as AppleIntelligenceChatClient, and line 79 registers it as a keyed service "local-model". However, there's a conflict: you cannot have both a non-keyed singleton and a keyed singleton for the same service type without potential resolution issues. Consider removing line 74 if only the keyed service is needed, or clarify the intended registration pattern.
| /// Extension methods for registering platform-specific AI chat clients. | ||
| /// </summary> | ||
| public static class Extensions | ||
| { |
There was a problem hiding this comment.
The Extensions class now has XML documentation claiming it contains "Extension methods for registering platform-specific AI chat clients", but the class is completely empty. Either remove this misleading documentation or implement the methods that the PublicAPI files declare.
| { | |
| { | |
| /// <summary> | |
| /// Registers a platform-specific AI chat client. | |
| /// </summary> | |
| /// <param name="services">The service collection to add the client to.</param> | |
| /// <returns>The updated service collection.</returns> | |
| public static IServiceCollection AddPlatformSpecificAIChatClient(this IServiceCollection services) | |
| { | |
| // TODO: Implement platform-specific registration logic here. | |
| return services; | |
| } |
| catch | ||
| { | ||
| var toolLookup = new ToolLookup | ||
| { | ||
| Id = functionResult.CallId, | ||
| Result = functionResult.Result | ||
| }; | ||
|
|
||
| yield return new ItineraryStreamUpdate { ToolLookupResult = toolLookup }; | ||
| // Skip if JSON is invalid | ||
| } |
There was a problem hiding this comment.
The catch block silently swallows all exceptions without logging or handling them. This makes debugging difficult and hides potential issues. At minimum, log the exception so that deserialization failures are visible during development and troubleshooting.
| { "Spanish", "es" }, | ||
| }; | ||
|
|
||
| public string SelectedLanguage { get; set; } = "English"; |
There was a problem hiding this comment.
The SupportedLanguages dictionary values (language codes like "zh", "en", etc.) are defined but never used in the codebase. The code only compares against language names (keys) like "English", "French", etc. Either use these language codes for proper i18n support or remove them if they serve no purpose.
| public string SelectedLanguage { get; set; } = "English"; | |
| public string SelectedLanguage { get; set; } = "English"; | |
| /// <summary> | |
| /// Gets the language code for the currently selected language. | |
| /// Returns null if the selected language is not supported. | |
| /// </summary> | |
| public string? SelectedLanguageCode | |
| { | |
| get | |
| { | |
| if (SupportedLanguages.TryGetValue(SelectedLanguage, out var code)) | |
| { | |
| return code; | |
| } | |
| return null; | |
| } | |
| } |
| public ToolLookup? ToolLookup { get; init; } | ||
| public ToolLookup? ToolLookupResult { get; init; } |
There was a problem hiding this comment.
The ItineraryStreamUpdate record defines ToolLookup and ToolLookupResult properties, but these are never populated by the new workflow-based implementation. The StreamItineraryAsync method only sets PartialItinerary and IsTranslated. Either remove these unused properties or implement the tool lookup tracking functionality if it's still needed.
| public ToolLookup? ToolLookup { get; init; } | |
| public ToolLookup? ToolLookupResult { get; init; } |
| #nullable enable | ||
| Microsoft.Maui.Essentials.AI.Extensions | ||
| static Microsoft.Maui.Essentials.AI.Extensions.AddPlatformChatClient(this Microsoft.Extensions.DependencyInjection.IServiceCollection! services, Microsoft.Extensions.DependencyInjection.ServiceLifetime lifetime = Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton) -> Microsoft.Extensions.DependencyInjection.IServiceCollection! | ||
| static Microsoft.Maui.Essentials.AI.Extensions.AddPlatformChatClient(this Microsoft.Extensions.DependencyInjection.IServiceCollection! services, Microsoft.Extensions.DependencyInjection.ServiceLifetime lifetime = Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton, bool useAgentFrameworkWrapper = false) -> Microsoft.Extensions.DependencyInjection.IServiceCollection! |
There was a problem hiding this comment.
The AddPlatformChatClient method declared in PublicAPI.Unshipped.txt does not exist in Extensions.cs. The Extensions class is currently empty. You need to either remove this API declaration if the method is no longer needed, or implement the method in Extensions.cs to match the public API signature.
| static Microsoft.Maui.Essentials.AI.Extensions.AddPlatformChatClient(this Microsoft.Extensions.DependencyInjection.IServiceCollection! services, Microsoft.Extensions.DependencyInjection.ServiceLifetime lifetime = Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton, bool useAgentFrameworkWrapper = false) -> Microsoft.Extensions.DependencyInjection.IServiceCollection! |
| abstract Microsoft.Maui.Essentials.AI.ChatClientBase.GetResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task<Microsoft.Extensions.AI.ChatResponse!>! | ||
| abstract Microsoft.Maui.Essentials.AI.ChatClientBase.GetStreamingResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Collections.Generic.IAsyncEnumerable<Microsoft.Extensions.AI.ChatResponseUpdate!>! | ||
| override Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient.GetResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Threading.Tasks.Task<Microsoft.Extensions.AI.ChatResponse!>! | ||
| override Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient.GetStreamingResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Collections.Generic.IAsyncEnumerable<Microsoft.Extensions.AI.ChatResponseUpdate!>! |
There was a problem hiding this comment.
The AddAppleIntelligenceChatClient and AddPlatformChatClient methods that were removed from this PublicAPI file need to be implemented somewhere or these removals are incorrect. If these methods are still public APIs on this platform, they should remain in the PublicAPI.Unshipped.txt file and the corresponding implementation should exist in the codebase.
| override Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient.GetStreamingResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Collections.Generic.IAsyncEnumerable<Microsoft.Extensions.AI.ChatResponseUpdate!>! | |
| override Microsoft.Maui.Essentials.AI.AppleIntelligenceChatClient.GetStreamingResponseAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage!>! messages, Microsoft.Extensions.AI.ChatOptions? options = null, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> System.Collections.Generic.IAsyncEnumerable<Microsoft.Extensions.AI.ChatResponseUpdate!>! | |
| Microsoft.Maui.Essentials.AI.Extensions.AddAppleIntelligenceChatClient(Microsoft.Extensions.DependencyInjection.IServiceCollection services) -> Microsoft.Extensions.DependencyInjection.IServiceCollection | |
| Microsoft.Maui.Essentials.AI.Extensions.AddPlatformChatClient(Microsoft.Extensions.DependencyInjection.IServiceCollection services) -> Microsoft.Extensions.DependencyInjection.IServiceCollection |
This pull request introduces significant enhancements to the AI-powered trip planner sample, focusing on multi-language support for AI responses, improved AI workflow integration, and updated dependencies. The most important changes include the addition of a language preference service, UI updates to allow language selection, refactoring the itinerary generation to use a multi-agent workflow, and dependency updates for AI and related packages.
Multi-language support and UI enhancements:
LanguagePreferenceServiceto track the user's preferred language for AI responses, including support for several languages.LandmarksPageUI to include a floating language selection button, and implemented the logic to update the selected language in the view model when the user chooses a language. [1] [2]AI workflow and service integration:
ItineraryServiceto use the new 4-agent AI workflow (Microsoft.Agents.AI.Workflows), streamlining the process of generating itineraries and handling streaming AI responses.MauiProgram.cs, including support for both cloud-based and local (Apple Intelligence) chat clients, and ensured correct service registration for multi-agent workflows.Dependency and configuration updates:
Microsoft.Extensions.AI.OpenAI,OpenAI, and added new dependencies forMicrosoft.Agents.AI,Hosting, andWorkflows. [1] [2]MicrosoftExtensionsAIandMicrosoftExtensionsAIAbstractionsinVersions.props.Other improvements:
Tracein debug builds for more detailed diagnostics.GetLandmarkNamestoGetDestinationNamesfor clarity.Configuration changes:
<clear />inNuGet.configto potentially allow inherited package sources.