Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve automated browser testing with real server #4892

Open
danroth27 opened this issue May 23, 2018 · 60 comments
Open

Improve automated browser testing with real server #4892

danroth27 opened this issue May 23, 2018 · 60 comments
Labels
affected-most This issue impacts most of the customers area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates enhancement This issue represents an ask for new feature or an enhancement to an existing one feature-mvc-testing MVC testing package severity-major This label is used by an internal tool
Milestone

Comments

@danroth27
Copy link
Member

Feedback from @shanselman:

We should do better with testing. There's issues with moving from the inside out:

LEVELS OF TESTING

  • GOOD - Unit Testing - Make a PageModel and call On Get
  • GOOD - Functional Testing - Make a WebApplicationFactory and make in-memory HTTP calls
  • BAD - Automated Browser Testing with Real Server - can't easily use Selenium or call a real server. I shouldn't have to do this. We should decouple the WebApplicationFactory from the concrete TestServer implementation. @davidfowl
public class RealServerFactory<TStartup> : WebApplicationFactory<Startup> where TStartup : class
{
    IWebHost _host;
    public string RootUri { get; set; }
    public RealServerFactory()
    {
        ClientOptions.BaseAddress = new Uri("https://localhost");
    }

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.UseEnvironment("Development"); //will be default in RC1
    }

    protected override TestServer CreateServer(IWebHostBuilder builder)
    {
        //Real TCP port
        _host = builder.Build();
        _host.Start();
        RootUri = _host.ServerFeatures.Get<IServerAddressesFeature>().Addresses.FirstOrDefault();

        //Fake Server we won't use...sad!
        return new TestServer(new WebHostBuilder().UseStartup<TStartup>());
    }

    protected override void Dispose(bool disposing) 
    {
         base.Dispose(disposing);
         if (disposing) 
        {
                _host.Dispose();
            }
        }
}

/cc @javiercn

@mkArtakMSFT
Copy link
Member

As this is not a priority for 2.2 for now, moving to the backlog.

@davidfowl
Copy link
Member

@mkArtakMSFT just to clarify the concrete change we should make in 2.2:

We should make it so that it's possible to boot up the WebApplicationFactory (or another derived type) without a test server. It makes functional testing of your application absolutely trivial and the changes required to do this should be small.

/cc @javiercn

@steveoh
Copy link

steveoh commented May 24, 2018

I'm not sure I follow. How would these changes make it easier to use something like puppeteer and chrome headless/selenium for automated browser testing. What is a real server?

@shanselman
Copy link
Contributor

@steveoh the current set up allows for very convenient Unit Testing by spinning up the App/WebHost and talking to it 'in memory." So no HTTP, no security issue, you're basically talking HTTP without actually putting bytes on the wire (or localhost). Super useful.

But if you want to use Automated Browser Testing and have it driven by a Unit Test Framework, you are hampered by the concrete TestServer class and have to do a little dance (above) to fake it out and start up the WebHost yourself. I'd propose no regular person could/would figure that out.

David is saying that if WebApplicationFactory could be started without the fake TestServer we could easily and cleanly do Unit Testing AND Browser Automation Testing with the piece we have here.

@rgamage
Copy link

rgamage commented May 25, 2018

This is why we love Scott! Always the advocate for the mainstream developers out there wanting to use MS tools, but stymied by various obscure limitations. Thank you sir!

@giggio
Copy link
Contributor

giggio commented Jul 24, 2018

The provided workaround has problems.

The problems start with the fact that now we have 2 hosts. One from the TestServer, and another built to work with http. And WebApplicationFactory.Server references the first one, which we are not testing against. And to make things worse, calls to methods that configure the builder, such as WebApplicationFactory.WithWebHostBuilder will not work with the dummy TestServer.

Because of these problems we cannot easily interact with the real host, the one being tested. It is very common that I change some backend service and configure it before a test is run. Suppose I need to access some service that is not callable during development, only production. I replace that service when I configure the services collection with a fake, and then configure it to respond the way I want it to respond. I can't do that through WebApplicationFactory.Server.Host.Services.

The resulting code I have works, but it is ugly as hell, it is an ugly ugly hack.

I hope we can move this forward and do not require TestServer, maybe an IServer. I thought about forking the whole TestServer and WebApplicationFactory infrastructure, but as this is somewhat planning I'll wait. I hope it gets fixed soon. I am just commenting to complement that the provided workaround is not enough and to really work around you have to avoid WebApplicationFactory.Server and create a terrible work around it.

@bchavez
Copy link
Contributor

bchavez commented Sep 11, 2018

One way I solved this is to stop using WebApplicationFactory<T> all together.

Refactored Program.Main() to:

public static Task<int> Main(string[] args)
{
   return RunServer(args);
}
public static async Task<int> RunServer(string[] args,
                                        CancellationToken cancellationToken = default)
{
    ...
    CreateWebHostBuilder()
          .Build()
          .RunAsync(cancellationToken)
}

So, my unit test fixtures new up a var cts = new CancellationTokenSource(), then pass the cancellation token by calling Program.RunServer(new string[0], cts.Token). The server starts up as normal without having to create a separate process.

Make real HTTP calls like normal. When your done and your unit test completes, call cts.Cancel() to clean up and shutdown the HTTP server.

One down side is you need to copy appsettings.json to an output directory from your test project; potentially managing two appsettings.json files (one in your server project, and one in your test project). Maybe a linked project file could help eliminate the issue.

YMMV.

🤔 ✨ "Do you ponder the manner of things... yeah yeah... like glitter and gold..."

@aspnet-hello aspnet-hello transferred this issue from aspnet/Mvc Dec 14, 2018
@aspnet-hello aspnet-hello added this to the Backlog milestone Dec 14, 2018
@aspnet-hello aspnet-hello added 1 - Ready area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates enhancement This issue represents an ask for new feature or an enhancement to an existing one labels Dec 14, 2018
@giggio
Copy link
Contributor

giggio commented Jan 4, 2019

Hello everyone, this issue is still unresolved and seems to keep being postponed. Just so we know the planning, are you considering resolving this for ASP.NET Core 3.0? If not, do you have a workaround that does not incur on the problems I mentioned earlier (#4892 (comment))?

@BennyM
Copy link

BennyM commented Jan 5, 2019

This would be very welcome. The workaround mentioned here is great, except when you have html files and what not that you would also need to copy over.

@Tratcher
Copy link
Member

WebApplicationFactory is designed for a very specific in-memory scenario. Having it start Kestrel instead and wire up HttpClient to match is a fairly different feature set. We actually do have components that do this in Microsoft.AspNetCore.Server.IntegrationTesting but we've never cleaned them up for broader usage. It might make more sense to leave WebApplicationFactory for in-memory and improve the IntegrationTesting components for this other scenario.

@giggio
Copy link
Contributor

giggio commented Jan 11, 2019

I'm fine with that, as long as we have a good end to end testing story.

@M-Yankov
Copy link

Thanks guys for this discussion and specially to @bchavez
I succeeded to run Selenium tests in Azure pipelines with the idea to start a local server on the agent, run tests and stop the server.

My test class:

public class SelenuimSampleTests : IDisposable
{
    private const string TestLocalHostUrl = "http://localhost:8080";

    private readonly CancellationTokenSource tokenSource;

    public SelenuimSampleTests()
    {
        this.tokenSource = new CancellationTokenSource();

        string projectName = typeof(Web.Startup).Assembly.GetName().Name;

        string currentDirectory = Directory.GetCurrentDirectory();
        string webProjectDirectory = Path.GetFullPath(Path.Combine(currentDirectory, $@"..\..\..\..\{projectName}"));

        IWebHost webHost = WebHost.CreateDefaultBuilder(new string[0])
            .UseSetting(WebHostDefaults.ApplicationKey, projectName)
            .UseContentRoot(webProjectDirectory) // This will make appsettings.json to work.
            .ConfigureServices(services =>
            {
                services.AddSingleton(typeof(IStartup), serviceProvider =>
                {
                    IHostingEnvironment hostingEnvironment = serviceProvider.GetRequiredService<IHostingEnvironment>();
                    StartupMethods startupMethods = StartupLoader.LoadMethods(
                        serviceProvider, 
                        typeof(TestStartup),
                        hostingEnvironment.EnvironmentName);

                    return new ConventionBasedStartup(startupMethods);
                });
            })
            .UseEnvironment(EnvironmentName.Development)
            .UseUrls(TestLocalHostUrl)
            //// .UseStartup<TestStartUp>() // It's not working
            .Build();

        webHost.RunAsync(this.tokenSource.Token);
    }

    public void Dispose()
    {
        this.tokenSource.Cancel();
    }

    [Fact]
    public void TestWithSelenium()
    {
        string assemblyLocation = System.Reflection.Assembly.GetExecutingAssembly().Location;
        string currentDirectory = Path.GetDirectoryName(assemblyLocation);

        using (ChromeDriver driver = new ChromeDriver(currentDirectory))
        {
            driver.Navigate().GoToUrl(TestLocalHostUrl);
            IWebElement webElement = driver.FindElementByCssSelector("a.navbar-brand");

            string expected = typeof(Web.Startup).Assembly.GetName().Name;

            Assert.Equal(expected, webElement.Text);

            string appSettingValue = driver.FindElementById("myvalue").Text;
            const string ExpectedAppSettingsValue = "44";
            Assert.Equal(ExpectedAppSettingsValue, appSettingValue);
        }
    }
}

Very similar to the @bchavez 's example, I'm starting the CreateDefaultBuilder. That approach gives me a little bit more flexibility to set custom settings. For example using a TestStartup. the .UseStartup<TestStartUp>() wasn't working, so I used logic from WebHostBuilderExtensions.UseStartUp to set WebHostDefaults.ApplicationKey and configure services.

I'm sure there is better approach somewhere, but after а few days of research, nothing helped me. So I share this solution in case someone need it.

The test method is just for an example, there are different approaches to initialize browser driver.

@arkiaconsulting
Copy link

@M-Yankov I had your code work by using IWebHostBuilder.UseStartup<>, and IWebHost.StartAsync as well as IWebHost.StopAsync, without CancellationTokenSource.
=> .NetCore 2.2

@M-Yankov
Copy link

M-Yankov commented May 21, 2019

The idea of the IWebHostBuilder.UseStartup<> is to simplify the code. But for me it was not enough: after applying the UseStartup<TestStartup>(), ChromeWEbDriver cannot open the home page.
I didn't mention that the TestStartup class inherits the real startup class with overriding two custom methods (if it matters).
That's why I've used the .ConfigureServices(services => ... approach.
About the IWebHost.StartAsync & IWebHost.StopAsync - it seems that they are working. Thanks.

@Sebazzz
Copy link

Sebazzz commented Oct 19, 2019

This is a solution that worked for me in the mean time. It ensures that everything on WebApplicationFactory, like the Services property keep working as expected:

    protected override void ConfigureWebHost(IWebHostBuilder builder) {
        if (builder == null) throw new ArgumentNullException(nameof(builder));

        IPEndPoint endPoint;
        // Assign available TCP port
        using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)){
            socket.Bind(new IPEndPoint(IPAddress.Loopback, 0));
            socket.Listen(1);
            endPoint = (IPEndPoint)socket.LocalEndPoint;
        }

        return builder
            .ConfigureKestrel(k => k.Listen(new IPEndPoint(IPAddress.Loopback, 0)));
    }

    protected override TestServer CreateServer(IWebHostBuilder builder)
    {
        // See: https://github.com/aspnet/AspNetCore/issues/4892
        this._webHost = builder.Build();

        var testServer = new TestServer(new PassthroughWebHostBuilder(this._webHost));
        var address = testServer.Host.ServerFeatures.Get<IServerAddressesFeature>();
        testServer.BaseAddress = new Uri(address.Addresses.First());

        return testServer;
    }

    private sealed class PassthroughWebHostBuilder : IWebHostBuilder
    {
        private readonly IWebHost _webHost;

        public PassthroughWebHostBuilder(IWebHost webHost)
        {
            this._webHost = webHost;
        }

        public IWebHost Build() => this._webHost;

        public IWebHostBuilder ConfigureAppConfiguration(Action<WebHostBuilderContext, IConfigurationBuilder> configureDelegate){
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(this.ConfigureAppConfiguration)}");
            return this;
        }

        public IWebHostBuilder ConfigureServices(Action<WebHostBuilderContext, IServiceCollection> configureServices) {
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(ConfigureServices)}");
            return this;
        }

        public IWebHostBuilder ConfigureServices(Action<IServiceCollection> configureServices)
        {
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(ConfigureServices)}");
            return this;
        }

        public string GetSetting(string key) => throw new NotImplementedException();

        public IWebHostBuilder UseSetting(string key, string value)
        {
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(this.UseSetting)}({key}, {value})");
            return this;
        }
    }

@JeroMiya
Copy link

Just a note: with .net core 3.0 apps using IHostBuilder, @danroth27's workaround no longer works.

Now I just have a test script that runs the server, then runs the tests, waits for them to finish, then kills the server. Have to start up the server manually when running in the Visual Studio test runner. Not a great experience.

@Sebazzz
Copy link

Sebazzz commented Oct 25, 2019 via email

@JeroMiya
Copy link

I'm not using IWebHostBuilder so CreateServer is never called.

@Sebazzz
Copy link

Sebazzz commented Oct 25, 2019 via email

@JeroMiya
Copy link

There's an override you can define create a custom IHostBuilder and another to create a custom IHost with a builder passed in, and that's likely where the solution would be. However, it didn't work when I tried it. I don't know what part of the internals of WebApplicationFactory adds the in-memory restrictions but it might to be outside of those two overloads. I've already spent too much time on it, and running the app directly without WebApplicationFactory seems to work fine for now.

@ffMathy
Copy link
Contributor

ffMathy commented Mar 1, 2020

What is the status on this? @davidfowl did this ever make it to 2.2? In that case, how do we use it?

@Tratcher
Copy link
Member

Tratcher commented Mar 1, 2020

@ffMathy not much progress has been made on this scenario, it's still in the backlog.

@ffMathy
Copy link
Contributor

ffMathy commented Mar 1, 2020

Alright. Is there an ETA? Rough estimate?

@Tratcher
Copy link
Member

Tratcher commented Mar 1, 2020

@ffMathy this is an uncommitted feature, there's no ETA until we decide to move it from the backlog to a milestone.

@lukos
Copy link

lukos commented Mar 5, 2020

@Sebazzz Your code above does not compile.

  • You are returning ConfigureKestrel, even though the method is void.
  • You are assigning a variable _webHost in CreateServer that is not in the listing. Is this just a class variable or is it supposed to be coming from somewhere else?
  • You setup an endpoint in ConfigureWebHost and then you don't use it, you just create another one in the call to ConfigureKestrel.

@hhyyrylainen
Copy link

@ThumbGen
I'm making a ASP.net hosted Blazor WASM app, and the code provided here helped me get this kind of testing working. You can see my code here:
https://github.com/Revolutionary-Games/ThriveDevCenter/blob/f282b6a8d766bcc869c80366fc1320791c637627/AutomatedUITests/Fixtures/WebHostServerFixture.cs

@CraigComeOnThisNameCantBeTaken

Sorry Im just dumping code but this is what I am using:

internal class TestApplicationFactory<TStartup> : WebApplicationFactory<TStartup>
        where TStartup : class
    {

        private readonly Action<IServiceCollection> _configureServicesAction;
        private IWebHost _externalHost;

        private string _externalUri;
        public string RootUri
        {
            get
            {
                return _externalUri ?? "http://localhost";
            }
        }

        public override IServiceProvider Services => _externalHost?.Services ?? base.Services;

        public TestApplicationFactory(Action<IServiceCollection> serviceConfiguration,
            bool createExternalTestServer)
        {
            _configureServicesAction = serviceConfiguration ?? NullActionHelper.Null<IServiceCollection>();

            if (createExternalTestServer)
                CreateServer(CreateWebHostBuilder());
        }

        #region EndToEndTesting
        // see https://github.com/dotnet/aspnetcore/issues/4892
        //https://github.com/dotnet/aspnetcore/issues/4892#issuecomment-391901173
        protected override TestServer CreateServer(IWebHostBuilder builder)
        {
            ClientOptions.BaseAddress = new Uri("https://localhost");

            _externalHost = builder.Build();
            _externalHost.Start();
            _externalUri = _externalHost.ServerFeatures.Get<IServerAddressesFeature>().Addresses.LastOrDefault();

            // not used but needed in the CreateServer method logic
            var externalServerHostBuilder = WebHost.CreateDefaultBuilder(Array.Empty<string>());
            SetupWebHostBuilder(externalServerHostBuilder);
            var testServer = new TestServer(externalServerHostBuilder);

            return testServer;
        }

        protected override IWebHostBuilder CreateWebHostBuilder()
        {
            var builder = WebHost.CreateDefaultBuilder(Array.Empty<string>());
            SetupWebHostBuilder(builder);
            builder.ConfigureTestServices(services =>
            {
                _configureServicesAction(services);
                services.AddAfterTestsCleanup();
            });

            return builder;
        }
        #endregion

        protected override IHost CreateHost(IHostBuilder builder)
        {
            var dir = Directory.GetCurrentDirectory();
            builder.UseContentRoot(dir);
            return base.CreateHost(builder);
        }

        protected override void ConfigureWebHost(IWebHostBuilder builder)
        {
            builder.ConfigureServices(services =>
            {
                _configureServicesAction(services);
                services.AddAfterTestsCleanup();
            });

            base.ConfigureWebHost(builder);
        }

        private void SetupWebHostBuilder(IWebHostBuilder builder)
        {
            builder.UseStartup<TStartup>();
            builder.UseSolutionRelativeContentRoot(typeof(TStartup).Assembly.GetName().Name);
        }

        public HttpClient CreateConfiguredHttpClient()
        {
            var client = this.CreateClient(new WebApplicationFactoryClientOptions
            {
                AllowAutoRedirect = false
            });

            client.DefaultRequestHeaders.Authorization =
                new AuthenticationHeaderValue("Test");

            return client;
        }

        protected override void Dispose(bool disposing)
        {
            base.Dispose(disposing);
            if (disposing)
            {
                _externalHost?.Dispose();
            }
        }
    }

It will not surprise me if this could be cleaned up or if I have misunderstood , but it does work for me using .Net 5 for both integration and end-to-end via the createExternalTestServer boolean. I am able to replace services, and the app's server and testing code are also using the same service provider by assigning the service provider from the application factory to a custom fixture class and only using that to perform setup in my tests.

Hopefully someone who knows more than me can take this a step further.

@mkArtakMSFT mkArtakMSFT added area-web-frameworks and removed area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates labels Nov 11, 2021
@Hoffs
Copy link

Hoffs commented Dec 29, 2021

The workarounds provided in the comments seems to be not compatible with .net6 and minimal api hosting as creating webhost returns null (due to minimal api model) and host builder/resolver/factory that is used by WebApplicationFactory is not exposed. So as I can see there is no way to start Api and bind to real port when using .net6 and minimal api?

@martincostello
Copy link
Member

I have a demo app available here that uses WebApplicationFactory<T> to implement UI tests with real browsers pointed at a real HTTP server running on a port, if anyone who's commented on this issue would find it useful: https://github.com/martincostello/dotnet-minimal-api-integration-testing#readme

@hhyyrylainen
Copy link

I was unable to find the "magic" from your example as to how to make things work.
Luckily it turns out that the approach I used before was pretty easy to update to work. I just had to change one line:

-                    var name = Path.ChangeExtension(applicationPath, ".StaticWebAssets.xml");
+                    var name = Path.ChangeExtension(applicationPath, ".staticwebassets.runtime.json");

Here's where's that in my source code if anyone wants to take a look at the entire thing: https://github.com/Revolutionary-Games/ThriveDevCenter/blob/140f920dd28668eaaee1e166dca687777aa75018/AutomatedUITests/Fixtures/WebHostServerFixture.cs#L117

Finding this page https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-6.0 it seems that the magic might be now in the WebApplicationFactory class, though I don't think I can switch to using that without porting my project structure to be like how a freshly created 6.0 blazor app sets itself up.

@martincostello
Copy link
Member

If I've understood you correctly, I think the "magic" is a mix of adding a reference to the Microsoft.AspNetCore.Mvc.Testing package to your project file (code). That includes the MSBuild targets that will copy the runtime assets to your test output (code).

The WebApplicationFactory class will also check that it's there if you use it, and throw an exception if it's missing:

private static void EnsureDepsFile()
{
if (typeof(TEntryPoint).Assembly.EntryPoint == null)
{
throw new InvalidOperationException(Resources.FormatInvalidAssemblyEntryPoint(typeof(TEntryPoint).Name));
}
var depsFileName = $"{typeof(TEntryPoint).Assembly.GetName().Name}.deps.json";
var depsFile = new FileInfo(Path.Combine(AppContext.BaseDirectory, depsFileName));
if (!depsFile.Exists)
{
throw new InvalidOperationException(Resources.FormatMissingDepsFile(
depsFile.FullName,
Path.GetFileName(depsFile.FullName)));
}
}

The class also contains a bunch of code for setting the content root and some other things:

private static bool SetContentRootFromSetting(IWebHostBuilder builder)
{
// Attempt to look for TEST_CONTENTROOT_APPNAME in settings. This should result in looking for
// ASPNETCORE_TEST_CONTENTROOT_APPNAME environment variable.
var assemblyName = typeof(TEntryPoint).Assembly.GetName().Name!;
var settingSuffix = assemblyName.ToUpperInvariant().Replace(".", "_");
var settingName = $"TEST_CONTENTROOT_{settingSuffix}";
var settingValue = builder.GetSetting(settingName);
if (settingValue == null)
{
return false;
}
builder.UseContentRoot(settingValue);
return true;
}

@SimonCropp
Copy link
Contributor

perhaps this doc should be updated to use the current versions and recommended approaches. it is currently using v3 nugets, netcoreapp3.1, the legacy Startup approach to hosting, etc

@jarekrzdbk
Copy link

Is there a way to host application in .net 6, that uses a minimal hosting model?
I can use WebApplicationFactory but then app would be hosted in memory.
I am working on adding contract tests using pact-net library, and it requires a "real server", I can use old Startup/Program approach, but I am just curious, is it possible to host an app that uses a minimal hosting model.

@martincostello
Copy link
Member

Yes it is possible, a potential approach is shown here #4892 (comment)

@msftbot
Copy link
Contributor

msftbot bot commented Oct 11, 2022

Thanks for contacting us.

We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

@IEvangelist
Copy link
Member

It's really sad that this is going on 5 years old now, and it's still not a priority. Automated testing of ASP.NET Core apps all up should be easy. The "testing" story for .NET app in general feels a bit like an after thought. With every new version of .NET, we have an opportunity to improve this, and I would certainly vote for it to become a priority.

I hope that we will start shipping first-class NuGet testing packages, that we ourselves use - so that our customers can use the same packages.

@JeroMiya
Copy link

Just an update on my side, I've long since moved to container based E2E testing and this works perfectly with .net core. I just run the API, frontend, and database containers (i.e. sql server for linux) using the .net wrapper over the docker api. Works great and the app runs in an environment that more closely simulates the production environment.

@ffMathy
Copy link
Contributor

ffMathy commented Apr 20, 2023

@JeroMiya that's not the problem though.

The problem is that that way of testing is not really exposed via actual network connections. So if you want (for instance) puppeteer, playwright, selenium etc, it can't reach the server.

Or if you're opening a reverse tunnel with ngrok to test that your webhooks are working properly in a test. That won't work either.

@BennyM
Copy link

BennyM commented Apr 20, 2023

I've been following this topic for quite some time. For the past year or two I've been running selenium tests against an angular app, you need to override some functionality of the WebApplicationFactory as illustrated in the sample code but it is possible.

@mkArtakMSFT mkArtakMSFT added area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates and removed area-web-frameworks labels Apr 20, 2023
@mkArtakMSFT mkArtakMSFT modified the milestones: .NET 8 Planning, Backlog Apr 20, 2023
@msftbot
Copy link
Contributor

msftbot bot commented Apr 20, 2023

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

@giggio
Copy link
Contributor

giggio commented Apr 20, 2023

This is again removed from the backlog. That's sad.
We need a good story for e2e testing, this is something that is easily doable in every other platform. ASPNET Core is almost 10 years old, now.
Please reconsider putting this back on the .NET 8 milestone. It is not so much work and it would help a lot of people.

@giggio
Copy link
Contributor

giggio commented Apr 25, 2023

Ok, I've managed to get the code to work with .NET 6. I'll post most of it here, but I'm thinking about making it a NuGet pkg, to make it available to everyone. I'm running my tests and they are passing, as soon as I get it more refined I'll do that.

I am having to use reflection on only a small location (Microsoft.Extensions.Hosting.HostFactoryResolver.ResolveHostFactory()), and with that I'm using the same infrastructure as the TestServer from Microsoft. It opens ports, and services work. But remember that this is taking a dependency into an internal method, so new .NET versions could break it, so be careful when you upgrade, even minor versions.

I have not tested it with Startup.cs, only with top level statements. This is rough code, it probably does not work in all cases, but as some asked here for help, I'm posting what I have, feel free to use, adapt, etc.

using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;

public class TestServer<TTypeAssemblyEntrypoint> : IAsyncDisposable where TTypeAssemblyEntrypoint : class
{
    private readonly string connectionString;

    public IHost Host { get; set; }

    public TestServer(string connectionString) => this.connectionString = connectionString;

    public async Task StartAsync()
    {
        Host = CreateWebHost(connectionString);
        var lifetime = Host.Services.GetRequiredService<IHostLifetime>();
        await lifetime.WaitForStartAsync(CancellationToken.None);
    }

    private Func<string[], object> ResolveHostFactory(Assembly assembly,
        TimeSpan? waitTimeout = null,
        bool stopApplication = true,
        Action<object> configureHostBuilder = null,
        Action<Exception> entrypointCompleted = null)
    {
        var hostFactoryResolver = typeof(Microsoft.AspNetCore.TestHost.TestServer).Assembly.GetType("Microsoft.Extensions.Hosting.HostFactoryResolver", true);
        var method = hostFactoryResolver.GetMethod("ResolveHostFactory", BindingFlags.Public | BindingFlags.Static)
            ?? throw new InvalidOperationException("Could not find method 'ResolveHostFactory' on type 'HostFactoryResolver'.");
        var hostFactory = (Func<string[], object>)method.Invoke(null, new object[] { assembly, waitTimeout, stopApplication, configureHostBuilder, entrypointCompleted });
        return hostFactory;
    }

    private IHost ResolveHost(Assembly assembly,
        IConfiguration configuration = null,
        Action<IHostBuilder> configureHostBuilder = null)
    {
        configuration ??= new ConfigurationBuilder().Build();
        var factory = ResolveHostFactory(assembly, null, false, (object o) => { configureHostBuilder((IHostBuilder)o); }, null);
        var configurationManager = new ConfigurationManager();
        configurationManager.AddConfiguration(configuration);
        var args = new List<string>();
        foreach (var (key, value) in configurationManager.AsEnumerable())
            args.Add($"--{key}={value}");
        var host = (IHost)factory(args.ToArray());
        return host;
    }

    private IHost CreateWebHost(string connectionString)
    {
        var applicationPath = typeof(TTypeAssemblyEntrypoint).Assembly.Location;
        var staticWebAssetsFile = Path.ChangeExtension(applicationPath, ".staticwebassets.runtime.json");
        if (!File.Exists(staticWebAssetsFile))
            throw new FileNotFoundException($"Static web assets file not found: {staticWebAssetsFile}");
        var webrootPath = GetWebRootPath();
        var wwwrootPath = Path.Combine(webrootPath, "wwwroot");
        if (!Directory.Exists(wwwrootPath))
            throw new DirectoryNotFoundException($"Directory does not exist at '{wwwrootPath}'.");
        var hostConfiguration = new ConfigurationBuilder()
            .AddInMemoryCollection(new Dictionary<string, string>
            {
                //[WebHostDefaults.StaticWebAssetsKey] = staticWebAssetsFile,
                [HostDefaults.ApplicationKey] = typeof(TTypeAssemblyEntrypoint).Assembly.GetName()?.Name ?? string.Empty,
                [HostDefaults.EnvironmentKey] = "Test",
                [WebHostDefaults.ContentRootKey] = webrootPath,
                [WebHostDefaults.WebRootKey] = wwwrootPath,
                [WebHostDefaults.ServerUrlsKey] = AmbienteTestesFuncionais.PublicUrl,
            })
            .Build();

        var host = ResolveHost(typeof(TTypeAssemblyEntrypoint).Assembly,
            hostConfiguration,
            configureHostBuilder: (IHostBuilder hostBuilder) =>
            {
                hostBuilder
                .ConfigureAppConfiguration(configurationBuilder =>
                {
                    var configuration = new ConfigurationBuilder()
                        .AddInMemoryCollection(new Dictionary<string, string>
                        {
                            ["ConnectionStrings:X"] = connectionString,
                        })
                        .Build();
                    configurationBuilder.AddConfiguration(configuration);
                })
                .ConfigureServices(services =>
                {
                    // add mocks
                })
                .ConfigureWebHost(webHostBuilder =>
                {
                    webHostBuilder.UseKestrel();
                });
            });
        return host;
    }

    private static string GetWebRootPath()
    {
        var currentExecutingAssemblyFile = new Uri(Assembly.GetExecutingAssembly().Location).LocalPath;
        var dir = Path.GetDirectoryName(currentExecutingAssemblyFile);
        var testCsproj = $"{Path.GetFileNameWithoutExtension(currentExecutingAssemblyFile)}.csproj";
        var testCsprojFullPath = "";
        while (!File.Exists(testCsprojFullPath))
        {
            testCsprojFullPath = Path.Combine(dir, testCsproj);
            var root = Path.GetPathRoot(testCsprojFullPath);
            if (root == dir)
                throw new Exception($"File {testCsproj} does not exist.");
            dir = Path.GetFullPath(Path.Combine(dir, ".."));
        }
        var testProjectDir = Path.GetDirectoryName(testCsprojFullPath);
        var assetsDoc = JsonDocument.Parse(File.ReadAllText(Path.Combine(testProjectDir, "obj", "project.assets.json")));
        var path = assetsDoc.RootElement.GetProperty("libraries").GetProperty("Web/1.0.0").GetProperty("path").GetString();
        var siteWebCsrojFile = Path.GetFullPath(Path.Combine(testProjectDir, path));
        var diretorioSiteWeb = Path.GetFullPath(Path.GetDirectoryName(siteWebCsrojFile));
        if (!Directory.Exists(diretorioSiteWeb))
            throw new Exception($"Directory does not exist at '{diretorioSiteWeb}'.");
        return diretorioSiteWeb;
    }

    public async ValueTask DisposeAsync()
    {
        if (Host == null)
            return;
        await Host.StopAsync();
        Host.Dispose();
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affected-most This issue impacts most of the customers area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates enhancement This issue represents an ask for new feature or an enhancement to an existing one feature-mvc-testing MVC testing package severity-major This label is used by an internal tool
Development

No branches or pull requests