Category: test

Setup ngTest step on Bitrise.io

Bitrise’s buit-in karma jasmine testRunner seems out-of-date, every time gives error:

‘@angular-devkit/build-angular/plugins/karma’ karma plugin is meant to be used from within Angular CLI and will not work correctly outside of it.

It’s not hard to create a script step, things to do:

1. Download chrome.

brew update && brew cask install chromedriver google-chrome

Quite often, we got checksum error, possible cause, network cache, solution: directly run script from github repo, ideally from your forked repo to avoid cache.

brew cask install https://raw.githubusercontent.com/maodd/homebrew-cask/master/Casks/google-chrome.rb

2. Run ngTest, non-watch mode, so it can close itself after done.

./node_modules/@angular/cli/bin/ng test –watch=false –code-coverage

3. (Optional) Collect code coverage report.

The challenge here is Bitrise can deploy a folder as artifact, we need to find a way display report in single html file, also to keep the original multiple test files as many as possible.

We use a html-inline npm package to compact test report files, also zip the whole test result into a single file, and artifact both files.

if [ -d "$BITRISE_SOURCE_DIR/coverage" ]; then
   zip -r $BITRISE_DEPLOY_DIR/coverage.zip $BITRISE_SOURCE_DIR/coverage
   npm i -g html-inline
   html-inline -i $BITRISE_SOURCE_DIR/coverage/index.html -o coverage.html
   cp coverage.html $BITRISE_DEPLOY_DIR
fi

The artifacts for each build looks like this:

Screen Shot 2020-04-03 at 9.20.20 AM.png

Subfolder links inside the coverage report won’t work, but it’s good enough for us, until we can find a better html-package tool.

How to unit-test class has dependency on HttpResponse

When getting report from reporting service through  asmx web service, the proxy class needs to set some custom header to HttpResponse, work can be done, but unit-test it is not easy. Google result indicates that we can take advantage of HttpResponseBase and HttpResponseWrapper from System.Web and System.Web.Abstraction.

The class diagram looks like this:

Instead of directly depending on HttpContext.Current.Response, we need to change the dependency to HttpResponseBase class, from which both HttpResponseWrapper (in production) and HttpResponseFake (in unit-test) are derived.

Code example:

    public class ReportingServiceProxy
    {
        private readonly HttpResponseBase _response;
        public ReportingServiceProxy()
            : this(new HttpResponseWrapper(HttpContext.Current.Response))
        {}
        public ReportingServiceProxy(HttpResponseBase httpResponse)
        {
            _response = httpResponse;
            //...
        }
        public FileContentResult GetReport(string reportName, ReportFormat reportFormat, IList parameters)
        {
            // ...
            _response.AddHeader("cache-control", "must-revalidate");
        }
    }

    public class HttpResponseFake : HttpResponseBase
    {
        public override void AddHeader(string name, string value)
        {
            Console.WriteLine("Header {0} added with value: {1}", name, value);
        }
    }

        [Test]
        public void should_get_report_from_test_env()
        {
            var reportingServiceProxy = new ReportingServiceProxy(new HttpResponseFake());
            ...
        }

Using log4net and Ninject auto-binding in FitNesse

FitNesse executes dotnet assembly through (Syterra) runner (for fitSharp), this brought up some path related problems. One problem we faced is log4net configuration, putting log4net.config in assembly path doesn’t work.

From fit content page we can see:


!define COMMAND_PATTERN {%m -a C:\Source\project\TestHelper\App.config -r fitnesse.fitserver.FitServer,dotnet4\fit.dll %p}
!define TEST_RUNNER {dotnet4\Runner.exe}
!path C:\Source\project\TestSuite\bin\Release\TestSuite.dll

!contents
'''Prerequisites:'''

'''Test Case:'''

The only config file runner can take is the one from test helper, putting log4net in runner’s path might work, but it will introduce maintenance headache. The cleaner way to configure log4net might be to do it through code, idea from this post.

 public static class SimpleLogger
    {
        static SimpleLogger()
        {
            ConfigureLog();
        }
        public static ILog For(object objectToLog)
        {
            return For(objectToLog.GetType());
        }
        public static ILog For(Type typeToLog)
        {
            return LogManager.GetLogger(typeToLog);
        }

        private static void ConfigureLog()
        {
            Logger root;
            root = ((Hierarchy)LogManager.GetRepository()).Root;

            root.AddAppender(GetFileAppender());
            root.Repository.Configured = true;
        }

        private static FileAppender GetFileAppender()
        {
            FileAppender lAppender = new FileAppender();
            lAppender.Name = "File";
            lAppender.AppendToFile = true;
            lAppender.File = @"c:\tmp\log.txt";
            lAppender.Layout = new
            log4net.Layout.PatternLayout("%date{dd-MM-yyyy HH:mm:ss,fff} %5level [%2thread] %message (%logger{1}:%line)%n");
            lAppender.Threshold = log4net.Core.Level.All;
            lAppender.ActivateOptions();

            return lAppender;
        }
    }

Another problem we got is Ninject convention extension, aka auto-binding. This code works in application but not in fitnesse runner:

 Kernel.Scan(x =>
              {
//                x.FromAssemblyContaining<SampleClass>();  // This works in fitnesse runner, but will be messy if sample classes are too many.
//                string path = Path.GetDirectoryName((new System.Uri(Assembly.GetExecutingAssembly().CodeBase)).AbsolutePath);
//               x.FromAssembliesInPath(path);
//               x.FromAssembliesInPath(".");
                  x.FromAssembliesMatching("*");  // Doesn't work in fitnesse runner
//                  x.AutoLoadModules();  // Don't do it if you have multiple modules defined.
                x.BindWithDefaultConventions();
                x.InTransientScope();
            });

The way Ninject doing auto-binding is to create a temporary appDomain, then try to load it into this temp domain.

 // private static IEnumerable FindAssemblies(IEnumerable filenames,
 // Predicate filter)
 {
 AppDomain temporaryDomain = CreateTemporaryAppDomain();

foreach (string file in filenames)
 {
 Assembly assembly = null;

if (File.Exists(file))
 {
 try
 {
 var name = new AssemblyName { CodeBase = file };
 assembly = temporaryDomain.Load(name);
 // assembly = Assembly.ReflectionOnlyLoadFrom(file); // this works in fitnesse
 }
 catch (.....
 

The construct of AssemblyName by setting file string to CodeBase actually failed in fitnesse environment, this can be proven by checking the AssemblyName.FullName. In this case, it’s empty.
Google search found a different way to construct Assembly simply by calling Assembly.ReflectionOnlyLoadFrom(file);
Our new IoC setup code looks then like this:

        static void Scan()
        {
              Kernel.Scan(x =>
              {
                x.From(get_all_assembly_instances());
                x.BindWithDefaultConventions();
                x.InTransientScope();
            });
        }

        static IEnumerable<Assembly> get_all_assembly_instances()
        {
            string path = Path.GetDirectoryName(Path.GetFullPath(new System.Uri(Assembly.GetExecutingAssembly().CodeBase)).AbsolutePath));
            IEnumerable<string> assemblies = Directory.GetFiles(path).Where(IsAssemblyFile);

            IEnumerable<AssemblyName> assemblyNames = FindAssemblies(assemblies, y=>true);
            return assemblyNames.Select(name => Assembly.Load(name));
        }
// IsAssemblyFile, FindAssemblies can be copied from https://github.com/ninject/ninject.extensions.conventions/blob/master/src/Ninject.Extensions.Conventions/AssemblyScanner.cs
// While FindAssemblies needs to tweet to use  Assembly.ReflectionOnlyLoadFrom(file); to get Assembly instance.
        private static IEnumerable<AssemblyName> FindAssemblies(IEnumerable<string> filenames,
                                                                 Predicate<Assembly> filter)
        {
            return
                filenames.Select(Assembly.ReflectionOnlyLoadFrom).Where(assembly => filter(assembly)).
                    Select(assembly => assembly.GetName());
        }

TDD javascript in NetBeans

The book Test-Driven JavaScript Development is great, http://tddjs.com/
. It indeed changed my thoughts on javascript,  it is such a powerful dynamic language, very close to ruby, e.g. similar features like everything is object, including functions. While some other feature like hoist variable declaration, this variable scope and closure are very weird.

JsTestDriver is very successful xUnit like test environment. Its community is very active, during only 3 days while I was playing it,  its version bumped from 1.2.2. to 1.3.0!

It’s very annoying that there is no GUI Runner for jsRunner , but picking NetBeans as my javascript IDE seems not a bad choice. I really like the ALT+Shit+F to reformat feature.

Here is some trick to embed jsTestDriver in NetBeans.

  1. Create Ruby project to hold all those javascript source code.
  2. Create Rake file at project root, content look like this:
    
    directory "test/output"
    
    task :run_js_test => "test/output" do
      ff = "C:\\Program Files\\Mozilla Firefox\\firefox.exe"
      output = "--testOutput test/output"
      sh "java -jar tools/JsTestDriver-1.3.0.jar --port 4224 --tests all #{output} --browser  \"#{ff}\" "
    end
    
    task :default => [ :run_js_test ]
    
    
  3. Run this Rake file, we can see the test results from output window:
    Note: Those MUST be a space before the dash in front of coverage name in conf file, otherwise you will get an error:while scanning a block mapping
    expected , but found org.jvyaml.tokens.BlockMappingStartToken
    org.jvyaml.ParserException: ParserException while scanning a block mapping we had this expected , but found org.jvyaml.tokens.BlockMappingStartToken

Room to improve for NetBeans IDE:

  • View codecoverage data file in IDE (Windows ENV)
  • GUI view of those test results
  • Better embed of those test-cases.

All those features already exist in NetBeans Ruby IDE, hope in the future it can also apply for javascript.

Automated Silverlight WCF asynchronous test

Using Silverlight NUnit + statlight really makes silverlight developing much safer, theoretically, at least for those really UNIT tests.

With the concern of integration tests, thing becomes a little bit difficult. At first I thought I could just write WCF test like this:


        [Test]
        public void Should_call_find_posts_by_keywords()
        {
            var client = new MyServiceClient();
            client.FindPostsByKeywordCompleted += (sender, args) => {};
        }

The first problem I encountered is, I couldn’t get the client config in Silverlight UnitTest project. Tried adding linked file and adding service reference, no good. My ugly workaround is to hardcode the endpoint address for now:


        [Test]
        public void Should_call_find_posts_by_keywords()
        {
            var client = new MyServiceClient(new System.ServiceModel.BasicHttpBinding(),
                                        new EndpointAddress(
                                            "http://localhost/WcfServices/PostService.svc"));
            client.FindPostsByKeywordCompleted += (sender, args) => {};
        }

OK, First problem solved. Now, another one, statlight couldn’t report the exception if service got something wrong. While in Visualstudio the exception alert can always pop-up.

Something needs be done for asynchronous test. After read the posts from Jeff Wilcox and Jonas Folleso, I ended up with my test looks like this:


 [TestFixture]
    public class ProxyTests : SilverlightTest
    {
        [Test]
        [Asynchronous]
        public void Should_call_find_posts_by_keywords()
        {
            var client = new VineOnlineServiceClient(new System.ServiceModel.BasicHttpBinding(),
                                        new EndpointAddress(
                                            "http://localhost2/WcfServices/PostService.svc"));

            bool isSearchDone = false;
            IEnumerable<PostedItemDto> searchResults = null;

            client.FindPostsByKeywordCompleted += (
                (sender, args) =>
                    {
                        searchResults = args.Result;
                        isSearchDone = true;
                    });

            EnqueueCallback(() => client.FindPostsByKeywordAsync("post"));
            EnqueueConditional( () => isSearchDone );

            EnqueueCallback( () => Assert.IsNotNull(searchResults));  // Optional if we don't care the return data

            EnqueueTestComplete(); // Cannot move to teardown coz asynchronous
        }

InMemoryDatabaseTest using FluentNHibernate

Ayende posted this wonderful idea before, Billy McCafferty also mentioned this to me during our NH training.

I created the FN syntax InMemoryDatabaseTestFixture based on Ayende’s code.

public class InMemoryDatabaseTestFixture : IDisposable
    {
        private static Configuration Configuration;
        private static ISessionFactory SessionFactory;
        protected ISession session;

        public InMemoryDatabaseTestFixture(Assembly assemblyContainingMapping)
        {
            if (Configuration == null)
            {

                FluentConfiguration config = Fluently.Configure()
                    .Database(FluentNHibernate.Cfg.Db.SQLiteConfiguration.Standard.InMemory()
                    )
                    .Mappings(m =>
                                  {
                                      m.FluentMappings.AddFromAssembly(assemblyContainingMapping);
                                      m.AutoMappings.Add(new AutoPersistenceModelGenerator(assemblyContainingMapping).Generate());
                                  }

                    );

                Configuration = config.BuildConfiguration();
                SessionFactory = config.BuildSessionFactory();
            }

            session = SessionFactory.OpenSession();

            new SchemaExport(Configuration).Execute(true, true, false, session.Connection, Console.Out);
        }

        public void Dispose()
        {
            session.Dispose();
        }
    }

Unit test in Silverlight

SilverLight NUnit project and NUnit SilverLight project are two different projects.

The first one can only be used inside IDE, launched by testRunner.

NUnit Silverlight is intend to replace the default MSTest with NUint syntax in default Silverlight unit-test project ship with Silverlight tool-kit. By default, SilverLight UNnit Test project from Toolkit comes with MSTest, it can be replaced by NUnit, by referencing NUnit SilverLight project.

To make it continuous-build friendly, I  had to create a local lib folder right under silverlight unit-test project then create those linked reference, otherwise build server kept complaining missing reference error.

RhinoMock has a SilverLight edtion in case you need to mock something.

This walk through explained how to use MsBuild to implement a command line runner, note, must navigate to test project path to launch MSbuild command, not the web one.

Unfortunately, I got the same broswer-not-closing problem. I haven’t got time to try this workaround yet.

A bettor solution is to use StatLight. Need to find out the xsl file for CC.net. A simple xsl file to cc.net can be found at here.

SilverUnit is only used by TypeMock, a commercial product.

Using memory appender to test logger

I found a bug in IglooCoder.Commons, the Logger doesn’t log the correct class. The fix is easy, but I want to create a test before fixing it, in case the next release the problem came back.

How to test the logger without logging at the log files generated? This is the perfect situation we should use memory appender.

  [TestFixture]
    public class LogTests
    {
        private MemoryAppender _appender;
        private LoggingEvent _event;

        [TestFixtureSetUp]
        public void Init()
        {
            Log.Initialize(new AglcLog4NetLogger(SystemInfo.ApplicationBaseDirectory + "\\Log4Net.config"));
            _appender = new MemoryAppender();

            _appender.Name = "Unit Testing Appender";

            _appender.Layout =
                new PatternLayout("%date{dd-MM-yyyy HH:mm:ss,fff} %5level [%2thread] %message (%logger{1}:%line)%n");

            _appender.Threshold = Level.All;

            _appender.ActivateOptions();

            Logger root = ((Hierarchy) LogManager.GetRepository()).Root;

            root.AddAppender(_appender);

            root.Repository.Configured = true;
        }

        [Test]
        public void should_get_the_correct_class_name_to_log()
        {
            _appender.Clear();
            Log.For(this).Info("testing");

            _event = _appender.GetEvents()[0];

            var data = _event.GetLoggingEventData();

            Assert.AreEqual(this.GetType().ToString(), data.LoggerName);
        }
    }

White.NUnit as UI Testing tool

Got suggestion from JP, looking at White.NUnit as our UI testing tool, I’m surprised that it didn’t take too long to figure it out:

First difference this is different than Phantom UI test is, we have to write test cases in VS which we love to. Here is the sample code to test the calculator.

using NUnit.Framework;
using White.Core;
using White.Core.Factory;
using White.Core.UIItems;
using White.Core.UIItems.Finders;
using White.Core.UIItems.WindowItems;

namespace solution
{
    [TestFixture]
    public class CalcTests
    {
        private Application application;

        [SetUp]
        public void Setup()
        {
            application = Application.Launch("calc.exe");
        }

        [TearDown]
        public void TearDown()
        {
            application.Kill();
        }

        [Test]
        public void should_launch_calculator()
        {
             Assert.IsNotNull(application);
        }

        [Test]
        public void should_do_a_simple_add_operation()
        {

            Window window = application.GetWindow("Calculator", InitializeOption.NoCache);

            window.Get<Button>("1").Click();
            var result = window.Get<Label>(SearchCriteria.ByText("1"));
            window.Get<Button>("Add").Click();
            window.Get<Button>("3").Click();
            window.Get<Button>("Equals").Click();

            Assert.AreEqual("4", result.Text);

        }
    }
}

Hint:

  1. use White.recorder generate class function to get the actual control name.
  2. use SearchCriteria to find the control.

I found that White doesn’t know Datawindow control. For PB developer,they have to turn to IBM rational Robot or HP QuickTestPro, and learn another test script language other than C#.

Stub or Expect, AssertWasCalled or VerifyAllExceptions

Regarding the following unit-test principles, I think Stub + AsserWasCalled is better.

  • Each test should only test one thing,
  • All setup code should be in one place,
  • Test behavior not method.

using System.Collections.Generic;
using NUnit.Framework;
using Rhino.Mocks;

namespace Demo
{
    public interface IDependencyA {IEnumerable<string> GetList();}

    public interface IDependencyB {string DoSth(IEnumerable<string> input);}

    public interface IDependencyC {string DoSthElse(string input);}

    public class SystemToTest
    {
        private readonly IDependencyA _a;
        private readonly IDependencyB _b;
        private readonly IDependencyC _c;

        public SystemToTest(IDependencyA a, IDependencyB b, IDependencyC c)
        {
            _a = a;
            _b = b;
            _c = c;
        }

        public string Action()
        {
            return _c.DoSthElse(_b.DoSth(_a.GetList()));
        }
    }

    [TestFixture]
    public class MockDemo
    {
        protected IDependencyA _mockDependencyA;
        protected IDependencyB _mockDependencyB;
        protected IDependencyC _mockDependencyC;

        private SystemToTest _systemUnderTest;

        [SetUp]
        public virtual void Setup()
        {
            _mockDependencyA = MockRepository.GenerateMock<IDependencyA>();
            _mockDependencyB = MockRepository.GenerateMock<IDependencyB>();
            _mockDependencyC = MockRepository.GenerateMock<IDependencyC>();

            _systemUnderTest = new SystemToTest(_mockDependencyA,
                                                _mockDependencyB,
                                                _mockDependencyC);

            // If B.DoSth can NOT run with null input, this stub is needed.
            _mockDependencyA.Stub(x => x.GetList()).Return(
                new List<string>()
                );
        }

        [TestFixture]
        public class When_Action_happens : MockDemo
        {
            [SetUp]
            public override void Setup()
            {
                base.Setup();
                _systemUnderTest.Action();
            }

            [Test]
            public void should_ask_a_to_get_list()
            {
                 _mockDependencyA.AssertWasCalled(x => x.GetList());
                // Can also run _mockRequestRepository.VerifyAllExpectations();
                //  if using Expect in Setup instead of Stub
            }

            [Test]
            public void should_ask_b_to_do_sth()
            {

                _mockDependencyB.AssertWasCalled(
                    x => x.DoSth(null),
                    opt => opt.IgnoreArguments());
            }

            [Test]
            public void should_ask_c_to_do_sth_else()
            {
                _mockDependencyC.AssertWasCalled(
                    x => x.DoSthElse(null),
                    opt => opt.IgnoreArguments());
            }
        }
    }
}