Now that GeneXproTools 5.0 is done and dusted I am pushing forward with the next version of GeneXproServer. One of the outstanding issues was to centralize all the unit and integration tests in a single nant file. This is a reasonably complex process because we have tests written in python, c++ and in c#. The latest ones are MSTest unit tests and, according to MSDN, can be run from the command line as such:
mstest /testmetadata:gxps5.vsmdi or mstest /testcontainer:gxps5_tests.dll
The first one runs the tests using Visual Studio’s configuration whereas the second one runs all the tests in the final binary. Let’s try the first one:
Loading C:\_svn\gxps5\gxps5.vsmdi... Starting execution... No tests to execute.
What do you mean no tests to execute??? Let’s try the second one:
... Passed gxps5_test.Dataset_Tests.Load_Database_DataSet Passed gxps5_test.Dataset_Tests.Load_Excel_DataSet Passed gxps5_test.Dataset_Tests.Load_File_DataSet Failed gxps5_test.Functions_Tests.Funtion_Add Failed gxps5_test.Functions_Tests.Funtion_Remove Failed gxps5_test.Functions_Tests.Funtion_Set Failed gxps5_test.Job_Tests.Consolidated_Run Failed gxps5_test.Job_Tests.Fixed_UseSubFolder_Requires_SubFolder_N ...
Wow, I didn’t know I had that many fails… in fact I wasn’t expecting any. I went back to the IDE and run the tests and they all pass(!). A quick look around and it looks like it is random. Some tests are very simple and fail in the command line, others are complex and pass and vice versa. So off to google I am. And indeed the first hit is promising “Tests pass in IDE but fail with MsTest.exe”. Although it gives good clues it is not very detailed.
The solution is to stick with the first example but there are a couple of configuration steps we need to perform: First you have to go to Visual Studio (2010 in this case) and select the menu Test->Edit Test Settings->Local(local.testsettings). Once there select the Deployment tab and check Enable Deployment. This enables the Add Directory on the right which you must use to select the build destination folder of your project. I believe this is required because these tests use a number of unmanaged dlls and the build folder is not standard. Secondly, you have to create a list of tests. Select the menu Test->Create New Test List… give it a name (All for example) and press OK. This creates an empty list. Now you have to select the All Loaded Tests on the left, select all the loaded tests and drag them to your new list (why is this so fiddly, there must be a better way).
So now the tests run properly using:
I was surprised that it took so long to do this.