Posted by archworx on June 10, 2007
After doing a lot some investigations and trying it for ourselves, we came to the conclusion that its better for us to use Nunit instead of using Mstest (the automated unit tests generated by Visual Studio 2005). Of coarse that’s not a general rule, depends of the project and your environment, but in most of our projects, we prefer Nunit for the following reasons
- It’s an open standard
- Works on all versions of Visual Studio, whereas MsTest only works with Team system or VSTS
- Much easier to integrate with Cruise Control (our continuous integration server)
- More suited for test driven development cause it allows you to start by coding the tests, whereas the main benefit of MsTest is that it can generates test stubs from your code.
Decide based on your project.
Posted in Continuous Integration, Extreme Programming, Nader | Leave a Comment »
Posted by archworx on June 6, 2007
I’ve trying to get MsTest, the unit tests generated by Visual Studio 2005, to work with cruise control for a while now. So finally I was able to get it to work following these steps below
Let’s say we have a solution called ITT in the following path c:/ITT, first we’ll look at the ccnet config file, after you compile using the msbuild task, you have to call an exec task to execute the MsTests, but before that you have to delete the test results file because MsTest won’t run if the file is already there, we are doing this by calling a batch file to delete this test results file and also delete the temp folders created by MsTest. After that you have to merge the results file with build output so I can be displayed in the dashbaord.
<intervalTrigger seconds=”100″ />
<!– here you do the task to get the project from the source control –>
<!–sourcecontrol type=”vss” autoGetSource=”true” applyLabel=”false”>
<!– task to build the solution–>
<!– call a batch file that deletes results.trx –>
<!– this is required as MsTest will not create the file if it exists–>
<!– the batch file also deletes the temp folders created by MsTest –>
<!– the exec task to execute the MsTest task –>
<executable>C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\MSTest.exe</executable>
<buildArgs>/testcontainer:\ITT\AnalysisLibTests\bin\Debug\AnalysisLibTests.dll /runconfig:localtestrun.testrunconfig /resultsfile:results.trx</buildArgs>
<!–to get the test results in the dashboard we have to merge the results XML file –>
<email from=”CruiseControl” mailhost=”mail” includeDetails=”TRUE”>
The next step is to modify the dashboard config file to display the test results, first we update the dashboard.config file to include the MsTests details menu item and then we add the MsTest results summary to the main build page. The required xsl files are already included with ccnet, you can customize the xsl to fit your needs
<xslFile>xsl\MsTestSummary.xsl</xslFile> <!– MsTest results summary in main page–>
<xslFile>xsl\unittests.xsl</xslFile> <!– Nunit test results, can be removed if not required any more –>
<xslReportBuildPlugin description=”NUnit Details” actionName=”NUnitDetailsBuildReport” xslFileName=”xsl\tests.xsl” />
<xslReportBuildPlugin description=”NUnit Timings” actionName=”NUnitTimingsBuildReport” xslFileName=”xsl\timing.xsl” />
<xslReportBuildPlugin description=”MSTest Report” actionName=”MSTESTReport” xslFileName=”xsl\MsTestReport.xsl”/> <!– menu item –>
<xslReportBuildPlugin description=”NAnt Output” actionName=”NAntOutputBuildReport” xslFileName=”xsl\NAnt.xsl” />
<xslReportBuildPlugin description=”NAnt Timings” actionName=”NAntTimingsBuildReport” xslFileName=”xsl\NAntTiming.xsl” />
<xslReportBuildPlugin description=”FxCop Report” actionName=”FxCopBuildReport” xslFileName=”xsl\FxCopReport.xsl” />
<xslReportBuildPlugin description=”NCover Report” actionName=”NCoverBuildReport” xslFileName=”xsl\NCover.xsl” />
<xslReportBuildPlugin description=”Simian Report” actionName=”SimianBuildReport” xslFileName=”xsl\SimianReport.xsl”/>
Posted in Continuous Integration, Nader, VS 2005 | 3 Comments »
Posted by archworx on January 22, 2007
Background: This post is about challenges in differencing binaries to identify if they are identical between assemblies on the testing environment and assemblies generated from code on SourceSafe. Read on for more details.
The Problem: At the end of your development efforts, you typically need to subject your code base to testing, and then upon testing approval, you ship your code to the client. However, bear in mind the following:
- Your Code Base is on a Source Control engine
- Your binaries are not – they are on the development and testing servers. It doesn’t make sense from a configuration management perspective to store binaries on source control (because:
- You can always generate them (theoretically) exactly from the source.
- It is a huge burden to ensure object/source consistency
- Even if you do keep them, you’ll have to check that the source & object are indeed consistent, which is an extra overhead)
- You typically need to ship to your client the binaries that are on your testing server – as those are the ones that have been approved by the testers.
- You can’t ship the source unless you are sure it generates the binaries the developers claim they have produced on the testing environment.
- So the obvious process is to retrieve your source code from your Source repository and recompile it.
Enter .NET Assemblies – which include the following obstacles to successfully being able to recompile the exact binary stream of code twice:
- By default .NET assemblies change their version # every time you compile – this is a good thing, as it provides for very good tracking of version numbers; something that is sadly lacking in many developer’s culture. However, this means that binary differencing will yield false positives.
- If you need your assembly to be hosted on the GAC, or otherwise want to sign your assembly, your assembly must be strongly named, this can pose challenges if you sign them with keys that are not controlled properly.
- The assembly header also contains a field called the “MVID” – which is the Module Version Identifier. This field’s purpose is solely to be unique for each time the module is compiled. This is a rather powerful concept, in the sense that this is the first time I’ve personally seen the concept of someone wanting to distinguish a compilation instance from another one, irrespective of the code being compiled itself.
The Solution: This article is about attempts to solve the three aspects of the problem described above. At this time, we have a simple solution and a workaround for the first two – about the version and the signatures, and we have hopeful indicators that the MVID issue too can be resolved.
- Version Numbers – can be explicitly defined through the removal of the “*” sign for release builds. You can find this field in the assembly info.
- Strongly Named – let’s ignore thise case temporarilly.
- MVID – we believe this can be controlled via a compiler option – but I am yet to find it.
The rest of this post is mostly dedicated to discussing the MVID issue.
Intermediate Language Disassembly:
ildasm /text /all file.dll
The MVID is used by the .NET CLR to determine whether or not to reload the precompiled assembly data.
This is to allow caching such precompiled data, and consequently ensuring cache integrity.
This would imply that the MVID is only useful when precompiled information exists in the assembly.
Typically precompilation only happens when you use NGEN.EXE.
Consequently not generating an MVID or generating it with the same ID is not necessarily a dangerous idea to contemplate.
Emperical Observation has shown that Nant manages to automagically generate the same MVID each time it recompiles, thus dispelling the myth that it must be unique for every ”compilation”. There must be a way to mimic Nant’s communication with the C# Compiler, as it must be using it to do the compilation. There is no way that Nant is faking a compilation. Or is there? 😉
The observations proposed herein are very encouraging, even in so far as they encourage extreme ideas, such as:
1. manually coercing the same uid value for the mvid for otherwise identical compilations (via injecting it into the binary for example); because this would theoretically not jeopardize the sanity of the ngen-generated data.
2. We could do a manual textual comparison of the assembly’s code via ildasm /text and a script that conceals the mvid information
More on this later.
Posted in Configuration Management, Continuous Integration, mkaram, Source Control Tools | 13 Comments »