Verify vs Storm Petrel: Pros and Cons
Among the best snapshot testing tools for .NET developers, there are many great options. But there are two tools that truly stand out: Verify and Storm Petrel. Both offer unique approaches to managing expected baselines in unit and integration tests, but they differ significantly in methodology and implementation.
Verify focuses on file-based snapshot management, storing serialized baselines, and leveraging specialized extensions and diff tools.
Storm Petrel, on the other hand, introduces a C# code-based approach, using .NET Incremental Generators to update baselines directly within the test code. Understanding their differences can help developers choose the right tool based on their project needs.
What Is Verify .NET?
Verify .NET is a collection of NuGet packages ,snapshot management tools, and extensions designed to simplify snapshot testing in .NET.
It provides extensions for serialization, comparison, and verification of expected baselines for the most popular .NET libraries and frameworks. Verify helps organize snapshot testing by comparing and rewriting baseline snapshots stored in the file system.
What Is Storm Petrel .NET?
Storm Petrel .NET is an Incremental Generator tool that updates expected baselines directly in C# test code (rather than in file snapshots). It supports unit and integration tests for the most popular .NET test frameworks.
File snapshots are a specific use case of baseline management in Storm Petrel, which can be implemented via the additional FileSnapshotInfrastructure NuGet package. This package partially implements Storm Petrel’s abstraction layer.
Technical Comparison: Verify vs. Storm Petrel
Here, we will compare Verify and Storm Petrel to help you see the most important differences in how they handle snapshots, test updates, and supported frameworks.
Test Refactoring Requirements
Developers need to refactor traditional assertion tests into snapshot tests using Verify. Below are examples of how to transition from traditional tests to Verify-style tests.
Traditional Assertion Test Example
[Fact]
public void TraditionalTest()
{
var person = ClassBeingTested.FindPerson();
Assert.Equal(new("ebced679-45d3-4653-8791-3d969c4a986c"), person.Id);
Assert.Equal(Title.Mr, person.Title);
Assert.Equal("John", person.GivenNames);
Assert.Equal("Smith", person.FamilyName);
Assert.Equal("Jill", person.Spouse);
Assert.Equal(2, person.Children.Count);
Assert.Equal("Sam", person.Children[0]);
Assert.Equal("Mary", person.Children[1]);
Assert.Equal("4 Puddle Lane", person.Address.Street);
Assert.Equal("USA", person.Address.Country);
}
Traditional Test Using FluentAssertions
[Fact]
public void TraditionalTestViaAssertionLibrary()
{
var person = ClassBeingTested.FindPerson();
person.Should().BeEquivalentTo(new Person
{
Id = new("ebced679-45d3-4653-8791-3d969c4a986c"),
Title = Title.Mr,
GivenNames = "John",
FamilyName = "Smith",
Spouse = "Jill",
Children = new List<string>
{
"Sam",
"Mary"
},
Address = new Address
{
Country = "USA",
Street = "4 Puddle Lane",
}
});
}
[Fact]
public Task SnapshotTest()
{
var person = ClassBeingTested.FindPerson(); return Verify(person); }
In Verify, the person baseline is serialized and stored in a file to follow Verify’s Initial Verification and Subsequent Verification flows.
Storm Petrel does not require developers to change traditional tests. As explained in the “Snapshot Testing with Scand.StormPetrel.Generator without Serialization” section, traditional tests retain their significant benefits over snapshot tests.
For file snapshot tests, Storm Petrel also follows the traditional approach with one natural requirement:
developers must call methods like
`Scand.StormPetrel.FileSnapshotInfrastructure.SnapshotProvider.ReadAllText()`
or
`SnapshotProvider.ReadAllBytes()`
based on the use cases.
Snapshot Storage & Management
Verify supports tools like ReSharper and Rider test runner Verify plugins. Storm Petrel .NET, however, does not require these tools.
Instead, it leverages .NET/IDE infrastructure to run autogenerated tests and update baselines. After updating the baselines, developers can compare them using any third-party tool to verify if the updates are correct.
Verify offers numerous extensions (e.g., Verify.AspNetCore, Verify.WinForms, Verify.Xaml) to verify objects from specific libraries. Storm Petrel eliminates the need for such extensions by representing baselines as C# code via third-party dumpers or using any serialization/representation of objects for file snapshots.
Supported Test Frameworks
Verify supports about six test frameworks, including xUnit, NUnit, and MSTest. Storm Petrel currently supports only these three most popular frameworks.
Diff & Update Mechanisms
Verify includes a diff engine with support for various tools. Storm Petrel, on the other hand, updates baselines directly, allowing developers to use any diff/merge tool of their choice for comparison.
Pros and Cons of Both Approaches
Below are the most notable pros and cons of each approach (though not an exhaustive list).
Pros of Verify
- Embedded Interactivity: Developers can manage actual/expected snapshot differences interactively while running tests or compare snapshots after execution.
- Specialized Extensions: Offers extensions like Verify.AspNetCore, Verify.WinForms, and Verify.Xaml to represent objects from popular libraries, optimized for snapshot testing out-of-the-box.
- Broader Framework Support: Supports more test frameworks, including those for F#.
Cons of Verify
- No Support for C# Code Baselines: Verify does not support expected baselines stored directly in C# code, which is a primary use case in traditional tests.
- Test Refactoring Required: Significant changes to traditional tests are needed to adopt Verify’s snapshot testing approach.
- Complexity: The large number of interactivity options and specialized extensions may require additional effort to learn and configure. Some object representations may not be sufficient out-of-the-box.
Pros of Storm Petrel
- Traditional Test Approach: Maintains the traditional testing approach without requiring significant changes. See the configuration for specific default/custom naming conventions of actual/expected variables and FAQs for other details.
- Flexibility: Supports both “expected baselines in C# code” and “expected baselines in snapshot files” scenarios.
- Simplicity: No need for specialized extensions or embedded interactivity tools. Any third-party solutions can be used here.
Cons of Storm Petrel
- No Embedded Interactivity: Developers must use third-party diff tools to compare actual and expected baselines after Storm Petrel .NET updates them.
- Limited Framework Support: Currently supports only xUnit, NUnit, and MSTest. No support for less popular frameworks or F#.
Use Cases
In this section, we’ll explore how Verify and Storm Petrel are used in real-world scenarios. Each tool serves specific needs, so we’ll break down when and why you might choose one over the other.
Initial and Subsequent Verification with Verify
Verify is only for projects where you want to manage snapshot testing through external files. You use Verify to both verify and rewrite expected baselines for Verify-style snapshot tests. This is useful when you need to store snapshots outside of your test code and easily compare them as your tests evolve.
Example:
You run a test, and the baseline snapshot doesn’t match the current output. Verify rewrites the snapshot to reflect the new state, making it easy to track changes over time.
Visual Aid:
Initial and Subsequent Verification with Verify.
Rewriting Baselines with Storm Petrel
Storm Petrel focuses on C# code baselines for snapshot tests. It also supports snapshot testing through external files, making it a good choice for developers. It offers two primary use cases for baseline rewriting:
- Traditional Tests: With Storm Petrel, you can automate the process of updating baselines that are stored directly as C# code within your test methods. This works well when you don’t want to rely on external snapshot files, keeping everything in your test code.
Example:
Storm Petrel adjusts the baseline in your C# code whenever the test results change, keeping it consistent with the latest state.
Visual Aid:
- File Snapshot Tests: Storm Petrel also supports automating the update of baselines for file-based snapshot tests. You can use the FileSnapshotInfrastructure NuGet package to manage file snapshots, allowing you to automate baseline updates for traditional snapshot tests as well.
Example:
When a test changes, Storm Petrel can automatically update the corresponding snapshot file based on your defined use case.
Visual Aid:
Verdict: Which Tool Should You Use?
By and large, your choice depends on your project circumstances:
- If you need to work with baselines stored in C# code, Storm Petrel is your best option.
- If you need to store baselines in external files, choose between Verify .NET and Storm Petrel. Refer to the sections above to determine the best option based on your specific requirements and scenario details.
When it comes to snapshot testing in .NET, both tools provide strong solutions. Verify vs Storm Petrel comparison boils down to your project demands and whether you prioritize baseline management through code or external files.
For more detailed information, feel free to contact us and learn more about these powerful .NET testing tools.
