×
Community Blog The Invalid Unit Tests We Wrote

The Invalid Unit Tests We Wrote

This article summarizes a set of methods and principles to avoid writing invalid unit test cases through daily unit test practice.

By Chen Changyi (Changyi)

_cover

Preface

The purpose of writing unit test cases is not to pursue unit test code coverage but to use unit tests to verify regression code, trying to find BUGs hidden in the code. Therefore, we should write valid unit test cases. This article summarizes a set of methods and principles to avoid writing invalid unit test cases through daily unit test practice.

1. An Introduction to Unit Test

1.1 Concepts

Wikipedia describes it as:

In computer programming, unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use.

1.2 Unit Test Cases

First, let's look at integration and unit testing with a simple case of service code.

1.2.1 Service Code Case

Here, take the paging queryUser of UserService as an example:

@Service
public class UserService {
    /** Define dependent objects. */
    /** User DAO */
    @Autowired
    private UserDAO userDAO;

    /**
     * Query users
     * 
     * @param companyId
     * @param startIndex
     * @param pageSize
     * @return user pageData
     */
    public PageDataVO<UserVO> queryUser(Long companyId, Long startIndex, Integer pageSize) {
        // Query user data.
        // Query user data: Total quantity
        Long totalSize = userDAO.countByCompany(companyId);
        // Query interface data: Data list
        List<UserVO> dataList = null;
        if (NumberHelper.isPositive(totalSize)) {
            dataList = userDAO.queryByCompany(companyId, startIndex, pageSize);
        }

        // Return pageData.
        return new PageDataVO<>(totalSize, dataList);
    }
}

1.2.2 Integration Test Cases

Many people think all test cases that use the JUnit test framework are unit test cases, so they write the following integration test cases:

@Slf4j
@RunWith(PandoraBootRunner.class)
@DelegateTo(SpringJUnit4ClassRunner.class)
@SpringBootTest(classes = {ExampleApplication.class})
public class UserServiceTest {
    /** UserServices */
    @Autowired
    private UserService userService;

    /**
     * Test: Query users
     */
    @Test
    public void testQueryUser() {
        Long companyId = 123L;
        Long startIndex = 90L;
        Integer pageSize = 10;
        PageDataVO<UserVO> pageData = userService.queryUser(companyId, startIndex, pageSize);
        log.info("testQueryUser: pageData={}", JSON.toJSONString(pageData));
    }
}

Integration test cases have the following features:

  1. Dependence on the external environment and data
  2. Need to launch the application and initialize the test object
  3. Inject test objects directly using @Autowired
  4. Sometimes, uncertain return values cannot be verified and can only be checked manually by printing logs.

1.2.3 Unit Test Cases

The unit test cases written in JUnit + Mockito are listed below:

@Slf4j
@RunWith(MockitoJUnitRunner.class)
public class UserServiceTest {
    /** Define static constants */
    /** Resource path */
    private static final String RESOURCE_PATH = "testUserService/";

    /** Mock dependent objects */
    /** User DAO */
    @Mock
    private UserDAO userDAO;

    /** Define test object */
    /** UserServices */
    @InjectMocks
    private UserService userService;

    /**
     * Test: Query users - no data
     */
    @Test
    public void testQueryUserWithoutData() {
        // Mock dependent methods
        // Mock dependent methods: userDAO.countByCompany
        Long companyId = 123L;
        Long startIndex = 90L;
        Integer pageSize = 10;
        Mockito.doReturn(0L).when(userDAO).countByCompany(companyId);

        // Call test methods
        String path = RESOURCE_PATH + "testQueryUserWithoutData/";
        PageDataVO<UserVO> pageData = userService.queryUser(companyId, startIndex, pageSize);
        String text = ResourceHelper.getResourceAsString(getClass(), path + "pageData.json");
        Assert.assertEquals("pageData inconsistency", text, JSON.toJSONString(pageData));

        // Verify dependent methods
        // Verify dependent methods: userDAO.countByCompany
        Mockito.verify(userDAO).countByCompany(companyId);

        // Verify dependent objects.
        Mockito.verifyNoMoreInteractions(userDAO);
    }

    /**
     * Test: Query user - data available
     */
    @Test
    public void testQueryUserWithData() {
        // Mock dependent methods.
        String path = RESOURCE_PATH + "testQueryUserWithData/";
        // Mock dependent methods: userDAO.countByCompany
        Long companyId = 123L;
        Mockito.doReturn(91L).when(userDAO).countByCompany(companyId);
        // Mock dependent methods: userDAO.queryByCompany
        Long startIndex = 90L;
        Integer pageSize = 10;
        String text = ResourceHelper.getResourceAsString(getClass(), path + "dataList.json");
        List<UserVO> dataList = JSON.parseArray(text, UserVO.class);
        Mockito.doReturn(dataList).when(userDAO).queryByCompany(companyId, startIndex, pageSize);

        // Call test methods.
        PageDataVO<UserVO> pageData = userService.queryUser(companyId, startIndex, pageSize);
        text = ResourceHelper.getResourceAsString(getClass(), path + "pageData.json");
        Assert.assertEquals("pageData inconsistency ", text, JSON.toJSONString(pageData));

        // Verify dependent methods.
        // Verify dependent methods: userDAO.countByCompany
        Mockito.verify(userDAO).countByCompany(companyId);
        // Verify dependent methods: userDAO.queryByCompany
        Mockito.verify(userDAO).queryByCompany(companyId, startIndex, pageSize);

        // Verify dependent objects.
        Mockito.verifyNoMoreInteractions(userDAO);
    }
}

Unit test cases have the following features:

  1. Do not depend on the external environment and data
  2. Do not need to launch the application and initialize the object
  3. You need to use @ Mock to initialize dependent objects and use @ InjectMocks to initialize test objects
  4. You need to mock dependent methods by yourself and specify what value or exception a parameter returns.
  5. Since the return value of the test method is determined, you can directly use the Assert method to make the assertion.
  6. You can verify the number of calls and parameter values of dependent methods and whether the method calls of dependent objects are verified.

1.3 Unit Test Principles

Why do integration tests not count as unit tests? We can judge from the principle of the unit test. In the industry, common unit test principles are the AIR principle and the FIRST principle.

1.3.1 AIR Principle

The AIR Principle is listed below:

1.  A-Automatic

Unit tests should be automated and non-interactive. Test cases are usually executed regularly, and the execution process must be automated to make sense. A unit test whose output requires manual review is not good. System.out is not allowed to be used for manual verification in unit tests, and assert must be used to verify.

2.  I-Independent

Unit tests should maintain independence. Unit test cases must not call each other or rely on external resources to ensure that unit tests are stable, reliable, and easy to maintain.

3.  R-Repeatable

Unit tests can be repeated and cannot be affected by the external environment. Unit tests are usually put into continuous integration and executed every time code is committed.

1.3.2 FIRST Principle

The FIRST Principle is listed below:

1.  F-Fast

Unit tests should be run quickly. Among various test methods, unit tests are the fastest, and unit tests of large projects should be run in a few minutes.

2.  I-Independent

Unit tests should be run independently. The unit test cases should have no dependency on each other and external resources.

3.  R-Repeatable

Unit tests should be able to run stably and repeatedly, and the results of each run should be stable and reliable.

4.  S-Self-Validating

Unit tests should be automatically verified by use cases and cannot rely on manual verification.

5.  T-Timely

Unit tests must be written, updated, and maintained timely to ensure that use cases can dynamically guarantee quality as business code changes.

1.3.3 ASCII Principle

Xihua of Alibaba proposed an ASCII Principle:

1.  A-Automatic

Unit tests should be automated and non-interactive.

2.  S-Self-Validating

Assertions must be used in unit tests to verify correctness, while manual verification based on the output cannot be used.

3.  C-Consistent

The parameters and results of the unit tests are determined and consistent.

4.  I-Independent

Unit tests cannot call each other or on the order of execution.

5.  I-Isolated

Unit tests need to be isolated and not dependent on external resources.

1.3.4 Compare Integration Tests and Unit Tests

Based on the unit test principles in the previous section, we can compare the satisfaction of integration tests and unit tests.

1

We can draw the following conclusions after comparing the table:

  1. The integration test does not necessarily satisfy all unit test principles.
  2. Unit tests must meet all unit test principles.

Therefore, according to these unit test principles, it can be seen that the integration test is highly uncertain and cannot completely replace the unit test. In addition, integration tests are always integration tests, even if they are used to replace unit tests (such as testing the DAO method with the H2 memory database).

2. Invalid Unit Test

You must put yourself in their shoes to identify invalid unit tests, thinking about how to write less unit test code while ensuring unit test coverage. Then, you must start with the unit test writing process and see which stages and methods can cut corners.

2.1 Unit Test Coverage

Wikipedia describes it as:

Code coverage is a percentage measure of the degree to which the source code of a program is executed when a particular test suite is run..

Common metrics of unit test coverage include:

1.  Line Coverage

It is used to measure whether each line of execution statement in the code under test has been tested.

2.  Branch Coverage

It is used to measure whether each code branch in the code under test has been tested.

3.  Condition Coverage

It is used to measure whether each subexpression (true and false) in the condition of the code under test has been tested.

4.  Path Coverage

It is used to measure whether each code branch combination in the code under test has been tested.

In addition, there are Method Coverage, Class Coverage, and other unit test coverage metrics.

Here's a simple way to analyze the coverage metrics of each unit test:

public static byte combine(boolean b0, boolean b1) {
    byte b = 0;
    if (b0) {
        b |= 0b01;
    }
    if (b1) {
        b |= 0b10;
    }
    return b;
}

2

Unit test coverage can only represent whether the class, method, execution statement, code branch, conditional subexpression, etc. of the code under test are executed. It does not represent whether the code is executed correctly and returns the correct result. So, it doesn't make any sense to look at unit test coverage without looking at unit test validity.

2.2 Unit Test Writing Process

First, introduce the unit test writing process summarized by the author:

3

2.2.1 Defining Object Phase

The defining object phase consists of defining the objects under test, mocking dependent objects (class members), and injecting dependent objects (class members).

4

2.2.2 Mocking Method Phase

The mocking method phase mainly includes mocking dependent objects (parameters, return values, and exceptions) and mocking dependent methods.

5

2.2.3 Calling Method Phase

The calling method phase mainly includes mocking dependent objects (parameters), calling tested methods, and verifying parameter objects (return values and exceptions).

6

2.2.4 Verifying Method Phase

The verifying method phase includes verifying dependent methods, verifying data objects (parameters), and verifying dependent objects.

7

2.3 Is It Possible to Cut Corners?

For the phases and methods of the unit test writing process, can we cut corners without affecting unit test coverage?

8

2.4 Final Conclusion

From the table, it can be concluded that cutting corners mainly exists in the verification phase.

1.  Calling method phase

  • Verify data objects (return values and exceptions)

2.  Verifying method phase

  • Verify dependent methods
  • Verify data object (parameter)
  • Verify dependent objects

Through some merging and splitting, the follow-up will be explored in the following three parts.

  1. Verify data objects (including attributes, parameters, and return values)
  2. Verify thrown exceptions
  3. Verify dependent methods (including dependent methods and dependent objects)

3. Verify Data Objects

In unit tests, verifying a data object is to verify whether the expected parameter values are passed in, the expected return values are returned, and the expected attribute values are set.

3.1 Source Method of Data Object

In unit tests, the data objects that need to be verified mainly come from the following sources.

3.1.1 Return Value of the Method under Test

The data object comes from the return value of calling the method under test. For example:

PageDataVO<UserVO> pageData = userService.queryUser(companyId, startIndex, pageSize);

3.1.2 Parameter Capture of Dependent Methods

The data object comes from the parameter capture of the verifying dependent methods. For example:

ArgumentCaptor<UserDO> userCreateCaptor = ArgumentCaptor.forClass(UserDO.class);
Mockito.verify(userDAO).create(userCreateCaptor.capture());
UserDO userCreate = userCreateCaptor.getValue();

3.1.3 Attribute Value of the Object under Test

The data object comes from the attribute values of obtaining the objects under test. For example:

userService.loadRoleMap();
Map<Long, String> roleMap = Whitebox.getInternalState(userService, "roleMap");

3.1.4 Attribute Value of Request Parameters

The data object comes from the attribute values of getting request parameters. For example:

OrderContext orderContext = new OrderContext();
orderContext.setOrderId(12345L);
orderService.supplyProducts(orderContext);
List<ProductDO> productList = orderContext.getProductList();

There are other sources of data objects, but I won't go into details.

3.2 Verification Methods of Data Object

When a method under test is called, the return value and exception need to be verified. When a method call is verified, the captured parameter value also needs to be verified.

3.2.1 Verify Null Value of Data Object

JUnit provides the Assert.assertNull and Assert.assertNotNull methods to verify whether the data object is null.

// 1. Verify that the data object is null.
Assert.assertNull( "The user ID must be null", userId);

// 2. Verify that the data object is not null.
Assert.assertNotNull( "The user ID cannot be null", userId);

3.2.2 Verify Boolean Values of Data Object

JUnit provides the Assert.assertTrue and Assert.assertFalse methods to verify that the Boolean value of a data object is true or false.

// 1. Verify that the data object is true.
Assert. assert True(, NumberHelper. "Return value must be true" isPositive (1) );

// 2. Verify that the data object is false.
Assert. assert False(, NumberHelper. "Return value must be false" isPositive (-1) );

3.2.3 Verify References of Data Object

JUnit provides the Assert.assertSame and Assert.assertNotSame methods to verify that data object references are consistent or not.

// 1. Verify consistency of data object.
Assert.assertSame("Users must be consistent", expectedUser, actualUser);

// 2. Verify that the data object is inconsistent.
Assert.assertNotSame("Users cannot be consistent", expectedUser, actualUser);

3.2.4 Verify the Value of the Data Object

JUnit provides Assert.assertEquals, Assert.assertNotEquals, and Assert.assertArrayEquals methods that you can use to verify that data object values are equal or not.

// 1. Verify a simple data object.
Assert.assertNotEquals("User name inconsistency", "admin", userName);
Assert. assert Equals("Account amount inconsistency", 10000.0D, accountAmount, 1E-6D);

// 2. Verify a simple collection object.
Assert. "User ID list inconsistency" ArrayEquals( assert, new Long[] {1L, 2L, 3L}, userIds);
Assert. assert, Arrays. "User ID list Inconsistent" Equals( asList (1L, 2L, 3L), userIdList);

// 3. Verify complex data objects.
Assert. assert, Long. "User ID inconsistency" Equals( valueOf (1L), user.get Id() );
Assert. "User name inconsistency" Equals( assert, "admin", user. getName () );
...

// 4. Verify complex collection objects.
Assert. assert "User list length inconsistency" Equals(, expectedUserList. size (), actualUserList.size () );
UserDO[] expectedUsers = expectedUserList.toArray(new UserDO[0]);
UserDO[] actualUsers = actualUserList.toArray(new UserDO[0]);
for (int i = 0; i < actualUsers.length; i++) { 
     Assert.assertEquals(String.format("User (%s) ID inconsistency", i), expectedUsers[i].getId(), actualUsers[i].getId()); 
     Assert.assertEquals(String.format("User (%s) name inconsistency ", i), expectedUsers[i].getName(), actualUsers[i].getName());
     ...
};

// 5. Verify data object through serialization.
String text = ResourceHelper.getResourceAsString(getClass(), "userList.json");
Assert. assert "User list consistency" Equals(, text, JSON. toJSONString (userList) );;

// 6. Verify the private attribute field of the data object.
Assert. assert "Base package inconsistency" Equals(, "com.alibaba.example", Whitebox. getInternalState (configurer, "basePackage") );

There are other verification methods for data objects, which will not be illustrated here.

3.3 Verify Data Object Issues

Here, a paging query of company users is used as an example to illustrate the problem existing in verifying data objects.

Code cases:

public PageDataVO<UserVO> queryUser(Long companyId, Long startIndex, Integer pageSize) {
    // Query user data.
    // Query user data: Total quantity
    Long totalSize = userDAO.countByCompany(companyId);
    // Query interface data: Data List
    List<UserVO> dataList = null;
    if (NumberHelper.isPositive(totalSize)) {
        List<UserDO> userList = userDAO.queryByCompany(companyId, startIndex, pageSize);
        dataList = userList.stream().map(UserService::convertUser)
            .collect(Collectors.toList());
    }

    // Return pageData.
    return new PageDataVO<>(totalSize, dataList);
}
private static UserVO convertUser(UserDO userDO) {
    UserVO userVO = new UserVO();
    userVO.setId(userDO.getId());
    userVO.setName(userDO.getName());
    userVO.setDesc(userDO.getDesc());
    ...
    return userVO;
}

3.3.1 Do Not Verify Data Objects

Negative case:

Many people are too lazy to perform any verification on data objects.

// Call test methods.
userService.queryUser(companyId, startIndex, pageSize);

Existing problems:

It is impossible to verify whether the data object is correct or not. For example, if the code under test makes the following modifications.

// Return pageData.
return null;

3.3.2 Verify That the Data Object is Not Null

Negative case:

Since there is a problem with not verifying the data object, I will verify that the data object is not null.

// Call test methods.
PageDataVO<UserVO> pageData = userService.queryUser(companyId, startIndex, pageSize);
Assert.assertNotNull( "pageData is not null", pageData);

Existing problems:

It is impossible to verify whether the data object is correct or not. For example, if the code under test makes the following modifications:

// Return pageData.
return new PageDataVO<>();

3.3.3 Verify Some Attributes of Data Objects

Negative case:

Since it is impossible to simply verify that the data object is not null, I will verify some attributes of the data object.

// Call test methods.
PageDataVO<UserVO> pageData = userService.queryUser(companyId, startIndex, pageSize);
Assert.assertEquals( "The total amount of data is not null.", totalSize, pageData.getTotalSize());

Existing problems:

It is impossible to verify whether the data object is correct or not. For example, if the code under test makes the following modifications:

// Return pageData.
return new PageDataVO<>(totalSize, null);

3.3.4 Verify All Attributes of Data Objects

Negative case:

Since it is impossible to verify some attributes of the data object, I will verify all the attributes of the data object.

// Call test methods.
PageDataVO<UserVO> pageData = userService.queryUser(companyId);
Assert.assertEquals( "The total amount of data is not null.", totalSize, pageData.getTotalSize());
Assert.assertEquals( "The data list is not null", dataList, pageData.getDataList());

Existing problems:

The codes look perfect. It verifies the two attribute values totalSize and dataList in PageDataVO. However, if you add startIndex and pageSize to PageDataVO, you cannot verify whether the two new attributes are assigned correctly. Sample code:

// Return pageData.
return new PageDataVO<>(startIndex, pageSize, totalSize, dataList);

Note: This method only applies to data objects whose attribute fields are immutable.

3.3.5 Verify Data Objects Perfectly

Is there a perfect verification scheme for the attribute field additions of data objects? Yes! The answer is to use JSON serialization and then compare the JSON text contents. If attribute fields add data objects, you will inevitably be prompted with an inconsistent JSON string.

Perfect case:

// Call test methods.
PageDataVO<UserVO> pageData = userService.queryUser(companyId, startIndex, pageSize);
text = ResourceHelper.getResourceAsString(getClass(), path + "pageData.json");
Assert.assertEquals( "pageData inconsistency", text, JSON.toJSONString(pageData));

Note: This method only applies to data objects with variable attribute fields.

3.4 Guidelines on Mock Data Objects

Since there is no mock data object section, guidelines on mock data objects are inserted in the verification of data object section.

3.4.1 Except for Triggering Condition Branch, All Attribute Values of the Mock Objects Cannot be Null.

In the previous section, we showed how to verify data objects perfectly. Is this approach perfect? The answer is no.

For example, we assign the attribute values, name, and desc of all UserDO objects in uesrList returned by userDAO.queryByCompany method to be null and then exchange the name and desc assignments of the convertUser method. The unit test cases above cannot be verified.

private static UserVO convertUser(UserDO userDO) {
    UserVO userVO = new UserVO();
    userVO.setId(userDO.getId());
    userVO.setName(userDO.getDesc());
    userVO.setDesc(userDO.getName());
    ...
    return userVO;
}

Therefore, in unit tests, all attribute values of the mock object cannot be null except for the triggering condition branch.

3.4.2 When Adding a Data Class Attribute Field, Mock the Attribute Value of the Data Object

In the preceding example, if the UserDO and UserVO added a new property field "age" (user age) and the following assignment statement was added:

userVO.setAge(userDO.getAge());

If the unit test is still executed with the original data object, we will find that the unit test case execution passes. Since the attribute field age is null, it makes no difference whether the assignment is made or not. So, the new attribute field of the attribute class is that the attribute value of the data object must be mocked.

Note: If you use a JSON string comparison and set the output null field, it is possible to trigger execution failure of unit test cases.

3.5 Guidelines for Verifying Data Objects

3.5.1 All Data Objects Must be Verified

In unit tests, all data objects must be verified:

  1. Return value from the method under test
  2. Parameter capture from dependent methods
  3. Attribute values from the objects under test
  4. Attribute value from the request parameters

Please see the Source Method of Data Object for more information.

3.5.2 Assertions with Explicit Semantics Must be Used

When you use assertions to verify data objects, you must use assertions with explicit semantics.

Positive example:

Assert.assertTrue( "The return value is not true", NumberHelper.isPositive( 1 ));
Assert.assertEquals( "User inconsistency", userId, userService.createUser(userCreateVO));

Negative example:

Assert.assertNotNull( "The user cannot be null", userService.getUser(userId));
Assert.assertNotEquals( "Users cannot be the same", user, userService.getUser(userId));

Beware of cases that attempt to bypass this guideline and try to make ambiguous semantic judgments with explicit semantic assertions.

Assert.assertTrue( "The user cannot be null", Objects.nonNull(userService.getUser(userId)));

3.5.3 Try to Adopt the Overall Verification Method

If it is a model class, fields will be added according to business requirements. Then, for the data object corresponding to this model class, try to use the overall verification method.

Positive example:

UserVO user = userService.getUser(userId);
String text = ResourceHelper.getResourceAsString(getClass(), path + "user.json");
Assert.assertEquals( "User inconsistency", text, JSON.toJSONString(user));

Negative example:

UserVO user = userService.getUser(userId);
Assert.assertEquals( "User ID inconsistency", Long.valueOf( 123L ), user.getId());
Assert.assertEquals( "User names inconsistency", "changyi", user.getName());
...

This data verification method can verify whether the model class deletes the attribute field. However, if a field is added to the model class, it cannot be verified. Therefore, if this verification method is adopted, the test cases need to be sorted out and completed after attribute fields of the model class are added. Otherwise, when you use a unit test case to regress your code, it will tell you there is no problem here.

4. Verify Thrown Exceptions

Exception, as an important feature of Java, is an important manifestation of the robustness of Java. Capturing and verifying a thrown exception is also a type of test case. Therefore, you need to verify thrown exceptions in unit tests.

4.1 Source Method of Thrown Exceptions

4.1.1 Judgment from Attribute Fields

Check whether the attribute field is invalid. Otherwise, an exception is thrown.

private Map<String, MessageHandler> messageHandlerMap = ...;
public void handleMessage(Message message) {
    ...
    // Check whether the processor map is null.
    if (CollectionUtils.isEmpty(messageHandlerMap)) {
        throw new ExampleException("Message processor map cannot be null ");
    }
    ...
}

4.1.2 Judgment from Input Parameters

Check whether the input parameters are valid. Otherwise, an exception is thrown.

public void handleMessage(Message message) {
    ...
    // Check whether the acquisition processor is null.
    MessageHandler messageHandler = messageHandlerMap.get(message.getType());
    if (CollectionUtils.isEmpty(messageHandler)) {
        throw new ExampleException("Message acquisition processor cannot be null ");
    }
    ...
}

Note: The Assert class provided by the Spring framework used here has the same effect as the if-throw statement.

4.1.3 Judgment from Return Values

Check whether the returned value is valid. Otherwise, an exception is thrown.

public void handleMessage(Message message) {
    ...
    // Process messages in processors.
    boolean result = messageHandler.handleMessage(message);
    if (!reuslt) {
        throw new ExampleException("Message processing exceptions ");
    }
    ...
}

4.1.4 Calls from Mock Methods

If you call a mock dependent method, the mock dependent method may throw an exception.

public void handleMessage(Message message) {
    ...
    // Process messages in processors.
    boolean result = messageHandler.handleMessage(message); // Throw an exception directly.
    ...
}

Here, you can catch exceptions, print out logs, or continue to throw exceptions.

4.1.5 Calls from Static Methods

Sometimes, static method calls may throw exceptions.

// IOException may be thrown.
String response = HttpHelper.httpGet(url, parameterMap);

In addition, there are other ways to throw exceptions, which are not described here.

4.2 Verification Method of Thrown Exceptions

There are usually four ways to verify thrown exceptions in unit tests.

4.2.1 Use the try-catch Statement to Verify Thrown Exceptions

The simplest and most direct way to catch exceptions in Java unit test cases is to use the try-catch statement.

@Test
public void testCreateUserWithException() {
    // Mock dependent methods.
    Mockito.doReturn(true).when(userDAO).existName(Mockito.anyString());

    // Call test methods.
    UserCreateVO userCreate = new UserCreateVO();
    try {
        userCreate.setName("changyi");
        userCreate.setDescription("Java Programmer");
        userService.createUser(userCreate);
    } catch (ExampleException e) {
        Assert.assertEquals("Exception encoding inconsistency ", ErrorCode.OBJECT_EXIST, e.getCode());
        Assert.assertEquals("Exception message inconsistency", "The user already exists", e.getMessage());
    }

    // Verify dependent methods.
    Mockito.verify(userDAO).existName(userCreate.getName());
}

4.2.2 Use the @Test Annotation to Verify Thrown Exceptions

The @Test annotation of JUnit provides an expected attribute that specifies an expected exception type for catching and verifying exceptions.

@Test(expected = ExampleException.class)
public void testCreateUserWithException() {
    // Mock dependent methods.
    Mockito.doReturn(true).when(userDAO).existName(Mockito.anyString());

    // Call test methods.
    UserCreateVO userCreate = new UserCreateVO();
    userCreate.setName("changyi");
    userCreate.setDescription("Java Programmer");
    userService.createUser(userCreate);

    // Verify dependent methods (not executed).
    Mockito.verify(userDAO).existName(userCreate.getName());
}

Note: The test case will jump out of the method after being executed to userService.createUser method. As a result, subsequent verification statements cannot be executed. Therefore, this method cannot verify the exception code, message, reason, and others, nor the dependent method and its parameters.

4.2.3 Use the @Rule Annotation to Verify Thrown Exceptions

If you want to verify the exception cause and message, you need to define the ExpectedException object with the @Rule annotation and then declare the exception type, cause, and message to be caught at the beginning of the test method.

@Rule
public ExpectedException exception = ExpectedException.none();
@Test
public void testCreateUserWithException1() {
    // Mock dependent methods.
    Mockito.doReturn(true).when(userDAO).existName(Mockito.anyString());

    // Call test methods.
    UserCreateVO userCreate = new UserCreateVO();
    userCreate.setName("changyi");
    userCreate.setDescription("Java Programmer");
    exception.expect(ExampleException.class);
    exception.expectMessage("The user already exists.");
    userService.createUser(userCreate);

    // Verify dependent methods (not executed).
    Mockito.verify(userDAO).existName(userCreate.getName());
}

Note: The test case will jump out of the method after being executed to userService.createUser method. As a result, subsequent verification statements cannot be executed. Therefore, this method cannot verify dependent methods and their parameters. Since the ExpectedException verification method only supports verifying the exception type, reason, and message, you cannot verify the custom attribute field values for exceptions. Currently, JUnit officially recommends replacing it with Assert.assertThrows.

4.2.4 Use the Assert.assertThrows Method to Verify Thrown Exceptions

A more concise exception verification method is provided in the latest version of JUnit: the Assert.assertThrows method.

@Test
public void testCreateUserWithException() {
    // Mock dependent methods.
    Mockito.doReturn(true).when(userDAO).existName(Mockito.anyString());

    // Call test methods.
    UserCreateVO userCreate = new UserCreateVO();
    userCreate.setName("changyi");
    userCreate.setDescription("Java Programmer");
    ExampleException exception = Assert.assertThrows("Exception type inconsistency ", ExampleException.class, () -> userService.createUser(userCreate));
    Assert.assertEquals("Exception encoding inconsistency ", ErrorCode.OBJECT_EXIST, exception.getCode());
    Assert.assertEquals("Exception message inconsistency ", " The user already exists ", exception.getMessage());

    // Verify dependent methods.
    Mockito.verify(userDAO).existName(userCreate.getName());
}

4.2.5 Comparison of the Four Verification Methods of Throwing Exceptions

Compare the four verification methods based on the different verification exception features. The results are listed below:

9

In summary, it is best to use the Assert.assertThrows method to verify thrown exceptions, which is officially recommended by JUnit.

4.3 Verify Thrown Exception Problems

Here, take the thrown exception when creating a user as an example to illustrate the problems when verifying thrown exceptions.

Code cases:

private UserDAO userDAO;
public void createUser(@Valid UserCreateVO userCreateVO) {
    try {
        UserDO userCreateDO = new UserDO();
        userCreateDO.setName(userCreateVO.getName());
        userCreateDO.setDesc(userCreateVO.getDesc());
        userDAO.create(userCreateDO);
    } catch (RuntimeException e) {
        log.error("User creation exception: userName={}", userName, e)
        throw new ExampleException(ErrorCode.DATABASE_ERROR, " User creation exception ", e);
    }
}

4.3.1 Do Not Verify Types of Thrown Exceptions

Negative case:

When verifying thrown exceptions, many people use the expected attribute of @ Test annotation and specify the value as Exception.class. The main reasons are listed below:

  1. The code of the unit test case is concise, with only one line @ Test annotation.
  2. No matter what exception is thrown, the unit test case is guaranteed to pass.
@Test(expected = Exception.class)
public void testCreateUserWithException() {
    // Mock dependent methods.
    Throwable e = new RuntimeException();
    Mockito.doThrow(e).when(userDAO).create(Mockito.any(UserCreateVO.class));

    // Call test methods.
    UserCreateVO userCreateVO = ...;
    userService.createUser(userCreate);
}

Existing problems:

The use case specifies a generic exception type and does not verify the type of thrown exceptions. Therefore, if the ExampleException exception is changed to the RuntimeException exception, the unit test case cannot be verified.

throw new RuntimeException("User creation exception", e);

4.3.2 Do Not Verify Attributes of Thrown Exceptions

Negative case:

Since you need to verify the exception type, simply specify the expected attribute of the @ Test annotation as ExampleException.class.

@Test(expected = ExampleException.class)
public void testCreateUserWithException() {
    // Mock dependent methods.
    Throwable e = new RuntimeException();
    Mockito.doThrow(e).when(userDAO).create(Mockito.any(UserCreateVO.class));

    // Call test methods.
    UserCreateVO userCreateVO = ...;
    userService.createUser(userCreate);
}

Existing problems:

The preceding use case only verifies the exception types but does not verify the attribute fields that throw exceptions (exception message, exception reason, and error code). Therefore, if the error code DATABASE_ERROR is changed to PARAMETER_ERROR, the unit test case cannot be verified.

throw newExampleException(ErrorCode.PARAMETER_ERROR, " User creation exception", e);

4.3.3 Only Verify Part of Attributes of Thrown Exceptions

Negative case:

If you want to verify exception attributes, you must use the Assert.assertThrows method to catch the exception and verify the common attributes of the exception. However, some people are lazy and only verify part of the attributes of thrown exceptions.

// Mock dependent methods.
Throwable e = new RuntimeException();
Mockito.doThrow(e).when(userDAO).create(Mockito.any(UserCreateVO.class));

// Call test methods.
UserCreateVO userCreateVO = ...;
ExampleException, "Exception type inconsistency" ExampleException exception=Assert.assertThrows. class, () -> userService. createUser (userCreateVO ));
Assert.assertEquals( "Exception encoding inconsistency", ErrorCode.DATABASE_ERROR, exception.getCode());

Existing problems:

The preceding use case only verifies the exception type and error code. If the error message User creation exception is changed to User creation error, the unit test case cannot be verified.

throw new ExampleException(ErrorCode.DATABASE_ERROR, "User creation error", e);

4.3.4 Do Not Verify the Cause of Thrown Exceptions

Negative case:

It looks perfect to catch the thrown exception first and then verify the exception encoding and exception message.

// Mock dependent methods.
Throwable e = new RuntimeException();
Mockito.doThrow(e).when(userDAO).create(Mockito.any(UserCreateVO.class));

// Call test methods.
UserCreateVO userCreateVO = ...;
ExampleException, "Exception type inconsistency" ExampleException exception=Assert.assertThrows. class, () -> userService. createUser (userCreateVO ));
Assert.assertEquals( "Exception encoding inconsistency", ErrorCode.OBJECT_EXIST, exception.getCode());
Assert.assertEquals( "Exception message inconsistency", "User creation exception", exception.getMessage());

Existing problems:

It can be seen from the code that when an ExampleException exception is thrown, the last parameter e is the RuntimeException exception thrown by our mock userService.createUser method. However, we do not verify the cause of throwing the exception. If you modify the code and remove the last parameter e, the unit test above case cannot be verified.

throw new ExampleException(ErrorCode.DATABASE_ERROR, "User creation exception");

4.3.5 Do Not Verify Related Method Calls

Negative case:

Many people think that verifying a thrown exception is enough, and verifying a dependent method call is unnecessary.

// Mock dependent methods.
Throwable e = new RuntimeException();
Mockito.doThrow(e).when(userDAO).create(Mockito.any(UserCreateVO.class));

// Call test methods.
UserCreateVO userCreateVO = ...;
ExampleException, "Exception type inconsistency" ExampleException exception=Assert.assertThrows. class, () -> userService. createUser (userCreateVO ));
Assert.assertEquals( "Exception encoding inconsistency", ErrorCode.OBJECT_EXIST, exception.getCode());
Assert.assertEquals( "Exception message inconsistency", "User creation exception", exception.getMessage());
Assert.assertEquals( "Exception cause inconsistency", e, exception.getCause());

Existing problems:

How can you prove the code has gone through this branch without verifying the relevant method calls? For example, before we create a user, we check that the username is invalid and throw an exception.

// Check that the username is valid.
String userName = userCreateVO.getName();
if (StringUtils.length(userName) < USER_NAME_LENGTH) {
    throw new ExampleException(ErrorCode.INVALID_USERNAME, " Invalid user name ");
}

4.3.6 Perfectly Verify Thrown Exceptions

A perfect exception validation should not only verify the exception type, exception properties, and exception causes, but also verify the dependent method calls before throwing the exception.

The perfect case:

// Mock dependent methods.
Throwable e = new RuntimeException();
Mockito.doThrow(e).when(userDAO).create(Mockito.any(UserCreateVO.class));

// Call test methods.
String text = ResourceHelper.getResourceAsString(getClass(), path + "userCreateVO.json");
UserCreateVO userCreateVO = JSON.parseObject(text, UserCreateVO.class);
ExampleException, "Exception type inconsistency" ExampleException exception=Assert.assertThrows. class, () -> userService. createUser (userCreateVO ));
Assert.assertEquals( "Exception encoding inconsistency", ErrorCode.OBJECT_EXIST, exception.getCode());
Assert.assertEquals( "Exception message inconsistency", "User creation exception", exception.getMessage());
Assert.assertEquals( "Exception cause inconsistency", e, exception.getCause());

// Verify dependent methods.
ArgumentCaptor<UserDO> userCreateCaptor = ArgumentCaptor.forClass(UserDO.class);
Mockito.verify(userDAO).create(userCreateCaptor.capture());
text = ResourceHelper.getResourceAsString(getClass(), path + "userCreateDO.json");
Assert.assertEquals( "User creation inconsistency", text, JSON.toJSONString(userCreateCaptor.getValue()));

4.4 Verify Guidelines for Throwing Exceptions

4.4.1 All Thrown Exceptions Must be Verified

You must verify all thrown exceptions in unit tests:

  1. Judgment from attribute fields
  2. Judgment from input parameters
  3. Judgment from the return value
  4. Calls from mock methods
  5. Calls from static methods

Please see the Source Method of Thrown Exception for more information.

4.4.2 Must Verify the Exception Type, Exception Attribute, and Exception Cause

When verifying a thrown exception, you must verify the exception type, exception attribute, exception cause, etc.

Positive example:

ExampleException, "Exception type inconsistency" ExampleException exception=Assert.assertThrows. class, () -> userService. createUser (userCreateVO ));
Assert.assertEquals( "Exception encoding inconsistency", ErrorCode.OBJECT_EXIST, exception.getCode());
Assert.assertEquals( "Exception message inconsistency", "The user already exists", exception.getMessage());
Assert.assertEquals( "Exception cause inconsistency", e, exception.getCause());

Negative example:

@Test(expected = ExampleException.class)
public void testCreateUserWithException() {
    ...
    userService.createUser(userCreateVO);
}

4.4.3 After Verifying a Thrown Exception, Must Verify the Related Method Calls

After verifying the thrown exception, the relevant method call must be verified to ensure the unit test case takes the desired branch.

Positive example:

// Call test methods.
...

// Verify dependent methods.
ArgumentCaptor<UserDO> userCreateCaptor = ArgumentCaptor.forClass(UserDO.class);
Mockito.verify(userDAO).create(userCreateCaptor.capture());
text = ResourceHelper.getResourceAsString(getClass(), path + "userCreateDO.json");
Assert.assertEquals( "User creation inconsistency", text, JSON.toJSONString(userCreateCaptor.getValue()));

5. Verify Method Calls

In unit tests, verification method calls verify the number and order of calls of dependent methods and whether the expected parameter values are passed in.

5.1 Source Method of Method Calls

5.1.1 Method Calls from Injected Objects

The most common method call is injecting a dependent object:

private UserDAO userDAO;
public UserVO getUser(Long userId) {
    UserDO user = userDAO.get(userId); // Method call
    return convertUser(user);
}

5.1.2 Method Calls from Input Parameters

Sometimes, you can pass in the dependent object through input parameters and then call the dependent object.

public <T> List<T> executeQuery(String sql, DataParser<T> dataParser) {
    List<T> dataList = new ArrayList<>();
    List<Record> recordList = SQLTask.getResult(sql);
    for (Record record : recordList) {
        T data = dataParser.parse(record); // Method call
        if (Objects.nonNull(data)) {
            dataList.add(data);
        }
    }
    return dataList;
}

5.1.3 Method Calls from Return Values

private UserHsfService userHsfService;
public User getUser(Long userId) {
    Result<User> result = userHsfService.getUser(userId);
    if (!result.isSuccess()) { // Method call 1
        throw new ExampleException("User acquisition exception ");
    }
    return result.getData(); // Method call 2
}

5.1.4 Calls from Static Methods

In Java, a static method is a member method that is modified by static and can be called without an object instance. In daily code, a static method calls the account for a certain proportion.

String text=JSON.toJSONString (user); // Method call

5.2 Verification Method of Method Call

In unit tests, verifying a dependent method call is the process of confirming whether the dependent method of the mock object is called as expected.

5.2.1 Verify the Call Parameters of the Dependent Method

// 1. Verify that the dependent method calls without parameters.
Mockito.verify(userDAO).deleteAll();

// 2. Verify dependent method call of specified parameters .
Mockito.verify(userDAO).delete(userId);

// 3. Verify dependent method calls of any parameters.
Mockito.verify(userDAO).delete(Mockito.anyLong());

// 4. Verify dependent method calls of nullable parameters.
Mockito.verify(userDAO).queryCompany(Mockito.anyLong(), Mockito.nullable(Long.class));

// 5. Verify dependent method calls of null parameters.
Mockito.verify(userDAO).queryCompany(Mockito.anyLong(), Mockito.isNull());

// 6. Verify dependent method calls of variable parameters.
Mockito.verify(userService).delete(1L, 2L, 3L);
Mockito.verify(userService).delete(Mockito.any(Long. class )) ;) // Single matching
Mockito.verify(userService).delete(Mockito.<Long>any()); // Multiple matching

5.2.2 Verify the Number of Dependent Method Calls

// 1. Verify that the dependent method is called once by default.
Mockito.verify(userDAO).delete(userId);

// 2. Verify that the dependent method is never called.
Mockito.verify(userDAO, Mockito.never()).delete(userId);

// 3. Verify that the dependent method is called n times.
Mockito.verify(userDAO, Mockito.times(n)).delete(userId);

// 4. Verify that the dependent method is called at least once.
Mockito.verify(userDAO, Mockito.atLeastOnce()).delete(userId);

// 5. Verify that the dependent method is called at least n times.
Mockito.verify(userDAO, Mockito.atLeast(n)).delete(userId);

// 6. Verify that the dependent method can be called up to once.
Mockito.verify(userDAO, Mockito.atMostOnce()).delete(userId);

// 7. Verify that the dependent method is called up to n times.
Mockito.verify(userDAO, Mockito.atMost(n)).delete(userId);

// 8. Verify that the dependent method is called the specified n times.
Mockito.verify(userDAO, Mockito.call(n)).delete(userId); // Not marked as verified.

// 9. Verify that the dependent object and its methods are called only once.
Mockito.verify(userDAO, Mockito.only()).delete(userId);

5.2.3 Verify Dependent Methods and Capture Parameter Values

// 1. Use the ArgumentCaptor.forClass method to define an ArgumentCaptor.
ArgumentCaptor<UserDO> userCaptor = ArgumentCaptor.forClass(UserDO.class);
Mockito.verify(userDAO).modify(userCaptor.capture());
UserDO user = userCaptor.getValue();

// 2. Use the @Captor annotation to define an  ArgumentCaptor.
@Captor
private ArgumentCaptor<UserDO> userCaptor;

// 3. Capture a list of parameter values for multiple method calls.
ArgumentCaptor<UserDO> userCaptor = ArgumentCaptor.forClass(UserDO.class);
Mockito.verify(userDAO, Mockito.atLeastOnce()).modify(userCaptor.capture());
List<UserDO> userList = userCaptor.getAllValues();

5.2.4 Verify Other Types of Dependent Method Calls

// 1. Verify the final method call.
The verification of the final method is similar to that of the normal method. 

// 2. Verify private method call.
PowerMockito.verifyPrivate(mockClass, times(1)).invoke("unload", any(List.class));

// 3. Verify construction method call.
PowerMockito.verifyNew(MockClass.class).withNoArguments();
PowerMockito.verifyNew(MockClass.class).withArguments(someArgs);

// 4. Verify static method call.
PowerMockito.verifyStatic(StringUtils.class);
StringUtils.isEmpty(string);

5.2.5 Verify That the Dependent Object Has No More Method Calls

// 1. Verify that the mock object does not have any method calls.
Mockito.verifyNoInteractions(idGenerator, userDAO);

// 2. Verify that the mock object has no more method calls.
Mockito.verifyNoMoreInteractions(idGenerator, userDAO);

5.3 Issues of Verifying Dependent Methods

Here, take cacheUser as an example to illustrate the problems in verifying dependent methods.

Code case:

private UserCache userCache;
public boolean cacheUser(List<User> userList) {
    boolean result = true;
    for (User user : userList) {
        result = result && userCache.set(user.getId(), user);
    }
    return result;
}

5.3.1 Do Not Verify Dependent Method Calls

Negative case:

Some people think that since the dependent method has been mocked and the method under test has returned the value as expected, it is unnecessary to verify the dependent method.

// Mock dependent methods.
Mockito.doReturn(true).when(userCache).set(Mockito.anyLong(), Mockito.any(User.class));

// Call test methods.
List<User> userList = ...;
Assert.assertTrue( "The processing result is not true", userService.cacheUser(userList));

// Do not verify the dependent method.

Existing problems:

A dependent method is mocked, and the method under test has returned a value as expected, which does not mean that the dependent method was called or called correctly.

For example, the unit test case cannot be verified by setting userList to a null list before the for the loop.

// Clear user list.
userList = Collections.emptyList();

5.3.2 Do Not Verify the Number of Dependent Method Calls

Negative case:

Some people like to use Mockito.verify to verify at least once and any combination of parameters because it can be applied to any verification of dependent method calls.

// Verify dependent methods.
Mockito.verify(userCache, Mockito.atLeastOnce()).set(Mockito.anyLong(), Mockito.any(User.class));

Existing problems:

Although this method is suitable for the verification of any dependent method call, it has no real effect.

For example, if we accidentally wrote the cache statement twice, this unit test case cannot be verified.

// The cache is written twice.
result = result && userCache.set(user.getId(), user);
result = result && userCache.set(user.getId(), user);

5.3.3 Do Not Verify Parameters of Dependent Method Calls

Negative case:

Since it is said that there is a problem with verification at least once, I will specify the number of verification times.

// Verify dependent methods.
Mockito.verify(userCache, Mockito.times(userList.size())).set(Mockito.anyLong(), Mockito.any(User.class));

Existing problems:

Although the problem of the number of verification methods has been solved, the problem of verification method parameters still exists.

For example, if we accidentally wrote each user of the loop cache as the first user of the loop cache, the unit test case cannot be verified.

User user = userList.get(0);
for (int i = 0; i < userList.size(); i++) {
    result = result && userCache.set(user.getId(), user);
}

5.3.4 Do Not Verify All Dependent Method Calls

Negative case:

You cannot use the verification method of any parameter, so you have to use the verification method of the actual parameter. However, there is too much code to verify all dependent method calls, so verifying one or two dependent method calls is enough.

Mockito.verify(userCache).set(user1.getId(), user1);
Mockito.verify(userCache).set(user2.getId(), user2);

Existing problems:

If only one or two method calls are verified, you can only guarantee that these one or two method calls are not problematic.

For example, we accidentally perform a user cache after the for the loop.

// Cache the last user.
User user = userList.get(userList.size() - 1);
userCache.set(user.getId(), user);

5.3.5 Verify All Dependent Method Calls

Negative case:

Since there is a problem with not verifying all method calls, I will verify all method calls.

for (User user : userList) {
    Mockito.verify(userCache).set(user.getId(), user);
}

Existing problems:

It seems there should be no problem if all method calls have been verified. However, if there are other method calls in the cache user method, this is still problematic.

For example, if clearing all user caches is added before we enter the cache user method, this unit test cannot be verified.

// Delete all user caches.
userCache.clearAll();

5.3.6 Verify Dependent Method Calls Perfectly

Verifying all method calls can only guarantee there is no problem with the current logic. If new method calls are involved, this unit test case cannot be verified. Therefore, we need to verify that all dependent objects have no more method calls.

The perfect case:

// Verify dependent methods.
ArgumentCaptor<Long> userIdCaptor = ArgumentCaptor.forClass(Long.class);
ArgumentCaptor<User> userCaptor = ArgumentCaptor.forClass(User.class);
Mockito.verify(userCache, Mockito.atLeastOnce()).set(userIdCaptor.capture(), userCaptor.capture());
Assert.assertEquals( "User ID list inconsistency", userIdList, userIdCaptor.getAllValues());
Assert.assertEquals( "User information list inconsistency", userList, userCaptor.getAllValues());

// Verify dependent objects.
Mockito.verifyNoMoreInteractions(userCache);

Note: You can verify the parameters and the number and order of calls using ArgumentCaptor.

5.4 Verify Guidelines for Method Calls

5.4.1 All Mock Method Calls Must be Verified

In unit tests, all mock methods involved are verified:

  1. Method calls from injected objects
  2. Method calls from input parameters
  3. Method calls from return values
  4. Calls from static methods

Please see the Source Method of Method Call for more information.

5.4.2 Must Verify That All Mock Objects Have No More Method Calls

In unit tests, it is necessary to verify that all mock objects have no more method calls to prevent the existence or addition of other method calls to the method under test.

Positive example:

// Verify dependent objects.
Mockito.verifyNoMoreInteractions(userDAO, userCache);

Remarks:

The author likes to verify all mock objects in the @ After method, so it is unnecessary to verify mock objects in each unit test case.

@After
public void afterTest() {
    Mockito.verifyNoMoreInteractions(userDAO, userCache);
}

Unfortunately, Mockito.verifyNoMoreInteractions does not support the ability to verify all mock objects without parameters, otherwise, this code would be more concise.

5.4.3 Must Use Parameter Values or Matchers with Explicit Semantics

When verifying dependent methods, you must use parameter values or matchers with explicit semantics. You cannot use matchers with ambiguous semantics (such as any series parameter matchers).

Positive example:

Mockito.verify(userDAO).get(userId);
Mockito.verify(userDAO).query(Mockito.eq(companyId), Mockito.isNull());

Negative example:

Mockito.verify(userDAO).get(Mockito.anyLong());
Mockito.verify(userDAO).query(Mockito.anyLong(), Mockito.isNotNull());
0 1 0
Share on

Alibaba Cloud Community

917 posts | 203 followers

You may also like

Comments