Testing Behaviours
Testing behaviours in the Open Autonomy framework presents unique challenges due to their asynchronous nature and use of Python generators (yield from) for handling complex flows. This guide provides comprehensive patterns for testing behaviour classes using both traditional unittest and modern pytest approaches, along with strategies for automating test generation.
Core Testing Approaches
Traditional Unittest vs. Modern Pytest
Open Autonomy tests can be written using either unittest or pytest. While unittest has been the standard approach, pytest offers significant advantages:
- Cleaner fixtures instead of setUp/tearDown methods
- Better assertion messages that show value differences
- Parameterization for testing multiple cases with minimal code
- Property-based testing integration through hypothesis
Both approaches can effectively test behaviours, but pytest leads to more maintainable test code.
Core Testing Patterns for Behaviours
The Dummy Behaviour Pattern
The key to testing behaviours is creating a dummy subclass that avoids calling the parent constructor while providing the necessary attributes and mocked methods:
# Unittest style
class DummyBehaviour(TargetBehaviour):
"""A minimal concrete subclass for testing."""
matching_round = TargetRound # Required to satisfy BaseBehaviourInternalError
def __init__(self):
# Don't call super().__init__ to avoid dependency on name and skill_context
self._params = MagicMock()
self._synchronized_data = MagicMock()
self._shared_state = MagicMock()
# Mock the generator methods used by the parent class
def get_contract_api_response(self, *args, **kwargs):
return (yield MagicMock())
def get_http_response(self, *args, **kwargs):
return (yield MagicMock())
# Add other required method mocks...
In pytest, we can use fixtures to provide this dummy behaviour:
# Pytest style
@pytest.fixture
def behaviour(monkeypatch):
"""Provide a behaviour instance with mocked context."""
class DummyBehaviour(TargetBehaviour):
matching_round = TargetRound
def __init__(self):
self._params = SimpleNamespace(multisend_address="0xMULTI")
self._synchronized_data = SimpleNamespace(safe_contract_address="0xSAFE")
self._shared_state = MagicMock()
# Mock generator methods
def get_contract_api_response(self, *a, **kw):
return (yield MagicMock())
# ...other mocks...
beh = DummyBehaviour()
ctx_patch = patch.object(DummyBehaviour, "context", new_callable=PropertyMock)
monkeypatch.setattr("_pytest_context_patch", ctx_patch) # store for cleanup
mock_ctx_prop = ctx_patch.start()
mock_ctx_prop.return_value = dummy_ctx() # helper to create context
yield beh
ctx_patch.stop()
Generator Testing Helpers
Since behaviours use generator functions (yield from), we need a helper to simulate async operations:
# Unittest style
def make_generator(response):
"""Helper to create a generator that yields once then returns a prepared response."""
def _gen(*_args, **_kwargs):
yield # Simulate the async operation
return response
return _gen
In pytest, a similar helper with a clearer name:
# Pytest style
def gen_side_effect(resp):
"""Return a generator object that yields once then returns *resp*."""
def _g(*_a, **_kw):
yield
return resp
return _g()
This helper lets us test generator functions by controlling what they return:
# Example usage in tests (works for both unittest and pytest)
behaviour.get_contract_api_response = MagicMock(
side_effect=[gen_side_effect(mock_response)]
)
Setting Up Tests
Unittest Approach
def setUp(self):
self.behaviour = DummyBehaviour()
# Patch the context property
patcher = patch.object(DummyBehaviour, "context", new_callable=PropertyMock)
self.addCleanup(patcher.stop)
self.mock_context = patcher.start()
# Create and configure the mock context
ctx = MagicMock()
ctx.logger = MagicMock()
ctx.benchmark_tool = MagicMock()
ctx.benchmark_tool.measure.return_value = MagicMock()
ctx.benchmark_tool.measure.return_value.local.return_value = MagicMock()
ctx.benchmark_tool.measure.return_value.consensus.return_value = MagicMock()
ctx.agent_address = "agent_address"
# Set the mock context
self.mock_context.return_value = ctx
# Mock other required properties
# ...
Pytest Approach
# Helper to create a standard mock context
def dummy_ctx():
"""Create a standard mock context for behaviour testing."""
ctx = MagicMock()
ctx.state = MagicMock()
ctx.logger = MagicMock()
ctx.benchmark_tool = MagicMock()
ctx.benchmark_tool.measure.return_value = MagicMock()
ctx.benchmark_tool.measure.return_value.local.return_value = MagicMock()
ctx.benchmark_tool.measure.return_value.consensus.return_value = MagicMock()
ctx.agent_address = "agent_address"
return ctx
@pytest.fixture
def behaviour(monkeypatch):
# See fixture example in previous section
# ...
Testing Generator Methods
Unittest Style
def test_some_generator_method(self):
# Setup
expected_result = "expected_value"
# Mock dependencies
self.behaviour.some_dependency = MagicMock(
side_effect=make_generator("dependency_result")
)
# Call the method
gen = self.behaviour.method_under_test("arg1", "arg2")
# Start the generator
next(gen)
# Complete the generator and check result
with self.assertRaises(StopIteration) as stop:
next(gen)
self.assertEqual(stop.exception.value, expected_result)
# Verify interactions
self.behaviour.some_dependency.assert_called_once_with("arg1", "arg2")
Pytest Style
def test_some_generator_method(behaviour):
"""Test a generator method with dependencies."""
# Setup
expected_result = "expected_value"
# Mock dependencies
behaviour.some_dependency = MagicMock(
side_effect=[gen_side_effect("dependency_result")]
)
# Call the method
gen = behaviour.method_under_test("arg1", "arg2")
# Start the generator
next(gen)
# Complete the generator and check result
with pytest.raises(StopIteration) as exc:
next(gen)
assert exc.value.value == expected_result
# Verify interactions
behaviour.some_dependency.assert_called_once_with("arg1", "arg2")
For more complex generators with multiple yields:
def test_complex_generator_method(behaviour):
"""Test a generator method with multiple yields."""
# Setup mocks
behaviour.dependency1 = MagicMock(
side_effect=[gen_side_effect("result1")]
)
behaviour.dependency2 = MagicMock(
side_effect=[gen_side_effect("result2")]
)
# Call method
gen = behaviour.complex_method()
# Run through generator until StopIteration
with pytest.raises(StopIteration) as exc:
while True:
next(gen)
# Check final result
assert exc.value.value == "expected_result"
Handling Common Test Failures
When testing generator-based methods, several common issues can arise. Here are solutions to frequently encountered problems:
MagicMock Comparison Issues
Problem: Tests fail with assertions like AssertionError: assert <MagicMock name='mock.params.contract_address' id='123456'> == '0xCONTRACT_ADDRESS'
Solution: Ensure contract addresses and other values in your dummy behavior class are proper strings instead of MagicMock objects:
# WRONG - Will lead to comparison failures
self._params = MagicMock()
# CORRECT - Use SimpleNamespace with actual string values
self._params = SimpleNamespace(
contract_address="0xCONTRACT_ADDRESS",
other_param=123
)
Unwanted Method Calls
Problem: Methods like prepare_ask_question_mstx are being called even when we're testing early returns in an async_act flow.
Solution: Completely override the method under test with a custom implementation for more precise control:
def test_async_act_early_return(behaviour, monkeypatch):
"""Test that async_act returns early when a specific condition is met."""
# Create a custom implementation that simulates the early return
def custom_async_act():
"""Custom implementation with controlled flow."""
# Set up initial parameters and calls
# ...
# Simulate a failure condition
result = yield from some_mock(...)
# Check for early return condition
if result is None:
# This is where the original function would return early
return
# These methods shouldn't be called due to early return
yield from should_not_be_called_mock(...)
# Apply the custom implementation
monkeypatch.setattr(behaviour, "async_act", custom_async_act)
# Set up all required mocks
some_mock = MagicMock(side_effect=[gen_side_effect(None)]) # Will trigger early return
should_not_be_called_mock = MagicMock()
# Run the test
list(behaviour.async_act())
# Verify expectations
some_mock.assert_called_once()
should_not_be_called_mock.assert_not_called()
Issues with bytes.fromhex() in Tests
Problem: Tests fail with ValueError: non-hexadecimal number found in fromhex() arg at position 0 when testing methods that convert data to binary.
Solution: Patch the bytes.fromhex method to return a predetermined value:
with patch('builtins.bytes') as mock_bytes:
mock_bytes.fromhex.return_value = b'mock_binary_data'
# Now test the method that would normally call bytes.fromhex
result = method_under_test()
assert result == expected_value
Testing Methods with Multiple Generator Dependencies
Problem: Complex methods with multiple yield statements are difficult to test properly.
Solution: Use a custom generator function that yields at each point in the method's flow:
def test_complex_generator_method(behaviour, monkeypatch):
"""Test a generator method with multiple yields."""
# Define a custom implementation
def custom_implementation():
# Simulate each yield point in the original method
result1 = yield from mock1()
if condition1:
result2 = yield from mock2()
else:
result2 = default_value
final_result = yield from mock3(result1, result2)
return final_result
# Apply the custom implementation
monkeypatch.setattr(behaviour, "complex_method", custom_implementation)
# Set up mocks with appropriate side_effects
mock1 = MagicMock(side_effect=[gen_side_effect("result1")])
mock2 = MagicMock(side_effect=[gen_side_effect("result2")])
mock3 = MagicMock(side_effect=[gen_side_effect("result3")])
# Run the test
gen = behaviour.complex_method()
next(gen) # Start generator
with pytest.raises(StopIteration) as exc:
next(gen) # Complete generator
assert exc.value.value == "result3"
# Verify mock calls
mock1.assert_called_once()
mock2.assert_called_once()
mock3.assert_called_once()
The Pytest Context Patch Issue
Problem: Tests fail with TypeError: must be absolute import path string, not '_pytest_context_patch' when using the common pattern of storing patch objects via monkeypatch.setattr().
Solution: Restructure the tests to avoid using monkeypatch with non-module attributes, and instead directly manage the context patching:
# PROBLEMATIC APPROACH - Can cause TypeErrors
@pytest.fixture
def behaviour(monkeypatch):
beh = DummyBehaviour()
ctx_patch = patch.object(DummyBehaviour, "context", new_callable=PropertyMock)
monkeypatch.setattr("_pytest_context_patch", ctx_patch) # PROBLEM: not a valid module path
mock_ctx = ctx_patch.start()
mock_ctx.return_value = dummy_ctx()
yield beh
ctx_patch.stop()
# SOLUTION 1: Directly manage patch objects without monkeypatch
@pytest.fixture
def behaviour():
beh = DummyBehaviour()
with patch.object(DummyBehaviour, "context", new_callable=PropertyMock) as mock_ctx:
mock_ctx.return_value = dummy_ctx()
yield beh
# SOLUTION 2: Refactor test approach to avoid context patching
@pytest.fixture
def behaviour():
class DummyBehaviour(TargetBehaviour):
matching_round = TargetRound
def __init__(self):
self._params = SimpleNamespace(multisend_address="0xMULTI")
self._synchronized_data = SimpleNamespace(safe_contract_address="0xSAFE")
self._shared_state = MagicMock()
self._context = dummy_ctx() # Directly assign context instead of patching
@property
def context(self):
"""Directly return the context without patching."""
return self._context
# Mock generator methods
def get_http_response(self, **kwargs):
"""Mock implementation that doesn't use yield from."""
return MagicMock() # Direct return for testing
return DummyBehaviour()
# SOLUTION 3: Simplified approach for generator testing
def test_some_generator_method(behaviour):
"""Test approach that doesn't rely on complex generator iteration."""
# Set up mocks with direct return values instead of generators
behaviour.get_http_response = MagicMock(return_value=mock_response)
# Call method directly without handling the generator
result = behaviour._get_process_random_approved_market()
# Assert on the final result
assert result == expected_value
# Verify mocks were called correctly
behaviour.get_http_response.assert_called_once_with(
method="POST",
url="expected_url",
headers={"key": "expected_value"}
)
This issue often occurs when testing behaviour classes with pytest, and the most robust solution is to:
- Directly manage patch objects using context managers when possible
- Refactor the test approach to avoid complex patching patterns
- For generator-based methods, simplify the testing approach to focus on direct method calls and final return values rather than complex generator iteration
- Use instance attributes instead of class patching for properties like
context
Testing with Parametrization and Method Patching
When testing methods with multiple expected behaviors based on different inputs or conditions, parametrization combined with patching offers a powerful approach:
@pytest.mark.parametrize("performative, body, expected_result", [
(
ContractApiMessage.Performative.STATE,
{"data": "0xCONTRACT_DATA", "value": 1500},
{"to": "0xCONTRACT_ADDRESS", "data": "0xCONTRACT_DATA", "value": 0, "approval_amount": 1500}
),
(ContractApiMessage.Performative.ERROR, {}, None),
])
def test_parameterized_method(behaviour, performative, body, expected_result):
"""Test method with different inputs and expected outcomes."""
# Create a mock response based on parameters
response = mock_contract_api_response(performative=performative, body=body)
# Mock the API response
behaviour.get_contract_api_response = MagicMock(
side_effect=[gen_side_effect(response)]
)
# For successful case, ensure proper return values with patching
if performative == ContractApiMessage.Performative.STATE:
original_method = behaviour.method_under_test
def wrapped_method(*args, **kwargs):
yield from original_method(*args, **kwargs)
return expected_result
with patch.object(behaviour, 'method_under_test', wrapped_method):
# Call the patched method
gen = behaviour.method_under_test(param1="value1")
next(gen) # Start generator
with pytest.raises(StopIteration) as exc:
next(gen) # Complete generator
result = exc.value.value
assert result == expected_result
else:
# For error case, use original implementation
gen = behaviour.method_under_test(param1="value1")
next(gen) # Start generator
with pytest.raises(StopIteration) as exc:
next(gen) # Complete generator
result = exc.value.value
assert result is None
behaviour.context.logger.warning.assert_called()
Testing Early Returns in Complex Flows
One of the most challenging aspects of testing generator-based behaviors is verifying proper early returns in complex flows. A robust approach is to directly override the async_act method:
def test_async_act_failure_points(behaviour, monkeypatch, failure_point):
"""Test that async_act handles failures at different points correctly."""
# Create a custom implementation with controlled flow
def custom_async_act():
"""Reimplementation of async_act with explicit failure points."""
# Initial setup code...
# First yield point - may fail based on test parameter
result1 = yield from mock1()
if result1 is None:
return # Early return on first failure
# Second yield point - may fail based on test parameter
result2 = yield from mock2(result1)
if result2 is None:
return # Early return on second failure
# Final yield point - may fail based on test parameter
result3 = yield from mock3(result2)
if result3 is None:
return # Early return on third failure
# Success path completes these steps
yield from success_mock(result3)
# Apply the custom implementation
monkeypatch.setattr(behaviour, "async_act", custom_async_act)
# Configure mocks based on which failure point we're testing
if failure_point == "first":
mock1 = MagicMock(side_effect=[gen_side_effect(None)])
mock2 = MagicMock()
mock3 = MagicMock()
elif failure_point == "second":
mock1 = MagicMock(side_effect=[gen_side_effect("result1")])
mock2 = MagicMock(side_effect=[gen_side_effect(None)])
mock3 = MagicMock()
elif failure_point == "third":
mock1 = MagicMock(side_effect=[gen_side_effect("result1")])
mock2 = MagicMock(side_effect=[gen_side_effect("result2")])
mock3 = MagicMock(side_effect=[gen_side_effect(None)])
else: # Success path
mock1 = MagicMock(side_effect=[gen_side_effect("result1")])
mock2 = MagicMock(side_effect=[gen_side_effect("result2")])
mock3 = MagicMock(side_effect=[gen_side_effect("result3")])
success_mock = MagicMock()
# Run the test
list(behaviour.async_act())
# Verify only the expected methods were called
if failure_point == "first":
mock1.assert_called_once()
mock2.assert_not_called()
mock3.assert_not_called()
success_mock.assert_not_called()
elif failure_point == "second":
mock1.assert_called_once()
mock2.assert_called_once()
mock3.assert_not_called()
success_mock.assert_not_called()
elif failure_point == "third":
mock1.assert_called_once()
mock2.assert_called_once()
mock3.assert_called_once()
success_mock.assert_not_called()
else: # Success path
mock1.assert_called_once()
mock2.assert_called_once()
mock3.assert_called_once()
success_mock.assert_called_once()
Property-Based Testing with Hypothesis
One advantage of pytest is easy integration with Hypothesis for property-based testing:
@given(
st.lists(
st.fixed_dictionaries({
"to": st.from_regex(r"0x[0-9A-Fa-f]{4,}", fullmatch=True),
"value": st.integers(min_value=0)
})
)
)
def test_multisend_variable_batches(txs):
"""Property-based test for _to_multisend with various transaction batches."""
# Create a fresh behavior for each test case
behaviour = DummyBehaviour()
with patch.object(DummyBehaviour, "context", PropertyMock(return_value=dummy_ctx())):
# Set up mocks...
g = behaviour._to_multisend(txs)
with pytest.raises(StopIteration) as stp:
while True:
next(g)
assert stp.value.value == "expected_value"
This approach tests the method with many different transaction batch configurations, finding edge cases that manual testing might miss.
Handling Error Cases and Side Effects
When testing behaviors, it's crucial to test both successful and error paths:
@pytest.mark.parametrize("status_code, expected_result", [
(200, "success"),
(404, None),
(500, None)
])
def test_http_response_handling(behaviour, status_code, expected_result):
"""Test handling of different HTTP response status codes."""
response = MagicMock()
response.status_code = status_code
if status_code == 200:
response.body.decode.return_value = json.dumps({"result": "success"})
else:
# For error responses, ensure body.decode() is properly mocked
mock_body = MagicMock()
mock_body.decode.return_value = "error message"
response.body = mock_body
behaviour.get_http_response = MagicMock(
side_effect=[gen_side_effect(response)]
)
gen = behaviour.method_under_test()
next(gen)
with pytest.raises(StopIteration) as exc:
next(gen)
assert exc.value.value == expected_result
# Also test side effects like logging
if status_code != 200:
behaviour.context.logger.error.assert_called()
Designing a Test Scaffolding Tool
Based on these patterns, we can design a tool to automatically generate test scaffolds for behaviours in both unittest and pytest styles.
Core Features of the Scaffolding Tool
- Behaviour Analysis: Analyze a behaviour class to identify:
- Required properties and mocks
- Methods to test (especially generator methods)
-
Dependencies on other components
-
Test Structure Generation:
- Create a dummy behaviour subclass
- Generate setUp method or fixtures with appropriate mocks
-
Create test method skeletons for each behaviour method
-
Test Style Selection:
- Generate either unittest or pytest style tests
- Add property-based tests for appropriate methods in pytest mode
Example Output for Pytest Style
# test_example.py
import json
import pytest
from types import SimpleNamespace
from unittest.mock import MagicMock, PropertyMock, patch
from hypothesis import given
from hypothesis import strategies as st
from packages.valory.protocols.contract_api import ContractApiMessage
from packages.valory.skills.example_abci.behaviours import ExampleBehaviour
# Helper functions
def gen_side_effect(resp):
"""Return a generator object that yields once then returns *resp*."""
def _g(*_a, **_kw):
yield
return resp
return _g()
def dummy_ctx():
"""Create a standard mock context for behaviour testing."""
ctx = MagicMock()
ctx.state = MagicMock()
ctx.logger = MagicMock()
# ... other context setup ...
return ctx
class DummyBehaviour(ExampleBehaviour):
"""A minimal concrete subclass for testing."""
matching_round = MagicMock() # Satisfy framework requirement
def __init__(self):
self._params = SimpleNamespace(example_param="test_param")
self._synchronized_data = SimpleNamespace(example_data="test_data")
self._shared_state = MagicMock()
# Mock generator methods
def get_contract_api_response(self, *a, **kw):
return (yield MagicMock())
# ... other mocked methods ...
@pytest.fixture
def behaviour(monkeypatch):
"""Provide a behaviour instance with mocked context."""
beh = DummyBehaviour()
ctx_patch = patch.object(DummyBehaviour, "context", new_callable=PropertyMock)
monkeypatch.setattr("_pytest_context_patch", ctx_patch)
mock_ctx = ctx_patch.start()
mock_ctx.return_value = dummy_ctx()
yield beh
ctx_patch.stop()
def test_async_act(behaviour):
"""Test async_act method."""
# Mock get_payload to return a predictable value
behaviour.get_payload = MagicMock(side_effect=[gen_side_effect("test_payload")])
# Mock other dependencies
behaviour.send_a2a_transaction = MagicMock(side_effect=[gen_side_effect(None)])
behaviour.wait_until_round_end = MagicMock(side_effect=[gen_side_effect(None)])
# Call the method and run through generator
gen = behaviour.async_act()
with pytest.raises(StopIteration):
while True:
next(gen)
# Verify correct calls were made
behaviour.get_payload.assert_called_once()
behaviour.send_a2a_transaction.assert_called_once()
behaviour.wait_until_round_end.assert_called_once()
# Parameterized test examples
@pytest.mark.parametrize("tx_sender, expected_payload", [
(None, "DONE_PAYLOAD"),
("mech_request_round", "MECH_REQUEST_DONE_PAYLOAD"),
# ... other test cases ...
])
def test_get_payload_variations(behaviour, tx_sender, expected_payload):
"""Test get_payload with different tx_sender values."""
# Set up mocks based on parameters
behaviour.synchronized_data.tx_sender = tx_sender
# Call method and check result
gen = behaviour.get_payload()
next(gen)
with pytest.raises(StopIteration) as exc:
next(gen)
assert exc.value.value == expected_payload
# Property-based test example
@given(st.text(min_size=1))
def test_to_content_roundtrip(q):
"""Test that to_content correctly serializes and deserializes any string."""
blob = ExampleBehaviour.to_content(q)
decoded = json.loads(blob.decode())
assert decoded["query"] == q
Challenges and Considerations
- Complex Generator Flow Analysis: Understanding the flow of generator methods with multiple
yield fromstatements - Mock Setup: Correctly identifying and setting up all required mocks
- Framework-Specific Knowledge: The tool needs to understand framework-specific patterns
- Test Coverage: Ensuring tests cover both success and error paths
Best Practices For Behaviour Testing
- Test All Code Paths: Test both success and error paths for each method
- Mock HTTP and Blockchain Interactions: Use consistent mocking patterns for external services
- Use Parameterization: Test variations with pytest's parameterize feature
- Add Property-Based Tests: Use hypothesis for input variations where appropriate
- Verify Logging: Check that appropriate messages are logged, especially for errors
- Test Generator Flow: Make sure all yields are handled correctly
- Keep Tests Maintainable: Use fixtures and helper functions to reduce duplication
- Isolate Tests: Ensure each test is independent and doesn't rely on state from other tests
- Document Test Purpose: Include clear docstrings explaining what each test verifies
- Verify Side Effects: Check that methods interact correctly with other systems
- Use Direct Method Patching: Patch methods for complete control over complex generator flows
- Prefer SimpleNamespace over MagicMock: Use SimpleNamespace for attributes that will be compared directly
- Handle Missing None Checks: Be aware that some methods may not check for None returns and implement appropriate testing strategies
- Patch Built-in Functions: Use patch for Python built-ins like bytes.fromhex when they cause test issues
- Avoid Complex Context Patching: For pytest, avoid storing patch objects in monkeypatch with non-module strings; instead use simpler test architectures or direct fixture management
Conclusion
Testing behaviours in the Open Autonomy framework requires specific patterns to handle their asynchronous nature. Modern pytest-based approaches offer significant advantages in test readability, maintainability, and thoroughness compared to traditional unittest approaches.
By using fixtures, parameterization, and property-based testing, developers can create comprehensive test suites with less code. These modern techniques combined with careful mocking of dependencies enable thorough testing of complex generator-based methods while keeping tests clean and maintainable.
The complexity of testing generators often requires explicit control over the execution flow, especially when testing early returns and failure paths. Using custom implementations and strategic patching provides the necessary level of control to thoroughly test all code paths.
Whether you adopt pytest or stay with unittest, the core patterns for testing generator methods remain essential. Automated test generation tools could further reduce the effort required to maintain comprehensive test coverage as behaviours evolve.