FAutomationTestFramework Class Reference

#include <AutomationTest.h>

Public Member Functions
CORE_API FOnTestSectionEvent &	GetOnEnteringTestSection (const FString &Section)
CORE_API void	TriggerOnEnteringTestSection (const FString &Section) const
CORE_API bool	IsAnyOnEnteringTestSectionBound () const
CORE_API FOnTestSectionEvent &	GetOnLeavingTestSection (const FString &Section)
CORE_API void	TriggerOnLeavingTestSection (const FString &Section) const
CORE_API bool	IsAnyOnLeavingTestSectionBound () const
CORE_API FString	GetUserAutomationDirectory () const
CORE_API bool	RegisterAutomationTest (const FString &InTestNameToRegister, FAutomationTestBase *InTestToRegister)
CORE_API bool	UnregisterAutomationTest (const FString &InTestNameToUnregister)
CORE_API bool	RegisterAutomationTestTags (const FString &InTestNameToRegister, const FString &InTestTagsToRegister, bool InImmutable=true)
CORE_API bool	UnregisterAutomationTestTags (const FString &InTestNameToUnregister)
CORE_API bool	RegisterComplexAutomationTestTags (const FAutomationTestBase *InTest, const FString &InBeautifiedTestName, const FString &InTestTagsToRegister)
CORE_API TSet< FString >	GetAllExistingTags ()
CORE_API bool	IsTagImmutable (const FString &InTestName, const FString &InTag) const
CORE_API FString	GetTagsForAutomationTest (const FString &InTestName)
CORE_API TArray< FString >	GetTagsForAutomationTestAsArray (const FString &InTestName)
CORE_API void	EnqueueLatentCommand (TSharedPtr< IAutomationLatentCommand > NewCommand)
CORE_API void	EnqueueNetworkCommand (TSharedPtr< IAutomationNetworkCommand > NewCommand)
CORE_API bool	ContainsTest (const FString &InTestName) const
CORE_API bool	RunSmokeTests ()
CORE_API void	ResetTests ()
CORE_API void	StartTestByName (const FString &InTestToRun, const int32 InRoleIndex, const FString &InFullTestPath=FString())
CORE_API bool	StopTest (FAutomationTestExecutionInfo &OutExecutionInfo)
CORE_API bool	ExecuteLatentCommands ()
CORE_API bool	ExecuteNetworkCommands ()
CORE_API void	DequeueAllCommands ()
bool	IsLatentCommandQueueEmpty () const
CORE_API void	LoadTestModules ()
CORE_API void	GetValidTestNames (TArray< FAutomationTestInfo > &TestInfo) const
CORE_API void	GetTestFullNamesMatchingTagPattern (TArray< FString > &OutTestNames, const FString &TagPattern) const
CORE_API void	GetPossibleRestrictedPaths (const FString &BasePath, const TArray< FString > &RestrictedFolderNames, TArray< FString > &OutRestrictedFolders) const
CORE_API void	LoadTestTagMappings ()
CORE_API void	LoadTestTagMappings (const FString &BasePath, const bool RestrictedPathsOnly)
CORE_API void	SaveTestTagMappings (const TArray< FString > &TestFullNames, const TArray< FString > &TestFilePaths, const FBeforeTagMappingConfigSaved &BeforeConfigSaved, const FAfterTagMappingConfigSaved &AfterConfigSaved) const
CORE_API bool	ShouldTestContent (const FString &Path) const
CORE_API void	SetDeveloperDirectoryIncluded (const bool bInDeveloperDirectoryIncluded)
CORE_API void	SetRequestedTestFilter (const EAutomationTestFlags InRequestedTestFlags)
CORE_API FOnTestScreenshotCaptured &	OnScreenshotCaptured ()
CORE_API FOnTestScreenshotAndTraceCaptured &	OnScreenshotAndTraceCaptured ()
void	SetForceSmokeTests (const bool bInForceSmokeTests)
bool	GetCaptureStack () const
void	SetCaptureStack (bool bCapture)
CORE_API void	AddAnalyticsItemToCurrentTest (const FString &AnalyticsItem)
FAutomationTestBase *	GetCurrentTest () const
FString	GetCurrentTestFullPath () const
CORE_API void	NotifyScreenshotComparisonComplete (const FAutomationScreenshotCompareResults &CompareResults)
CORE_API void	NotifyScreenshotComparisonReport (const FAutomationScreenshotCompareResults &CompareResults)
CORE_API void	NotifyTestDataRetrieved (bool bWasNew, const FString &JsonData)
CORE_API void	NotifyPerformanceDataRetrieved (bool bSuccess, const FString &ErrorMessage)
CORE_API void	NotifyScreenshotTakenAndCompared ()
CORE_API bool	CanRunTestInEnvironment (const FString &InTestToRun, FString *OutReason, bool *OutWarn) const

Static Public Member Functions
static CORE_API FAutomationTestFramework &	Get ()
static FAutomationTestFramework &	GetInstance ()
static CORE_API bool	NeedSkipStackWalk ()
static CORE_API bool	NeedLogBPTestMetadata ()
static CORE_API bool	NeedPerformStereoTestVariants ()
static CORE_API bool	NeedUseLightweightStereoTestVariants ()

Public Attributes
FSimpleMulticastDelegate	PreTestingEvent
FSimpleMulticastDelegate	PostTestingEvent
FOnTestEvent	OnTestStartEvent
FOnTestEvent	OnTestEndEvent
FOnTestScreenshotComparisonComplete	OnScreenshotCompared
FOnTestScreenshotComparisonReport	OnScreenshotComparisonReport
FOnTestDataRetrieved	OnTestDataRetrieved
FOnPerformanceDataRetrieved	OnPerformanceDataRetrieved
FSimpleMulticastDelegate	OnScreenshotTakenAndCompared
FSimpleMulticastDelegate	OnBeforeAllTestsEvent
FSimpleMulticastDelegate	OnAfterAllTestsEvent

Friends
class	FAutomationTestOutputDevice

Detailed Description

Class representing the main framework for running automation tests

Member Function Documentation

AddAnalyticsItemToCurrentTest()

void FAutomationTestFramework::AddAnalyticsItemToCurrentTest

(

const FString &

AnalyticsItem

)

Adds a analytics string to the current test to be parsed later. Must be called only when an automation test is in progress

Parameters

AnalyticsItem

Log item to add to the current test

CanRunTestInEnvironment()

bool FAutomationTestFramework::CanRunTestInEnvironment	(	const FString &	InTestToRun,
		FString *	OutReason,
		bool *	OutWarn
	)		const

Internal helper method designed to check if the given test is able to run in the current environment.

Parameters

InTestToRun	test name
OutReason	the related reason of the skipping
OutWarn	the related warning of the skipping

Returns

true if the test is able to run; false if it is unable to run.

ContainsTest()

bool FAutomationTestFramework::ContainsTest

(

const FString &

InTestName

)

const

Checks if a provided test is contained within the framework.

Parameters

InTestName

Name of the test to check

Returns

true if the provided test is within the framework; false otherwise

DequeueAllCommands()

void FAutomationTestFramework::DequeueAllCommands

(

)

Dequeue all latent and network commands

EnqueueLatentCommand()

void FAutomationTestFramework::EnqueueLatentCommand

(

TSharedPtr< IAutomationLatentCommand >

NewCommand

)

Enqueues a latent command for execution on a subsequent frame

Parameters

NewCommand

- The new command to enqueue for deferred execution

EnqueueNetworkCommand()

void FAutomationTestFramework::EnqueueNetworkCommand

(

TSharedPtr< IAutomationNetworkCommand >

NewCommand

)

Enqueues a network command for execution in accordance with this workers role

Parameters

NewCommand

- The new command to enqueue for network execution

ExecuteLatentCommands()

bool FAutomationTestFramework::ExecuteLatentCommands

(

)

Execute all latent functions that complete during update

Returns

- true if the latent command queue is now empty and the test is complete

ExecuteNetworkCommands()

bool FAutomationTestFramework::ExecuteNetworkCommands

(

)

Execute the next network command if you match the role, otherwise just dequeue

Returns

- true if any network commands were in the queue to give subsequent latent commands a chance to execute next frame

Get()

FAutomationTestFramework & FAutomationTestFramework::Get ( )

static

Return the singleton instance of the framework.

Returns

The singleton instance of the framework.

GetAllExistingTags()

TSet< FString > FAutomationTestFramework::GetAllExistingTags

(

)

GetCaptureStack()

bool FAutomationTestFramework::GetCaptureStack ( ) const

inline

GetCurrentTest()

FAutomationTestBase * FAutomationTestFramework::GetCurrentTest ( ) const

inline

Returns the actively executing test or null if there isn't one

GetCurrentTestFullPath()

FString FAutomationTestFramework::GetCurrentTestFullPath ( ) const

inline

Returns the actively executing test full path

GetInstance()

static FAutomationTestFramework & FAutomationTestFramework::GetInstance ( )

inlinestatic

GetOnEnteringTestSection()

FOnTestSectionEvent & FAutomationTestFramework::GetOnEnteringTestSection

(

const FString &

Section

)

Called entering test section.

GetOnLeavingTestSection()

FOnTestSectionEvent & FAutomationTestFramework::GetOnLeavingTestSection

(

const FString &

Section

)

Called leaving test section.

GetPossibleRestrictedPaths()

void FAutomationTestFramework::GetPossibleRestrictedPaths	(	const FString &	BasePath,
		const TArray< FString > &	RestrictedFolderNames,
		TArray< FString > &	OutRestrictedFolders
	)		const

GetTagsForAutomationTest()

FString FAutomationTestFramework::GetTagsForAutomationTest

(

const FString &

InTestName

)

Fetch the tags associated with a test

Parameters

InTestName

Name of the test to find tags for

Returns

a string of all concatenated tags; empty string if test is not registered

GetTagsForAutomationTestAsArray()

TArray< FString > FAutomationTestFramework::GetTagsForAutomationTestAsArray

(

const FString &

InTestName

)

Fetch the tags associated with a test as an array of strings

Parameters

InTestName

Name of the test to find tags for

Returns

an array of tags; empty if test is not registered

GetTestFullNamesMatchingTagPattern()

void FAutomationTestFramework::GetTestFullNamesMatchingTagPattern	(	TArray< FString > &	OutTestNames,
		const FString &	TagPattern
	)		const

Collects all registered tests with associated tags matching the pattern

Parameters

OutTestNames	Array that will be modified to contain only the set of matching tests
TagPattern	An "Advanced Search Syntax" query to select matching tags from the registered tests

See also

FTextFilterExpressionEvaluator

FAutomationTestBase::GetTestFullName

GetUserAutomationDirectory()

FString FAutomationTestFramework::GetUserAutomationDirectory

(

)

const

Gets a scratch space location outside of the project and saved directories. When an automation test needs to do something like generate project files, or create new projects it should use this directory, rather than pollute other areas of the machine.

GetValidTestNames()

void FAutomationTestFramework::GetValidTestNames

(

TArray< FAutomationTestInfo > &

TestInfo

)

const

Populates the provided array with the names of all tests in the framework that are valid to run for the current application settings.

Parameters

TestInfo

Array to populate with the test information

IsAnyOnEnteringTestSectionBound()

bool FAutomationTestFramework::IsAnyOnEnteringTestSectionBound

(

)

const

IsAnyOnLeavingTestSectionBound()

bool FAutomationTestFramework::IsAnyOnLeavingTestSectionBound

(

)

const

IsLatentCommandQueueEmpty()

bool FAutomationTestFramework::IsLatentCommandQueueEmpty ( ) const

inline

Whether there is no latent command in queue

IsTagImmutable()

bool FAutomationTestFramework::IsTagImmutable	(	const FString &	InTestName,
		const FString &	InTag
	)		const

LoadTestModules()

void FAutomationTestFramework::LoadTestModules

(

)

Load any modules that are not loaded by default and have test classes in them

LoadTestTagMappings() [1/2]

void FAutomationTestFramework::LoadTestTagMappings

(

)

Reads test tag mappings from all Tags.ini config files

LoadTestTagMappings() [2/2]

void FAutomationTestFramework::LoadTestTagMappings	(	const FString &	BasePath,
		const bool	RestrictedPathsOnly
	)

Reads test tag mappings under a certain base path

NeedLogBPTestMetadata()

bool FAutomationTestFramework::NeedLogBPTestMetadata ( )

static

Whether to output blueprint functional test metadata to the log when test is running

NeedPerformStereoTestVariants()

bool FAutomationTestFramework::NeedPerformStereoTestVariants ( )

static

Whether to also run stereo test variants for screenshot functional tests

NeedSkipStackWalk()

bool FAutomationTestFramework::NeedSkipStackWalk ( )

static

Whether to skip stack walk while iterating for listing the tests

NeedUseLightweightStereoTestVariants()

bool FAutomationTestFramework::NeedUseLightweightStereoTestVariants ( )

static

Whether to skip variants when the baseline test fails, and skip saving screenshots for successful variants

NotifyPerformanceDataRetrieved()

void FAutomationTestFramework::NotifyPerformanceDataRetrieved	(	bool	bSuccess,
		const FString &	ErrorMessage
	)

NotifyScreenshotComparisonComplete()

void FAutomationTestFramework::NotifyScreenshotComparisonComplete

(

const FAutomationScreenshotCompareResults &

CompareResults

)

Notify that the screenshot comparison has completed

NotifyScreenshotComparisonReport()

void FAutomationTestFramework::NotifyScreenshotComparisonReport

(

const FAutomationScreenshotCompareResults &

CompareResults

)

Notify the screenshot comparison report to the framework

NotifyScreenshotTakenAndCompared()

void FAutomationTestFramework::NotifyScreenshotTakenAndCompared

(

)

NotifyTestDataRetrieved()

void FAutomationTestFramework::NotifyTestDataRetrieved	(	bool	bWasNew,
		const FString &	JsonData
	)

OnScreenshotAndTraceCaptured()

FOnTestScreenshotAndTraceCaptured & FAutomationTestFramework::OnScreenshotAndTraceCaptured

(

)

Accessor for delegate called when a png screenshot is captured and a frame trace

OnScreenshotCaptured()

FOnTestScreenshotCaptured & FAutomationTestFramework::OnScreenshotCaptured

(

)

Accessor for delegate called when a png screenshot is captured

RegisterAutomationTest()

bool FAutomationTestFramework::RegisterAutomationTest	(	const FString &	InTestNameToRegister,
		FAutomationTestBase *	InTestToRegister
	)

Register a automation test into the framework. The automation test may or may not be necessarily valid for the particular application configuration, but that will be determined when tests are attempted to be run.

Parameters

InTestNameToRegister	Name of the test being registered
InTestToRegister	Actual test to register

Returns

true if the test was successfully registered; false if a test was already registered under the same name as before

RegisterAutomationTestTags()

bool FAutomationTestFramework::RegisterAutomationTestTags	(	const FString &	InTestNameToRegister,
		const FString &	InTestTagsToRegister,
		bool	InImmutable = true
	)

Register tag metadata for a test into the framework.

Parameters

InTestNameToRegister	FullName of the test to associate with tags
InTestTagsToRegister	Concatenated string of tags to register
InImmutable	Whether tags are immutable. If false, merge with existing tags, keeping the immutable

Returns

true if tags for the test successfully registered or overwritten; false if tags were already registered for the test

RegisterComplexAutomationTestTags()

bool FAutomationTestFramework::RegisterComplexAutomationTestTags	(	const FAutomationTestBase *	InTest,
		const FString &	InBeautifiedTestName,
		const FString &	InTestTagsToRegister
	)

Helper to register tags for a individual test enumerated by a Complex test

Parameters

InTest	The Complex test to associate with
InBeautifiedTestName	The name of the individual test
InTestTagsToRegister	Concatenated string of tags to register

Returns

true if test tags were successfully unregistered; false if tags for a test with that name was not found in the framework.

ResetTests()

void FAutomationTestFramework::ResetTests

(

)

Reset status of worker (delete local files, etc)

RunSmokeTests()

bool FAutomationTestFramework::RunSmokeTests

(

)

Attempt to run all fast smoke tests that are valid for the current application configuration.

Returns

true if all smoke tests run were successful, false if any failed

SaveTestTagMappings()

void FAutomationTestFramework::SaveTestTagMappings	(	const TArray< FString > &	TestFullNames,
		const TArray< FString > &	TestFilePaths,
		const FBeforeTagMappingConfigSaved &	BeforeConfigSaved,
		const FAfterTagMappingConfigSaved &	AfterConfigSaved
	)		const

Saves test tag mappings optionally invoking before / after events.

SetCaptureStack()

void FAutomationTestFramework::SetCaptureStack ( bool  bCapture )

inline

Used to disabled stack capture when an error or warning event is triggered. Setting bCapture=true does not guarantees GetCaptureStack()=true because that method also depend on NeedSkipStackWalk().

SetDeveloperDirectoryIncluded()

void FAutomationTestFramework::SetDeveloperDirectoryIncluded

(

const bool

bInDeveloperDirectoryIncluded

)

Sets whether we want to include content in developer directories in automation testing

SetForceSmokeTests()

void FAutomationTestFramework::SetForceSmokeTests ( const bool  bInForceSmokeTests )

inline

Sets forcing smoke tests.

SetRequestedTestFilter()

void FAutomationTestFramework::SetRequestedTestFilter

(

const EAutomationTestFlags

InRequestedTestFlags

)

Sets which set of tests to pull from.

ShouldTestContent()

bool FAutomationTestFramework::ShouldTestContent

(

const FString &

Path

)

const

Whether the testing framework should allow content to be tested or not. Intended to block developer directories.

Parameters

Path	- Full path to the content in question

Returns

- Whether this content should have tests performed on it

StartTestByName()

void FAutomationTestFramework::StartTestByName	(	const FString &	InTestToRun,
		const int32	InRoleIndex,
		const FString &	InFullTestPath = FString()
	)

Attempt to start the specified test.

Parameters

InTestToRun	Name of the test that should be run
InRoleIndex	Identifier for which worker in this group that should execute a command
InFullTestPath	Full test path

StopTest()

bool FAutomationTestFramework::StopTest

(

FAutomationTestExecutionInfo &

OutExecutionInfo

)

Stop the current test and return the results of execution

Returns

true if the test ran successfully, false if it did not (or the test could not be found/was invalid)

TriggerOnEnteringTestSection()

void FAutomationTestFramework::TriggerOnEnteringTestSection

(

const FString &

Section

)

const

TriggerOnLeavingTestSection()

void FAutomationTestFramework::TriggerOnLeavingTestSection

(

const FString &

Section

)

const

UnregisterAutomationTest()

bool FAutomationTestFramework::UnregisterAutomationTest

(

const FString &

InTestNameToUnregister

)

Unregister a automation test with the provided name from the framework.

Returns

true if the test was successfully unregistered; false if a test with that name was not found in the framework.

UnregisterAutomationTestTags()

bool FAutomationTestFramework::UnregisterAutomationTestTags

(

const FString &

InTestNameToUnregister

)

Unregister tags for a automation test from the framework.

Parameters

InTestNameToUnregister

FullName of the test to remove tags from

Returns

true if test tags were successfully unregistered; false if tags for a test with that name was not found in the framework.

Friends And Related Symbol Documentation

FAutomationTestOutputDevice

friend class FAutomationTestOutputDevice

friend

Member Data Documentation

OnAfterAllTestsEvent

FSimpleMulticastDelegate FAutomationTestFramework::OnAfterAllTestsEvent

Called after all chosen tests run have finished.

OnBeforeAllTestsEvent

FSimpleMulticastDelegate FAutomationTestFramework::OnBeforeAllTestsEvent

Called before all chosen tests run.

OnPerformanceDataRetrieved

FOnPerformanceDataRetrieved FAutomationTestFramework::OnPerformanceDataRetrieved

Called when the performance data is retrieved.

OnScreenshotCompared

FOnTestScreenshotComparisonComplete FAutomationTestFramework::OnScreenshotCompared

Called when a screenshot comparison completes.

OnScreenshotComparisonReport

FOnTestScreenshotComparisonReport FAutomationTestFramework::OnScreenshotComparisonReport

Called when a screenshot comparison result is reported

OnScreenshotTakenAndCompared

FSimpleMulticastDelegate FAutomationTestFramework::OnScreenshotTakenAndCompared

The final call related to screenshots, after they've been taken, and after they've been compared (or not if automation isn't running).

OnTestDataRetrieved

FOnTestDataRetrieved FAutomationTestFramework::OnTestDataRetrieved

Called when the test data is retrieved.

OnTestEndEvent

FOnTestEvent FAutomationTestFramework::OnTestEndEvent

Called when each automated test is ending

OnTestStartEvent

FOnTestEvent FAutomationTestFramework::OnTestStartEvent

Called when each automated test is starting

PostTestingEvent

FSimpleMulticastDelegate FAutomationTestFramework::PostTestingEvent

Called after all automated tests have completed

PreTestingEvent

FSimpleMulticastDelegate FAutomationTestFramework::PreTestingEvent

Called right before automated test is about to begin

---

The documentation for this class was generated from the following files:

• Engine/Source/Runtime/Core/Public/Misc/AutomationTest.h
• Engine/Source/Runtime/Core/Private/Misc/AutomationTest.cpp