ReṼoman is an API automation tool for JVM (Java/Kotlin) from the API-first SaaS company, Salesforce. It re-imagines API automation by letting you execute a Postman collection in a JVM program/test.
To start with, think of it as Postman for JVM (Java/Kotlin); that emulates the Postman Collection Runner through a Java program, essentially translating your manual testing into Automation, without any loss or resistance. But it’s even better!
It strikes a balance between flexibility provided by low-level tools like REST Assured or Cucumber and ease of use provided by UI tools like Postman
Maven
<dependency>
<groupId>com.salesforce.revoman</groupId>
<artifactId>revoman</artifactId>
<version>0.5.7</version>
</dependency>
Bazel
"com.salesforce.revoman:revoman"
Gradle Kts
implementation("com.salesforce.revoman:revoman:0.5.7")
Minimum Java Version required = 17
-
The majority of JVM SaaS applications are REST-based. But the API automation is done through a Mul-T-verse of Integration/Functional tests, E2E tests and Manual tests, each with its own frameworks, tools, and internal utilities, testing almost the same code flow.
-
These custom alien automation frameworks, often built using low-level tools like REST Assured, are specific to a service or domain and are rigid to reuse, extend and difficult to maintain.
-
This automation competes on cognitive complexity and learning curve with the Prod code, and mostly, automation wins.
-
After a point, the API automation may deviate from its purpose of augmenting real end-user interaction and turns into a foot-chain for development.
Contrary to these custom frameworks, almost every team uses Postman for manual testing their APIs. Postman collections contain a lot of information about your APIs and the order in which they need to be executed for manual testing, in a Structured Template. Leveraging it can mitigate writing a lot of code as we translate those manual steps into automation.
How productive would it be, if you could plug your exported Postman collection template, that you anyway would have created for your manual testing and executed through your JVM tests?
How about a Universal API automation tool that promotes low code and low-cognitive-complexity and strikes a balance between flexibility and ease of use?
-
The exported Postman collection JSON file is referred to as a Postman template, as it contains some placeholders/variables in the
{{variable-key}}
pattern. You can read more about it here -
ReṼoman understands these templates and replaces these variables at the runtime, similar to Postman. It supports
-
Nested variables, e.g.,
{{variable-key{{nested-variable-key}}}}
-
Dynamic variables, e.g.,
{{$randomUUID}}
,{{$randomEmail}}
-
ℹ️
|
In the case of collision between variable keys, the precedence order to derive a value to replace any key is:
|
You can kick off this Template-Driven Testing by invoking ReVoman.revUp()
,
supplying your Postman templates and environments, and all your customizations through a Configuration:
final var rundown =
ReVoman.revUp(
Kick.configure()
...
.off())
Here is a simple Exported Postman collection and Environment,
to hit a free public RESTFUL-API.
You can import and manually test this collection through the Run collection
button like this:
You can automate the same using ReṼoman in a Junit test by supplying the template and environment path:
@Test
@DisplayName("restful-api.dev")
void restfulApiDev() {
final var rundown =
ReVoman.revUp( // (1)
Kick.configure()
.templatePath(PM_COLLECTION_PATH) // (2)
.environmentPath(PM_ENVIRONMENT_PATH) // (3)
.off());
assertThat(rundown.firstUnIgnoredUnsuccessfulStepReport()).isNull(); // (4)
assertThat(rundown.stepReports).hasSize(3); // (5)
}
-
revUp
is the method to call passing a configuration, built as below -
Supply an exported Postman collection JSON file path
-
Supply an exported Postman environment JSON file path
-
Assert that the execution doesn’t have any failures
-
Run more assertions on the Rundown
After all this, you receive back a detailed Rundown in return. It contains everything you need to know about what happened in an execution, such that you can seamlessly run more assertions on top of the run.
Rundown(
val stepReports: List<StepReport>,
val mutableEnv: PostmanEnvironment<Any?>)
StepReport(
step: Step,
requestInfo: Either<RequestFailure, TxnInfo<Request>>? = null, // (1)
preStepHookFailure: PreStepHookFailure? = null,
responseInfo: Either<ResponseFailure, TxnInfo<Response>>? = null,
postStepHookFailure: PostStepHookFailure? = null,
envSnapshot: PostmanEnvironment<Any?> // (2)
)
-
Either
type from the VAVR library represents either of the two states, used here to represent error or success -
Snapshot of Environment at the end of each step execution. It can be compared with previous or next step environment snapshots to see what changed in this step
Rundown
has many convenient methods to ease applying further assertions on top of it.
💡
|
Other simple examples to see in Action: PokemonTest.java |
ReṼoman isn’t just limited to executing Collection like Postman; you can add more bells and whistles 🔔:
final var pqRundown =
ReVoman.revUp( // (1)
Kick.configure()
.templatePaths(PQ_TEMPLATE_PATHS) // (2)
.environmentPath(PQ_ENV_PATH) // (3)
.dynamicEnvironment( // (4)
Map.of(
"$quoteFieldsToQuery", "LineItemCount, CalculationStatus",
"$qliFieldsToQuery", "Id, Product2Id",
"$qlrFieldsToQuery", "Id, QuoteId, MainQuoteLineId, AssociatedQuoteLineId"))
.customDynamicVariableGenerator( // (5)
"$unitPrice",
(ignore1, ignore2, ignore3) -> String.valueOf(Random.Default.nextInt(999) + 1))
.nodeModulesRelativePath("js") // (6)
.haltOnFailureOfTypeExcept(
HTTP_STATUS,
afterAllStepsContainingHeader("ignoreHTTPStatusUnsuccessful")) // (7)
.requestConfig( // (8)
unmarshallRequest(
beforeStepContainingURIPathOfAny(PQ_URI_PATH),
PlaceQuoteInputRepresentation.class,
adapter(PlaceQuoteInputRepresentation.class)))
.responseConfig( // (9)
unmarshallResponse(
afterStepContainingURIPathOfAny(PQ_URI_PATH),
PlaceQuoteOutputRepresentation.class),
unmarshallResponse(
afterStepContainingURIPathOfAny(COMPOSITE_GRAPH_URI_PATH),
CompositeGraphResponse.class,
CompositeGraphResponse.ADAPTER))
.hooks( // (10)
pre(
beforeStepContainingURIPathOfAny(PQ_URI_PATH),
(step, requestInfo, rundown) -> {
if (requestInfo.containsHeader(IS_SYNC_HEADER)) {
LOGGER.info("This is a Sync step: {}", step);
}
}),
post(
afterStepContainingURIPathOfAny(PQ_URI_PATH),
(stepReport, ignore) -> {
validatePQResponse(stepReport); // (11)
final var isSyncStep =
stepReport.responseInfo.get().containsHeader(IS_SYNC_HEADER);
if (!isSyncStep) {
LOGGER.info(
"Waiting in PostHook of the Async Step: {}, for the Quote's Asynchronous processing to finish",
stepReport.step);
// ! CAUTION 10/09/23 gopala.akshintala: This can be flaky until
// polling is implemented
Thread.sleep(5000);
}
}),
post(
afterStepContainingURIPathOfAny(COMPOSITE_GRAPH_URI_PATH),
(stepReport, ignore) -> validateCompositeGraphResponse(stepReport)),
post(
afterStepName("query-quote-and-related-records"),
(ignore, rundown) -> assertAfterPQCreate(rundown.mutableEnv)))
.globalCustomTypeAdapter(new IDAdapter()) // (12)
.insecureHttp(true) // (13)
.off()); // Kick-off
assertThat(pqRundown.firstUnIgnoredUnsuccessfulStepReport()).isNull(); // (14)
assertThat(pqRundown.mutableEnv)
.containsAtLeastEntriesIn(
Map.of(
"quoteCalculationStatusForSkipPricing", PricingPref.Skip.completeStatus,
"quoteCalculationStatus", PricingPref.System.completeStatus,
"quoteCalculationStatusAfterAllUpdates", PricingPref.System.completeStatus));
-
revUp()
is the method to call passing a configuration, built as below -
Supply the path (relative to resources) to the Template Collection JSON file/files
-
Supply the path (relative to resources) to the Environment JSON file/files
-
Supply any dynamic environment that is runtime-specific
-
Node modules path (relative to resources) to be used inside Pre-req and Post-res scripts
-
Ignore Java cert issues when firing HTTP calls
-
Run more assertions on the Rundown
You can define a base common config and reuse it by overriding certain properties using the override…()
methods,
which are present for each config attribute
This tool has a particular emphasis on the Debugging experience. Here is what a debugger view of a Rundown looks like:
🔍 Let’s zoom into a detailed view of one of those Step reports, which contains complete Request and Response info along with failure information if any:
Here are the environment key-value pairs accumulated along the entire execution and appended to the environment from the file and dynamicEnvironment
supplied:
If something goes wrong at any stage during the Step execution, ReṼoman fails-fast and captures the Failure
in StepReport:
Here is the failure hierarchy of what can go wrong in this process
ReṼoman logs all the key operations that happen inside its source-code, including how the environment variables are being mutated by a step in its Pre-req and Post-res scripts. Watch your console to check what’s going on in the execution or troubleshoot from CI/CD logs
ℹ️
|
📝Sample log printed during execution |
Postman operates purely with JSON. When interoperating Postman with JVM, unmarshalling/deserialization of JSON into a POJO and vice versa helps in writing Type-safe JVM code and enhances the debugging experience on JVM. ReṼoman internally uses a modern JSON library called Moshi. Simple types whose JSON structure aligns with the POJO data structure can directly be converted. But when the JSON structure may not align with the POJO, you may need a Custom Type Adapter for Marshalling to JSON or Unmarshalling from JSON. Moshi has it covered for you with its advanced adapter mechanism and ReṼoman accepts Moshi adapters. Checkout these methods that help us interop between Postman and JVM:
There may be a POJO that inherits or contains legacy types that are hard or impossible to serialize.
ReṼoman lets you serialize only types that matter, through globalSkipTypes
,
where you can filter out these legacy types from Marshalling/Unmarshalling
-
Configure Moshi adapters to unmarshall/deserialize Request JSON payload to a POJO on certain steps.
-
You may use the bundled static factory methods named
RequestConfig.unmarshallResponse()
for expressiveness. -
You can pass a
PreTxnStepPick
which is aPredicate
used to qualify a step whose Request JSON payload needs to be unmarshalled/deserialized. -
If you have set up
requestConfig()
once, wherever you wish to read the request in your Pre-Step Hooks, you can callstepReport.requestInfo.get().<TypeT>getTypedTxnObj()
which returns your request JSON as a Strong type. -
If you don’t configure it for a step, Moshi unmarshalls the step request JSON into default data structures like
LinkedHashMap
-
Configure Moshi adapters to unmarshall/deserialize Response JSON payload to a POJO on certain steps.
-
You can configure separate adapters for success and error response. Success or Error response is determined by default with HTTP Status Code (SUCCESSFUL:
200 < = statusCode < = 299
). -
Use bundled static factory methods like
ResponseConfig.unmarshallSuccessResponse()
andResponseConfig.unmarshallErrorResponse()
for expressiveness. -
You can pass a
PostTxnStepPick
which is aPredicate
used to qualify a step whose Response JSON payload needs to be unmarshalled/deserialized. -
If you have set up
responseConfig()
once, wherever you wish to read or assert the response in your Post-Step Hooks, you can callstepReport.responseInfo.get().<TypeT>getTypedTxnObj()
which returns your response JSON as a Strong type to conveniently assert. -
If you don’t configure it for a step, Moshi unmarshalls the Step response JSON into default data structures like
LinkedHashMap
💡
|
|
-
These come handy when the same POJO/Data structure (e.g
ID
) is present across multiple Request or Response POJOs. These adapters compliment the custom adapters setup inrequestConfig()
orresponseConfig()
wherever these types are present. -
But these adapters won’t be used to marshall/unmarshall before or after each step execution. Which means you won’t be able to Strong types in the debug view. Although, you can get them on-demand using the respective
getTypedObj()
method.
ReṼoman also comes bundled with JSON Reader utils and JSON Writer utils to help build Moshi adapters.
💡
|
Refer ConnectInputRepWithGraphAdapter on how these utils come in handy in building an advanced Moshi adapter |
The bundled JSON POJO Utils can be used to directly to convert JSON to POJO and vice versa.
💡
|
See in Action: |
The configuration offers methods through which the execution strategy can be controlled without making any changes to the template:
-
haltOnAnyFailure
— This defaults tofalse
. If set totrue
, the execution fails-fast when it encounters a failure. -
haltOnFailureOfTypeExcept
— This accepts pairs ofExeType
and aPostTxnStepPick
which are used to check if a Step can be ignored for failure for a specific failure type -
runOnlySteps
,skipSteps
— All these accept apredicate
of typeExeStepPick
, which is invoked passing the currentStep
instance to decide whether to execute or skip a step.-
There are some
ExeStepPick
predicates bundled with ReṼoman underExeStepPick.PickUtils
e.gwithName
,inFolder
etc. You can write a custom predicate of your own too.
-
A hook lets you fiddle with the execution by plugging in your custom JVM code before or after a Step execution.
You can pass a PreTxnStepPick/PostTxnStepPick
which is a Predicate
used
to qualify a step for Pre-Step/Post-Step Hook respectively.
ReṼoman comes
bundled with some predicates under the namespace PreTxnStepPick.PickUtils
/PostTxnStepPick.PickUtils
e.g beforeStepContainingURIPathOfAny()
,
afterStepName()
etc. If those don’t fit your needs, you can write your own custom predicates like below:
final var preTxnStepPick = (currentStep, requestInfo, rundown) -> LOGGER.info("Picked `preLogHook` before stepName: {}", currentStep)
final var postTxnStepPick = (stepReport, rundown) -> LOGGER.info("Picked `postLogHook` after stepName: {}", stepReport.step.displayName)
Add them to the config as below:
.hooks(
pre(
preTxnStepPick,
(currentStepName, requestInfo, rundown) -> {
//...code...
}),
post(
postTxnStepPick,
(currentStepName, rundown) -> {
//...code...
})
)
-
You can do things like assertion on the rundown, Response Validations, or even mutate the environment with a value you programmatically derived, such that the execution of later steps picks up those changes.
-
Reserve hooks for plugging in your custom code or asserting and fail-fast in the middle of execution. If your assertions can wait till the final rundown, it’s cleaner to write them after the
revUp()
returns the rundown instead of adding hooks for each step
You can plug in your java code
to create/generate values for environment variables
which can be populated and picked-up by subsequent steps.
For example, you may want some xyzId
but you don’t have a Postman collection to create it.
Instead, you have a Java utility to generate/create it.
You can invoke the utility in a pre-hook of a step and set the value in rundown.mutableEnv
,
so the later steps can pick up value for {{xyzId}}
variable from the environment.
-
Post-Hooks are the best place to validate response right after the step.
-
If you have configured a strong type for your response through
responseConfig
, you can write type-safe validations by extracting your Strong type Object usingstepReport.responseInfo.get().<TypeT>getTypedTxnObj()
(if you have configuredresponseConfig()
orglobalCustomTypeAdapters()
) or useJsonPojoUtils.jsonToPojo(TypeT, stepReport.responseInfo.get().httpMsg.bodyString())
to convert it inline. -
If your response data structure is non-trivial and has requirements to execute validations with different strategies like
fail-fast
orerror-accumulation
, consider using a library like Vador
-
Postman lets you write custom JavaScript in Pre-req and Post-res tabs that get executed before and after a step respectively. When you export the collection as a template, these scripts also come bundled.
-
ReṼoman can execute this JavaScript on JVM. This support ensures that the Postman collection used for manual testing can be used as-is for the automation also, without any resistance to modify or overhead of maintaining separate versions for manual and automation.
-
Pre-req JS script is executed as the first step before Unmarshall request.
-
Post-res JS script is executed right after receiving an HTTP response.
-
-
ReṼoman supports using
npm
modules inside your Pre-req and Post-res JS scripts. You can installnpm
modules in any folder using traditional commands likenpm install <module>
and supply in theKick
config, the relative path of the parent folder that contains thenode_modules
folder usingnodeModulesRelativePath(…)
. Use thosenpm
modules inside your scripts withrequire(…)
, for example:
moment
with npmnpm install moment
var moment = require("moment")
var _ = require('lodash')
pm.environment.set("$currentDate", moment().format(("YYYY-MM-DD")))
var futureDateTime = moment().add(365, 'days')
pm.environment.set('$randomFutureDate', futureDateTime.format('YYYY-MM-DD'))
pm.environment.set("$quantity", _.random(1, 10))
💡
|
|
🔥
|
The recommendation is not to add too much code in Pre-req and Post-res scripts, as it’s not intuitive to troubleshoot through debugging. Use it for simple operations that can be understood without debugging, and use Pre-Step /Post-Step Hooks for any non-trivial operations, which are intuitive to debug. |
-
Environment is the only mutable-shared state across step executions, which can be used for data passing between the consumer and the library.
-
This can be mutated (set key-value pairs) through Pre-req and Post-res scripts (using
pm.environment.set()
) and Pre-Step /Post-Step Hooks (using the referencerundown.mutableEnv
) during execution.
You may want to troubleshoot manually with Postman using the Mutable environment built during the ReṼoman execution.
rundown.mutableEnv.postmanEnvJSONFormat()
converts the mutable environment into a Postman JSON format,
so you can copy and import that environment conveniently into Postman.
💡
|
You do NOT need to save the copied Postman environment JSON from the debugger into file. You can paste (kbd:[Ctrl+v]) directly in the Postman environment window |
You can read any value from mutableEnv as a Strong type using rundown.mutableEnv.<TypeT>getTypedObj()
See it in action: getTypedObj()
test from PostmanEnvironmentTest
-
You don’t have to squash all your steps into one mega collection. Instead, you can break them into easy-to-manage modular collections.
ReVoman.revUp()
accepts a list of collection paths throughtemplatePaths()
-
But that doesn’t mean you have to execute all these templates in one-go. You can make multiple
ReVoman.revUp()
calls for different collections. -
If you wish to compose these executions in a specific order, you can use the
revUp()
overload which accepts a varargKick
configs.-
You can also achieve the same yourself, by adding the previous execution’s
mutableEnv
to the current execution using thedynamicEnvironment
parameter. This also comes in handy when you wish to execute a common step (e.g.UserSetup
) inside a Test setup method and use that environment for all the tests.
-
💡
|
Here is an example of a low-code E2E test that automates ~75 steps |
Compared to a traditional Integration/Functional or E2E test, approximately, the amount of code needed is 89% less using ReṼoman. The above test doesn’t just have low code, but also low in Cognitive Complexity and Transparent in what it does.
Familiarity with Postman gets you a long way in understanding this tool. Playing with the existing Integration Tests and writing a couple of hands-on tests should get you started.
-
ReṼoman is like any JVM library that you can plug into any JVM program/test (e.g., JUnit tests or Integration tests).
-
Apart from adding a dependency in the build tool, there is no extra setup needed to execute these tests with ReṼoman in CI/CD.
-
A nice side effect is, this lets the Postman collections always stay up to date and the entire Postman collection guards each check-in in the form of a Test suite augmenting manual testing.
-
Any day, you can find an up-to-date Postman collection for every feature you need to test, right in your VCS (Git) along with your code. Developers can import these templates directly from VCS into Postman for manual testing. This comes in very handy during a team blitz.
-
ReṼoman brings a Unified & Simplified Test strategy across the mul-T-verse (Integration Tests, E2E Tests, and Manual testing with Postman) for any API.
-
The automation stays as close as possible to Persona-based Manual testing, leading to Transparency and better Traceability of issues
-
This forces engineers to think like API-first customers while writing tests.
-
Test Data setup: You can use the ReṼoman for the Test data setup too. This eliminates the need for different teams to write their own internal utilities for data setup.
-
E2E Test can even reside outside the Service repo, as long as it can hit the service API
The future looks bright with multiple impactful features in the pipeline:
-
API metrics and Analytics
-
It’s built with extensibility in mind. It can easily be extended to support other template formats, such as Kaiju templates used for availability testing.
-
In-built polling support for Async steps
-
Payload generation
-
Flow control through YAML config
-
You can add a pre-hook to the Step you are interested and add a debug point inside that. This gets hit before ReṼoman fires the request in that Step
-
You can get more adventurous by attaching revoman jar sources and directly adding conditional debug points inside this library source-code. You can search for logs in the source-code that indicate key operations to add conditional debug points with conditions like StepName etc.
-
You can add key-value pairs to a Step’s HTTP Headers section (e.g.,
ignoreHTTPStatusUnsuccessful=true
). -
You can use this information in Step Picks or Pre-Step and Post-Step Hooks to identify a particular step to execute any conditional logic
You don’t have to. This is a JVM-first tool, and you can interlace your TestUtils through Pre-Step/Post-Step Hooks
Why not use Newman or Postman CLI?
-
ReṼoman may be similar to Newman or Postman CLI when it comes to executing a Postman collection, but the similarities end there.
-
Newman or Postman CLI are built for node and cannot be executed within a JVM. Even if you are able to run in some hacky way, there is no easy way to assert results.
-
ReṼoman is JVM first that lets you configure a lot more, and gives you back a detailed report to assert in a typesafe way
-
This CONTRIBUTING doc has all the information to set up this library on your local and get hands-on.
-
Any issues or PRs are welcome!
♥️ -
Join this Slack Community to discuss issues or PRs related to Consumption-Collaboration-Contribution