Integration and Unit testing Nifi process groups

[JamesZimmerman]

Hi

I have a few Nifi process groups which I want to run integration tests on before promoting to production. The issue is that I can't seem to find any documentation on how to do so.

Data Provenance seems like a promising tool to accomplish what I want, however, over the course of the flowfile's lifecycle, data is published to/from kafka or the file system. As a result, the flowfile UUID changes so I cannot query for it using the nifi-api.

Additionally, I know that Nifi offers a TestRunner library to run tests, however, this seems to only be for processors/processor groups generated via code and not the UI.

Does anyone know of a tool, framework, or pattern for integration and unit testing nifi process groups. Ideally this would be a solution where you can programmatically compare input/output of the processor/processor group without modifying the existing workflow.

[StanGilbertland]

In the case of a "test harness" wouldn't that require some kind of input/output ports of added to the process group? If that's the case then it kind of botches the idea of using some sort of automated workflow test suite so as tests pass I can promote to QA, prod, etc. as each environment would need to modify the workflow to meet it's needs. I've seen Nifi Registry and the Nipy or Nifi CLI to accomplish tasks of promoting between environments, but I have not seen anything on testing.

[VictorJohnsonan]

With the introduction of the Apache NiFi Registry, we have seen users promote flows from a development/sandbox environment to a test/QE environment where there are existing "test harness" flows surrounding the "flow under test" so that they can send repeatable and deterministic (or an anonymized sample of real production data) through the flow and compare the results to an expected value.

As you point out, there is a TestRunner class and a whole testing framework provided for unit tests. While it can be difficult to manually translate a UI-constructed flow to the programmatic construction, you could also create something like a translator to accept a flow template or flow.xml.gz file and convert it into something processable by  the test framework.