Making Sure We Get It Right: A Technical View (Guest Blog from Spotlight CTO Dan Finkelstein)

Here at Spotlight, we take a number of steps to ensure that the reports that we deliver to you are unfailingly precise.  Whether you want one report or 100,000 reports, each report must faithfully reflect your intentions.  In this blog, I wanted to describe our how our automated quality assurance works.

When we produce a report, we run any “QA Rules” that are part of a report.  These QA Rules are simply functions that assert that what you expect to happen really happened.  Let me run through two examples to show you how it might work.

As a first example, here is a rule that checks that the number of occupations is exactly 10.  The name of this rule is the name of the function, checkOccupation().  Our report generation is based on the concept of “parameters,” each of which will have a calculated value.  In this example, params.occupations represents the list of occupations and hopefully will have a value of “10.”  Finally, the function returns a RuleResult object, which can be either PASS, FAIL or WARN. 

def RuleResult checkOccupation() {

params.occupations.size == 10 ? PASS : FAIL

}

As a second example, here’s an example of how to check that the report has the number of pages that you expect:

def RuleResult checkPagesCount() {

pdf.numberOfPages == 3 ? PASS : FAIL

}

What is special about this example is that we read the PDF report that we wrote, and use this for checking.  By basing this test on the exact file that will be delivered to a client, it increases our confidence of delivering the expected report.

I hope these examples give you a quick task for what our QA Rules can do.

Using this feature, our report designers can easily write rules to ensure that a report has been created correctly.  In fact, it’s not uncommon that a report might have hundreds of rules.

When we generate reports, we keep track of any rules that failed or produced warnings.  Warnings are useful for issues that you consider non-fatal but would still like to review later.  When the report generation process has completed, you can easily view any errors or warnings that have been recorded.

I hope this blog gives you a sense of how our automated QA Rules processing works.  In future blogs, I’ll discuss other QA processes which alongside these QA Rules ensure consistent, high-quality results.