Here we create a page in the Spring Boot app containing a table that shows the data we’ve extracted using the Form Recognizer API. Then, use a charting library like C3.js to visualize the form data we’ve extracted.
Synopsis
This three-article series demonstrates how to use Azure Form Recognizer to build a realistic end-to-end form recognition app using Java and Spring Boot.
- Part 1 — App Creation and Deployment via Azure App Service
- Part 2 — Adding Image Upload to the Spring Boot App and Processing via Form Recognizer
- Part 3 — Making Practical Use of Data Returned by Form Recognizer
The complete code repository is available on GitHub.
Introduction
We created our project and infrastructure in the first article in this series.
In the second article, we learned how to upload and send receipt images for recognition. We stored the recognition results in an instance of Azure database for PostgreSQL.
Now that we have our data in the database, we can move forward and make practical use of it. In this final article of the tutorial series, we’ll demonstrate how to build a couple sample use cases.
The first use case displays a table with our recognition results.
The second use case renders a chart displaying a line plot of our receipt totals.
Of course, these are just two examples. There are endless opportunities for much more advanced scenarios. You are only limited by your imagination.
Presenting the Table of Recognition Results
To display a table with our recognition results, we need to make some changes to the results.html view.
Open resources/templates/results.html and add the code between the <!-- comment --> tags in the following example:
<html lang="en">
<head>
<link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css">
</head>
<body class="w3-black">
<header class="w3-container w3-padding-32 w3-center w3-black">
<h1 class="w3-jumbo">Form recognizer</h1>
<p>Recognition results</p>
<h3>
<a th:href="@{/}">Home</a>
</h3>
</header>
<!--
<table align="center" cellpadding="10">
<thead>
<tr>
<th>File name</th>
<th>Merchant name</th>
<th>Transaction date</th>
<th>Total</th>
</tr>
</thead>
<tbody>
<tr th:each="result : ${recognitionResults}">
<td th:text="${result.receiptFileName}">receiptFileName</td>
<td th:text="${result.merchantName}">merchantName</td>
<td th:text="${result.transactionDate}">transactionDate</td>
<td th:text="${#numbers.formatCurrency(result.total)}">total</td>
</tr>
</tbody>
</table>
<!--
</body>
</html>
Our table uses the Thymeleaf rendering engine to display four fields from the RecognitionResult class: receiptFileName, merchantName, transactionDate, and total. These values are rendered within their dedicated table cells. Note that the total field is formatted as currency using Thymleaf’s formatter, the numbers.formatCurrency method.
The table has a row for each item. It injects these rows into the view from the results method of the FormRecognitionController:
@GetMapping("/results")
public ModelAndView results() {
List<RecognitionResult> recognitionResults = resultsRepository.findAll();
ModelAndView modelAndView = new ModelAndView("results");
modelAndView.addObject("recognitionResults", recognitionResults);
return modelAndView;
}
After recompiling and rerunning the app, select the Recognition results link on the index page to see the table.
Presenting the Line Plot Chart
To render our chart, we must first modify index.html by adding a hyperlink to another view:
In index.html, add a Chart link as shown in the following example:
<header class="w3-container w3-padding-32 w3-center w3-black">
<h1 class="w3-jumbo">Form recognizer</h1>
<p>Azure AI</p>
<h3>
<a th:href="@{/upload}">Upload image</a> |
<a th:href="@{/results}">Recognition results</a> |
<a th:href="@{/chart}">Chart</a>
</h3>
</header>
Next, we implement the chart method in FormRecognitionController. This method simply returns the name of the template:
@GetMapping("/chart")
public String chart() {
return "chart";
}
Then, we need to update the resources/templates folder with the chart.html view. As with the other views in our application, it uses W3.CSS:
<html lang="en">
<head>
<link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.9.3/Chart.bundle.min.js" type="text/javascript"></script>
</head>
<body class="w3-black">
<header class="w3-container w3-padding-32 w3-center w3-black">
<h1 class="w3-jumbo">Form recognizer</h1>
<p>Chart</p>
<h3>
<a th:href="@{/}">Home</a>
</h3>
<canvas id="chart" height="100"></canvas>
</header>
</body>
</html>
We use the Chart.js library to rapidly render the chart. To do this, we only need to reference the Chart.bundle.min.js file in our chart.html head via the Cloudflare CDN.
Note that Chart.js also requires that we provide an HTML canvas element on which to render our chart. We define this element near the bottom of the HTML body.
Now let’s ensure that the chart.html view is rendered properly.
Rerun the app. The canvas renders, but the plot line is missing. To display it, we still need to provide data and explicitly instantiate the Chart.js object.
Populating the Chart
To provide the data for the chart, we update the FormRecognitionController by adding the getChartData method:
@RequestMapping(value = "/getChartData", method = RequestMethod.GET)
@ResponseBody
public String getChartData() throws JsonProcessingException {
ObjectMapper objectMapper = new ObjectMapper();
List<Double> items = resultsRepository.findAll().stream().map(
r -> r.getTotal()).collect(Collectors.toList());
return objectMapper.writeValueAsString(items);
}
This method exposes a GET method that returns the JSON array we need. This array is constructed using data from our database.
Specifically, we first retrieve all the items from the table containing our recognition results. This collection is remapped to the array of doubles that comprises the total field on each recognized receipt. Then, the array of doubles is converted to the JSON array using the ObjectMapper from the Jackson package. This package was previously imported by azure-sdk-bom, so we don’t need to modify pom.xml.
To test the method, first recompile your app using mvn clean install. Then, run the app using mvn spring-boot:run.
Next, navigate to localhost/getChartData. The output looks like this:
The back-end functionality is now ready, so let’s use it to render our chart.
To render the chart, update chart.html by adding the JavaScript code between the <!-- comment --> tags in the following example:
<head>
<link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.9.3/Chart.bundle.min.js" type="text/javascript"></script>
<!--
<script>
$(() => {
const ctx = $('#chart');
$.get('/getChartData').done((response) => {
const chartData = JSON.parse(response);
const generateLabels = (count) => {
var xData = [];
for(var i = 0; i < count; i++){
xData.push(i+1);
}
return xData;
};
var chart = new Chart(ctx, {
type: 'line',
data: {
labels: generateLabels(chartData.length),
datasets: [{
label: "Total receipt value",
data: chartData,
borderWidth: 3,
lineTension: 0,
borderColor: [
'rgba(255, 255, 0, 1)'
]
}]
}
});
});
});
</script>
<!--
</head>
This script sends the GET request via the /getChartData endpoint. It then deserializes the resulting JSON to the JavaScript array, which is stored in the chartData variable. Finally, we construct the Chart object from Chart.js to display a line plot based on the series of (X,Y) points it receives.
The Chart constructor accepts two arguments. The first argument is a reference to the HTML canvas element.
The second argument is the object representing the data to be plotted. This object contains the data and the chart type — in our case, a line plot.
We pass the data in the form of another anonymous object. In our case, this object contains two explicit fields: labels and datasets. For labels, we generate an array of integers (1,2,…N), where N is the number of items in the chartData variable. The labels array is generated dynamically using the helper function generateLabels. Chart.js uses these values to set the X-value variables for the plotted line.
For datasets, we construct another anonymous object. This object contains labels used for the chart legend, the actual data, and plotting parameters like line width, tension, and color. We set the line thickness to 3 pixels and the color to yellow. The Y-values of the plot data itself come from the chartData variable, which is retrieved from the back end.
Now, after recompiling and rerunning the app, select the Chart link. The app directs you to another page, which displays the chart.
You can now modify this template to meet the needs of your business scenario. For additional modification options, refer to the Chart.js documentation.
Final Deployment to Azure Service
We can now deploy the final version of the application to Azure App Service. Keep in mind that we can deploy the app straight from Visual Studio Code.
To do deploy the app from Visual Studio Code, select Deploy to Web App..., which appears as a cloud icon to the right of APP SERVICE on the AZURE panel.
All the settings are already stored, so Visual Studio only displays a confirmation dialog.
Click Deploy.
The service takes a moment to update the deployment. When you open your app, which now runs in the Azure cloud, the output looks like this:
Alternatively, you can deploy using the Azure CLI with the az webapp deploy command. This is beneficial if you want to automate your deployment pipeline.
Summary
In this three-part tutorial series, we learned how to rapidly create a customizable solution for the needs of a business traveler by utilizing Spring Boot MVC, AI, and Microsoft Azure.
We use a pre-trained receipt recognizer to automate the tedious process of entering data from business expense receipts. Azure Form Recognizer enables us to automatically extract the fields we need using AI without having to train our own AI models. This lets us build advanced solutions with the potential to simplify a wide variety of manual tasks.
Another possible use for our AI form recognition functionality is exposing the receipt recognition as a REST API and calling it from mobile apps.
Azure Form Recognizer also provides a general document model to recognize fields in varied document formats. We can use it to analyze company documents or automate search patterns, among other uses.
Finally, Azure Form Recognizer enables us to train custom models for meeting specialized needs or building on an established technological foundation. This is particularly useful in specialized applications that require optical character recognition supported by deep learning.
To learn more tips for the easiest ways to deliver Java code to Azure and other clouds, check out the webinar Azure webinar series - Delivering Java to the Cloud with Azure and GitHub.
Dawid Borycki is a software engineer and biomedical researcher with extensive experience in Microsoft technologies. He has completed a broad range of challenging projects involving the development of software for device prototypes (mostly medical equipment), embedded device interfacing, and desktop and mobile programming. Borycki is an author of two Microsoft Press books: “Programming for Mixed Reality (2018)” and “Programming for the Internet of Things (2017).”
Submit an article or tip