Authorizing Resources Based On Who Created Them

A colleague of mine pointed me to an interesting question on StackOverflow and suggested it may be a good one for me to answer because of my experience with Spring.

The question was, “How to authorize specific resources based on users who created those in REST, using annotations.”

The gist of it is this:

What I’m trying to do is create an annotation named @Authorize and use it on methods which needs user authorization in order to perform some action( the user is already authenticated at this point). eg. I have an order service with a getOrder() method. I want only the user who created this order to access it.

My Answer on StackOverflow

To implement authorization controls on methods in Java, I highly recommend Spring Security with an eXtensible Access Control Markup Language (XACML) implementation that has a Spring Security API.

Spring Security

Spring Security provides two main means to protect access to methods:

  • Preauthorization: this allows for certain conditions/constraints to be checked before the execution of the method is allowed. Failure to verify these conditions will result in the failure to call the method.
  • Postauthorization: this allows for certain conditions/constraints to be checked after the method returns. This is used less often that preauthorization check, but can be used to provide extra security around complex interconnected business tier methods, especially around constraints related to the object returned by the method.

Say for example, that one of the access control rule is that the user has have the ROLE_ADMIN authority before being able to invoke a method getEvents(). The way to do that within the Spring Security framework would be to use the PreAuthorize annotation as below:

public interface Sample { ... 
@PostAuthorize("hasRole('ROLE_ADMIN')") 
Event getEvent(); } 

In essence Spring Security uses a runtime Aspect Oriented Programming (AOP) pointcut to execute before an advice on the method and throw an o.s.s.access.AccessDeniedException if the security constraints specified are not met.

More can be found about Spring Security’s Method Level Security in section 27.3 of this documentation.

eXtensible Access Control Markup Language (XACML) – a policy language for ABAC

Spring Security does a great job of implementing access control with its expression based access control, but attribute based access control (ABAC) allows more fine grained control of access and is recommended by the National Institute of Standards and Technology.

To address the limitations of Role Based Access Control (RBAC), NIST came up with a new model called ABAC (Attribute Based Access Control). In ABAC, you can now use more metadata / parameters. You can for instance consider:

  • a user’s identity, role, job title, location, department, date of birth…
  • a resource’s type, location, owner, value, department…
  • contextual information e.g. time of day the action the user is attempting on the resource

All these are called attributes. Attributes are the foundation of ABAC, hence the name. You can assemble these attributes into policies. Policies are a bit like the secret sauce of ABAC. Policies can grant and deny access. For instance:

  • An employee can view a record if the employee and the record are in the same region
  • Deny access to reading records between 5pm and 8am.

Policies can be used to express advanced scenarios e.g.

  • segregation of duty
  • time-based constraints (see above)
  • relationship-based access control (see above)
  • delegation rules delegate Bob access to Alice’s document.

There are 2 main syntaxes available to write policies:

ABAC also comes with an architecture to define how the policies will get evaluated and enforced.

ABAC GRAPH

The architecture contains the following components:

  • the Policy Enforcement Point (PEP): this is the component that secures the API / application you want to protect. The PEP intercepts the flow, analyzes it, and send an authorization request to the PDP (see below). It then receives a decision (Permit/Deny) which it enforces.
  • the Policy Decision Point (PDP) receives an authorization request (e.g. can Alice view record #123?) and evaluates it against the set of policies it has been configured with. It eventually reaches a decision which it sends back to the PEP. During the evaluation process, the PDP may need additional metadata e.g. a user’s job title. To that effect, it can turn to policy information points (PIP)
  • the Policy Information Point (PIP) is the interface between the PDP and underlying data sources e.g. an LDAP, a database, a REST service which contain metadata about users, resources, or other. You can use PIPs to retrieve information the PDP may need at runtime e.g. a risk score, a record’s location, or other.

Implementations of XACML

Full disclosure – I am on the XACML Technical Committee and work for Axiomatics, a provider of dynamic authorization that implements XACML.

Axiomatics provides a Spring Security SDK for their Axiomatics Policy Server and it provides four expressions that can be used to query the PDP as a part of protecting a method invocation

  1. xacmlDecisionPreAuthz, called with @PreAuthorize
  2. xacmlDecisionPostAuthz, called with @PostAuthorize
  3. xacmlDecisionPreFilter, called with @PreFilter
  4. xacmlDecisionPostFilter, called with @PostFilter

The exact signatures for these methods are as follows:

  1. xacmlDecisionPreAuthz(Collection<String> attributeCats,
    Collection<String> attributeTypes, Collection<String> attributeIds,
    ArrayList<Object> attributeValues)
  2. xacmlDecisionPostAuthz(Collection<String> attributeCats,
    Collection<String> attributeTypes, Collection<String> attributeIds,
    ArrayList<Object> attributeValues)
  3. xacmlDecisionPreFilter(Collection<String> attributeCats, Collection<String>
    attributeTypes, Collection<String> attributeIds, ArrayList<Object>
    attributeValues)
  4. xacmlDecisionPostFilter (Collection<String>
    attributeCats, Collection<String> attributeTypes, Collection<String>
    attributeIds, ArrayList<Object> attributeValues)

For an entire list of XACML implementations, you can check this list on Wikipedia.

Selenium with Java: Google Search

1. Overview

In this tutorial, we will be exploring the basics of how to use Selenium with Java. We will use Selenium to open Google, search, and click a URL.

The code is available on Github.

2. What is Selenium?

Selenium automates web browsers. That’s really it.

Selenium enables us to emulate user interaction with a web page. There are two Selenium products we can use: Selenium WebDriver and Selenium IDE. We will be using WebDriver.

What is WebDriver? WebDriver is an official W3C Specification, and in essence it is a way of interacting with a web browser. Previously, with Selenium RC, Selenium would operate with the browser by injecting JavaScript to interact with elements. With the adoption of the WebDriver specification, companies like Google, Mozilla, and Microsoft release their browser with the ability to be controlled by a hook, that Selenium can tap into. This hook enables Selenium to interact with the web browser in the same way that humans do.

We will be using Google Chrome and therefore it is required that we download the chromedriver.

After downloading the driver, we need to execute the file.
On Macs, we can simply do this for instance:

./chromedriver

3. pom.xml

I use Spring Tool Suite and created a new Spring Starter project, which wasn’t necessary, but I tend to like Spring. So Selenium is managed by the Spring Boot Starter Parent actually. The version is 2.53.1.

<!-- typical pom beginning-->
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.10.RELEASE</version>
		<relativePath /> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>

		<dependency>
			<groupId>com.h2database</groupId>
			<artifactId>h2</artifactId>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		
		<dependency>
			<groupId>org.seleniumhq.selenium</groupId>
			<artifactId>selenium-java</artifactId>
		</dependency>

	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>
<!-- typical pom ending-->

4. Open Chrome and Search

For this step, we will be establishing the connection to the chromedriver, opening the browser, and searching for “Selenium”.

The port we target for our localhost is 9515 because the chromedriver runs on the local server’s port 9515.

RemoteWebDriver implements WebDriver and WebDriver’s goal is to supply an object-oriented API that provides support for modern advanced web-app testing problems. So, we can tell based on these facts that RemoteWebDriver is the implementation that allows the use of a remote browser. The benefits include separation of where the tests run from where the browser is and the ability to test with browsers not available on the current OS. The cons include the fact that we need an external servlet container to be running and there may be latency if an exception is thrown.

            // create a Chrome Web Driver
            URL local = new URL("http://localhost:9515");
            WebDriver driver = new RemoteWebDriver(local, DesiredCapabilities.chrome());
            // open the browser and go to open google.com
            driver.get("https://www.google.com"); 
            
            driver.findElement(By.id("lst-ib")).sendKeys("Selenium");
            driver.findElement(By.name("btnK")).click();
            driver.manage().window().maximize();

5. Get Pages and Click

WebDriver gives us the methods findElement and findElements method to locate element(s) on a web page. These methods accept a By object as a parameter. By has methods to locate elements within a document with the help of a locator value. Selenium has documented their API well.

Once we understand how Selenium is used to identify elements, it is easy to read any of the the driver.findElements(By…) methods. But we need to know how to write them too. Using a browser, like Chrome, we can right click (or the equivalent) to Inspect an element to get its HTML/CSS information. Also, we can “View Source” to get more complete information as well.

To demonstrate how to scroll on a web page, we perform jse.executeScript(“window.scrollBy(0,250)”, “”).
Like the name suggests, JavaScriptExecutor executes JavaScript. JavaScriptExecutor is an interface provided through Selenium WebDriver. It provides two methods “executescript” & “executeAsyncScript” to run javascript on the selected window or current page.

With the code below, it may be possible to create a more comprehensive bot to search Google and click URLs for several pages.

         // get the number of pages
            int size = driver.findElements(By.cssSelector("[valign='top'] > td")).size();
            for(int j = 1 ; j < size ; j++) {
                if (j > 1) {// we don't need to navigate to the first page
                    driver.findElement(By.cssSelector("[aria-label='Page " + j + "']")).click(); // navigate to page number j
                }

                String pagesearch = driver.getCurrentUrl();

                List<WebElement> findElements = driver.findElements(By.xpath("//*[@id='rso']//h3/a"));
                System.out.println(findElements.size());

                for(int i=0;i<findElements.size();i++){
                    findElements= driver.findElements(By.xpath("//*[@id='rso']//h3/a"));                
                    findElements.get(i).click(); 

                    driver.navigate().to(pagesearch);
                    JavascriptExecutor jse = (JavascriptExecutor) driver;
                    //Scroll vertically downward by 250 pixels
                    jse.executeScript("window.scrollBy(0,250)", "");
                }
            }

6. Conclusion

This was a basic introduction to Selenium with Java. As we discovered, in Selenium Webdriver, locators like XPath, CSS, etc. are used to identify and perform operations on a web page. It is also possible to execute arbitrary JavaScript.
The complete code can be found on Github.

Converting HTML to RichTextString for Apache POI

1. Overview

In this tutorial, we will be building an application that takes HTML as an input and creates a Microsoft Excel Workbook with a RichText representation of the HTML that was provided. To generate the Microsoft Excel Workbook, we will be using Apache POI. To analyze the HTML, we will be using Jericho.

The full source code for this tutorial is available on Github.

2. What is Jericho?

Jericho is a java library that allows analysis and manipulation of parts of an HTML document, including server-side tags, while reproducing verbatim any unrecognized or invalid HTML. It also provides high-level HTML form manipulation functions. It is an open source library released under the following licenses: Eclipse Public License (EPL)GNU Lesser General Public License (LGPL), and Apache License.

I found Jericho to be very easy to use for achieving my goal of converting HTML to RichText.

3. pom.xml

Here are the required dependencies for the application we are building. Please take note that for this application we have to use Java 9. This is because of a java.util.regex appendReplacement method we use that has only been available since Java 9.

<parent>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-parent</artifactId>
	<version>1.5.9.RELEASE</version>
	<relativePath /> <!-- lookup parent from repository -->
</parent>

<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
	<java.version>9</java.version>
</properties>

<dependencies>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-batch</artifactId>
	</dependency>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-thymeleaf</artifactId>
	</dependency>

	<dependency>
		<groupId>com.h2database</groupId>
		<artifactId>h2</artifactId>
		<scope>runtime</scope>
	</dependency>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-test</artifactId>
		<scope>test</scope>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
	<dependency>
		<groupId>org.apache.commons</groupId>
		<artifactId>commons-lang3</artifactId>
		<version>3.7</version>
	</dependency>
	<dependency>
		<groupId>org.springframework.batch</groupId>
		<artifactId>spring-batch-test</artifactId>
		<scope>test</scope>
	</dependency>
	<dependency>
		<groupId>org.apache.poi</groupId>
		<artifactId>poi</artifactId>
		<version>3.15</version>
	</dependency>

	<dependency>
		<groupId>org.apache.poi</groupId>
		<artifactId>poi-ooxml</artifactId>
		<version>3.15</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/net.htmlparser.jericho/jericho-html -->
	<dependency>
		<groupId>net.htmlparser.jericho</groupId>
		<artifactId>jericho-html</artifactId>
		<version>3.4</version>
	</dependency>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-configuration-processor</artifactId>
		<optional>true</optional>
	</dependency>
	<!-- legacy html allow -->
	<dependency>
		<groupId>net.sourceforge.nekohtml</groupId>
		<artifactId>nekohtml</artifactId>
	</dependency>
</dependencies>

4. Web page – Thymeleaf

We use Thymeleaf to create a basic webpage that has a form with a textarea. The source code for Thymeleaf page is available here on Github. This textarea could be replaced with a RichText Editor if we like, such as CKEditor. We just must be mindful to make the data for AJAX correct, using an appropriate setData method. There is a previous tutorial about CKeditor titled AJAX with CKEditor in Spring Boot.

5. Controller

In our controller, we Autowire JobLauncher and a Spring Batch job we are going to create called GenerateExcel. Autowiring these two classes allow us to run the Spring Batch Job GenerateExcel on demand when a POST request is sent to “/export”.

Another thing to note is that to ensure that the Spring Batch job will run more than once we include unique parameters with this code: addLong(“uniqueness”, System.nanoTime()).toJobParameters(). An error may occur if we do not include unique parameters because only unique JobInstances may be created and executed, and Spring Batch has no way of distinguishing between the first and second JobInstance otherwise.

@Controller
public class WebController {

    private String currentContent;

    @Autowired
    JobLauncher jobLauncher;
    
    @Autowired
    GenerateExcel exceljob; 

    @GetMapping("/")
    public ModelAndView getHome() {
        ModelAndView modelAndView = new ModelAndView("index");
        return modelAndView;

    }
    

    @PostMapping("/export")
    public String postTheFile(@RequestBody String body, RedirectAttributes redirectAttributes, Model model)
        throws IOException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException, JobParametersInvalidException {


        setCurrentContent(body);

        Job job = exceljob.ExcelGenerator();
        jobLauncher.run(job, new JobParametersBuilder().addLong("uniqueness", System.nanoTime()).toJobParameters()
            );

        return "redirect:/";
    }

    //standard getters and setters

}



6. Batch Job

In Step1 of our Batch job, we call the getCurrentContent() method to get the content that was passed into the Thymeleaf form, create a new XSSFWorkbook, specify an arbitrary Microsoft Excel Sheet tab name, and then pass all three variables into the createWorksheet method that we will be making in the next step of our tutorial :

@Configuration
@EnableBatchProcessing
@Lazy
public class GenerateExcel {
    
    List<String> docIds = new ArrayList<String>();

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Autowired
    WebController webcontroller;
    
    @Autowired
    CreateWorksheet createexcel;

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
            .tasklet(new Tasklet() {
                @Override
                public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception, JSONException {

                    String content = webcontroller.getCurrentContent();
                    
                    System.out.println("content is ::" + content);
                    Workbook wb = new XSSFWorkbook();
                    String tabName = "some";
                    createexcel.createWorkSheet(wb, content, tabName);

                    return RepeatStatus.FINISHED;
                }
            })
            .build();
    }

    @Bean
    public Job ExcelGenerator() {
        return jobBuilderFactory.get("ExcelGenerator")
            .start(step1())
            .build();

    }

}

We have covered Spring Batch in other tutorials such as Converting XML to JSON + Spring Batch and Spring Batch CSV Processing.

7. Excel Creation Service

We use a variety of classes to create our Microsoft Excel file. Order matters when dealing with converting HTML to RichText, so this will be a focus.

7.1 RichTextDetails

A class with two parameters: a String that will have our contents that will become RichText and a font map.

public class RichTextDetails {
    private String richText;
    private Map<Integer, Font> fontMap;
    //standard getters and setters
    @Override
    public int hashCode() {
     
        // The goal is to have a more efficient hashcode than standard one.
        return richText.hashCode();
    }

7.2 RichTextInfo

A POJO that will keep track of the location of the RichText and what not:

public class RichTextInfo {
    private int startIndex;
    private int endIndex;
    private STYLES fontStyle;
    private String fontValue;
    // standard getters and setters, and the like

7.3 Styles

A enum to contains HTML tags that we want to process. We can add to this as necessary:

public enum STYLES {
    BOLD("b"), 
    EM("em"), 
    STRONG("strong"), 
    COLOR("color"), 
    UNDERLINE("u"), 
    SPAN("span"), 
    ITALLICS("i"), 
    UNKNOWN("unknown"),
    PRE("pre");
    // standard getters and setters

7.4 TagInfo

A POJO to keep track of tag info:

public class TagInfo {
    private String tagName;
    private String style;
    private int tagType;
    // standard getters and setters

7.5 HTML to RichText

This is not a small class, so let’s break it down by method.

Essentially, we are surrounding any arbitrary HTML with a div tag, so we know what we are looking for. Then we look for all elements within the div tag, add each to an ArrayList of RichTextDetails , and then pass the whole ArrayList to the mergeTextDetails method. mergeTextDetails returns RichtextString, which is what we need to set a cell value:

   public RichTextString fromHtmlToCellValue(String html, Workbook workBook){
       Config.IsHTMLEmptyElementTagRecognised = true;
       
       Matcher m = HEAVY_REGEX.matcher(html);
       String replacedhtml =  m.replaceAll("");
       StringBuilder sb = new StringBuilder();
       sb.insert(0, "<div>");
       sb.append(replacedhtml);
       sb.append("</div>");
       String newhtml = sb.toString();
       Source source = new Source(newhtml);
       List<RichTextDetails> cellValues = new ArrayList<RichTextDetails>();
       for(Element el : source.getAllElements("div")){
           cellValues.add(createCellValue(el.toString(), workBook));
       }
       RichTextString cellValue = mergeTextDetails(cellValues);

       
       return cellValue;
   }

As we saw above, we pass an ArrayList of RichTextDetails in this method. Jericho has a setting that takes boolean value to recognize empty tag elements such as
: Config.IsHTMLEmptyElementTagRecognised. This can be important when dealing with online rich text editors, so we set this to true. Because we need to keep track of the order of the elements, we use a LinkedHashMap instead of a HashMap.

    private static RichTextString mergeTextDetails(List<RichTextDetails> cellValues) {
        Config.IsHTMLEmptyElementTagRecognised = true;
        StringBuilder textBuffer = new StringBuilder();
        Map<Integer, Font> mergedMap = new LinkedHashMap<Integer, Font>(550, .95f);
        int currentIndex = 0;
        for (RichTextDetails richTextDetail : cellValues) {
            //textBuffer.append(BULLET_CHARACTER + " ");
            currentIndex = textBuffer.length();
            for (Entry<Integer, Font> entry : richTextDetail.getFontMap()
                .entrySet()) {
                mergedMap.put(entry.getKey() + currentIndex, entry.getValue());
            }
            textBuffer.append(richTextDetail.getRichText())
                .append(NEW_LINE);
        }

        RichTextString richText = new XSSFRichTextString(textBuffer.toString());
        for (int i = 0; i < textBuffer.length(); i++) {
            Font currentFont = mergedMap.get(i);
            if (currentFont != null) {
                richText.applyFont(i, i + 1, currentFont);
            }
        }
        return richText;
    }

As mentioned above, we are using Java 9 in order to use StringBuilder with the java.util.regex.Matcher.appendReplacement. Why? Well that’s because StringBuffer slower than StringBuilder for operations. StringBuffer functions are synchronized for thread safety and thus slower.

We are using Deque instead of Stack because a more complete and consistent set of LIFO stack operations is provided by the Deque interface:

    static RichTextDetails createCellValue(String html, Workbook workBook) {
        Config.IsHTMLEmptyElementTagRecognised  = true;
        Source source = new Source(html);
        Map<String, TagInfo> tagMap = new LinkedHashMap<String, TagInfo>(550, .95f);
        for (Element e : source.getChildElements()) {
            getInfo(e, tagMap);
        }

        StringBuilder sbPatt = new StringBuilder();
        sbPatt.append("(").append(StringUtils.join(tagMap.keySet(), "|")).append(")");
        String patternString = sbPatt.toString();
        Pattern pattern = Pattern.compile(patternString);
        Matcher matcher = pattern.matcher(html);

        StringBuilder textBuffer = new StringBuilder();
        List<RichTextInfo> textInfos = new ArrayList<RichTextInfo>();
        ArrayDeque<RichTextInfo> richTextBuffer = new ArrayDeque<RichTextInfo>();
        while (matcher.find()) {
            matcher.appendReplacement(textBuffer, "");
            TagInfo currentTag = tagMap.get(matcher.group(1));
            if (START_TAG == currentTag.getTagType()) {
                richTextBuffer.push(getRichTextInfo(currentTag, textBuffer.length(), workBook));
            } else {
                if (!richTextBuffer.isEmpty()) {
                    RichTextInfo info = richTextBuffer.pop();
                    if (info != null) {
                        info.setEndIndex(textBuffer.length());
                        textInfos.add(info);
                    }
                }
            }
        }
        matcher.appendTail(textBuffer);
        Map<Integer, Font> fontMap = buildFontMap(textInfos, workBook);

        return new RichTextDetails(textBuffer.toString(), fontMap);
    }

We can see where RichTextInfo comes in to use here:

    private static Map<Integer, Font> buildFontMap(List<RichTextInfo> textInfos, Workbook workBook) {
        Map<Integer, Font> fontMap = new LinkedHashMap<Integer, Font>(550, .95f);

        for (RichTextInfo richTextInfo : textInfos) {
            if (richTextInfo.isValid()) {
                for (int i = richTextInfo.getStartIndex(); i < richTextInfo.getEndIndex(); i++) {
                    fontMap.put(i, mergeFont(fontMap.get(i), richTextInfo.getFontStyle(), richTextInfo.getFontValue(), workBook));
                }
            }
        }

        return fontMap;
    }

Where we use STYLES enum:

    private static Font mergeFont(Font font, STYLES fontStyle, String fontValue, Workbook workBook) {
        if (font == null) {
            font = workBook.createFont();
        }

        switch (fontStyle) {
        case BOLD:
        case EM:
        case STRONG:
            font.setBoldweight(Font.BOLDWEIGHT_BOLD);
            break;
        case UNDERLINE:
            font.setUnderline(Font.U_SINGLE);
            break;
        case ITALLICS:
            font.setItalic(true);
            break;
        case PRE:
            font.setFontName("Courier New");
        case COLOR:
            if (!isEmpty(fontValue)) {

                font.setColor(IndexedColors.BLACK.getIndex());
            }
            break;
        default:
            break;
        }

        return font;
    }

We are making use of the TagInfo class to track the current tag:

    private static RichTextInfo getRichTextInfo(TagInfo currentTag, int startIndex, Workbook workBook) {
        RichTextInfo info = null;
        switch (STYLES.fromValue(currentTag.getTagName())) {
        case SPAN:
            if (!isEmpty(currentTag.getStyle())) {
                for (String style : currentTag.getStyle()
                    .split(";")) {
                    String[] styleDetails = style.split(":");
                    if (styleDetails != null && styleDetails.length > 1) {
                        if ("COLOR".equalsIgnoreCase(styleDetails[0].trim())) {
                            info = new RichTextInfo(startIndex, -1, STYLES.COLOR, styleDetails[1]);
                        }
                    }
                }
            }
            break;
        default:
            info = new RichTextInfo(startIndex, -1, STYLES.fromValue(currentTag.getTagName()));
            break;
        }
        return info;
    }

We process the HTML tags:

    private static void getInfo(Element e, Map<String, TagInfo> tagMap) {
        tagMap.put(e.getStartTag()
            .toString(),
            new TagInfo(e.getStartTag()
                .getName(), e.getAttributeValue("style"), START_TAG));
        if (e.getChildElements()
            .size() > 0) {
            List<Element> children = e.getChildElements();
            for (Element child : children) {
                getInfo(child, tagMap);
            }
        }
        if (e.getEndTag() != null) {
            tagMap.put(e.getEndTag()
                .toString(),
                new TagInfo(e.getEndTag()
                    .getName(), END_TAG));
        } else {
            // Handling self closing tags
            tagMap.put(e.getStartTag()
                .toString(),
                new TagInfo(e.getStartTag()
                    .getName(), END_TAG));
        }
    }

7.6 Create Worksheet

Using StringBuilder, I create a String that is going to written to FileOutPutStream. In a real application this should be user defined. I appended my folder path and filename on two different lines. Please change the file path to your own.

sheet.createRow(0) creates a row on the very first line and dataRow.createCell(0) creates a cell in column A of the row.

public void createWorkSheet(Workbook wb, String content, String tabName) {
        StringBuilder sbFileName = new StringBuilder();
        sbFileName.append("/Users/mike/javaSTS/michaelcgood-apache-poi-richtext/");
        sbFileName.append("myfile.xlsx");
        String fileMacTest = sbFileName.toString();
        try {
            this.fileOut = new FileOutputStream(fileMacTest);
        } catch (FileNotFoundException ex) {
            Logger.getLogger(CreateWorksheet.class.getName())
                .log(Level.SEVERE, null, ex);
        }

        Sheet sheet = wb.createSheet(tabName); // Create new sheet w/ Tab name

        sheet.setZoom(85); // Set sheet zoom: 85%
        

        // content rich text
        RichTextString contentRich = null;
        if (content != null) {
            contentRich = htmlToExcel.fromHtmlToCellValue(content, wb);
        }


        // begin insertion of values into cells
        Row dataRow = sheet.createRow(0);
        Cell A = dataRow.createCell(0); // Row Number
        A.setCellValue(contentRich);
        sheet.autoSizeColumn(0);
        
        
        try {
            /////////////////////////////////
            // Write the output to a file
            wb.write(fileOut);
            fileOut.close();
        } catch (IOException ex) {
            Logger.getLogger(CreateWorksheet.class.getName())
                .log(Level.SEVERE, null, ex);
        }


    }

8. Demo

We visit localhost:8080.

We input some text with some HTML:
example of input for apache poi richtext app

We open up our excel file and see the RichText we created:
showing richtext in excel from apache poi app

9. Conclusion

We can see it is not trivial to convert HTML to Apache POI’s RichTextString class; however, for business applications converting HTML to RichTextString can be essential because readability is important in Microsoft Excel files. There’s likely room to improve upon the performance of the application we build, but we covered the foundation of building such an application .

The full source code is available on Github.



Intro to Spring Cloud Config Server

1. Overview

In this tutorial, we will review the basics of Spring Cloud Config Server. We will setup a Config Server and then build a client application that consumes the configuration on startup and then refreshes the configuration without restarting. The application we are building is the same “Hello World” application discussed in the Centralized Configuration Getting Started Guide, but we go into more depth about the concepts of Spring Cloud Config Server in this article.

The full source code for the tutorial is on Github.

2. What is Spring Cloud Config Server?

As the documentation succinctly states, “Spring Cloud Config provides server and client-side support for externalized configuration in a distributed system.” The default implementation of the server storage backend uses git, so it supports labelled versions of configuration environments with ease and is accessible to many tools for managing the content.

Spring Cloud Config fits very well into Spring applications because its concepts of both client and server map precisely to the Spring Environment and PropertySource abstractions. However, Spring Cloud Config can be used with any application running in any language.

3. Create a Multi Module Project

The application we are creating will have two modules: one for the Configuration Service and the other for the Configuration client. Because of this, we need to create a parent pom.

3.1 Parent

In our IDE, let’s create a new project. I’m using Spring Tool Suite, but that’s just a personal preference.

In our pom.xml, let’s specify our two modules:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>com.michaelcgood</artifactId>
	<version>0.0.1</version>
	<packaging>pom</packaging>

	<name>michaelcgood-spring-cloud-config-server</name>
	<description>Intro to Spring Cloud Config Server</description>


    <modules>
        <module>mcg-configuration-client</module>
        <module>mcg-configuration-service</module>
    </modules>

</project>

3.2 Configuration Service

In our IDE, let’s create a new Maven module for our configuration service and insert this in our pom:

<?xml version="1.0"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.michaelcgood</groupId>
  <artifactId>mcg-configuration-service</artifactId>
  <version>0.0.1</version>
  <packaging>jar</packaging>
  <name>mcg-configuration-service</name>
 
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.9.RELEASE</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-config-server</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>Edgware.RELEASE</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

3.3 Configuration Client

Now we just need to make a module for our configuration client. So, let’s make another Maven module and insert this into our pom:

<?xml version="1.0"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.michaelcgood</groupId>
  <artifactId>mcg-configuration-client</artifactId>
  <version>0.0.1</version>
  <packaging>jar</packaging>
  	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.9.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-config</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.springframework.cloud</groupId>
				<artifactId>spring-cloud-dependencies</artifactId>
				<version>Edgware.RELEASE</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>
</project>

Our project structure looks like this now:
shows project structure of sample spring cloud config. two modules in project.

4. Config Server

Now we will create a Config Service to act as an intermediary between our client and a git repository.

4.1 Enable Config Server

We use Spring Cloud’s @EnableConfigServer to create a config server that can be communicated with. So, this is just a normal Spring Boot application with one annotation added to enable the Config Server.:

@EnableConfigServer
@SpringBootApplication
public class ConfigServiceApplication {
    
    public static void main(String[] args) {
        SpringApplication.run(ConfigServiceApplication.class, args);
    }
}

4.2 application.properties

To ensure that there is no conflict between ports for our Config Service and client, we specify a different port for the Config Service:

server.port=8888

spring.cloud.config.server.git.uri=${HOME}/Desktop/mcg-config

The second line spring.cloud.config.server.git.uri=${HOME}/Desktop/mcg-config points to a git repository, which we will create next.

4.3 Git

On a *nix system, we can do everything on the command line.
We make a folder on our desktop:

mkdir mcg-config

We create a file named a-bootiful-client.properties using vim:

vim a-bootiful-client.properties

We add the message, “Hello World” but this could be whatever we would like. After we write (:w) we quit (:q) vim.

Now let’s create a new repo:

git init

Now we add the file that contains our message:

git add a-bootiful-client.properties

Let’s commit:

git commit

 5. Configuration Client

Now let’s create a new Spring Boot application that uses the Config Server to load its own configuration and that refreshes its configuration to reflect changes to the Config Server on-demand, without restarting the JVM.

Spring will see the configuration property files just like it would any property file loaded from application.properties, application.yml or any other PropertySource.

5.1 Reflecting Changes

The client may access any value in the Config Server using the standard Spring ways, such as @ConfigurationProperties or @Value(“${…​}”).

With this in mind, we create a REST controller that returns the resolved message property’s value:

@SpringBootApplication
public class ConfigClientApplication {

    public static void main(String[] args) {
        SpringApplication.run(ConfigClientApplication.class, args);
    }
}

@RefreshScope
@RestController
class MessageRestController {

    @Value("${message:Hello default}")
    private String message;

    @RequestMapping("/message")
    String getMessage() {
        return this.message;
    }
}

The default configuration only allows the values to be read on the client’s startup and not again. So, using @RefreshScope we force the bean to refresh its configuration, which means it will pull updated values from the Config Server, and then trigger a refresh event.

5.2 bootstrap.properties

The properties to configure the Config Client must be read in before the rest of the application’s configuration is read from the Config Server, during the bootstrap phase.

We specify the client’s spring.application.name and the location of the Config Server spring.cloud.config.uri:

spring.application.name=a-bootiful-client
spring.cloud.config.uri=http://localhost:8888
management.security.enabled=false

Notice:
We disabled security with our setting management.security.enabled=false to make testing and tinkering easy for us.

6. Demo

First we need to change directory to our configuration service and start it:

mcg-configuration-service mike$  mvn spring-boot:run

And then do the same for our client:

mcg-configuration-client mike$  mvn spring-boot:run

We can see in our terminal for configuration service when the a-bootiful-client.properties is added:

INFO 5921 --- [nio-8888-exec-1] o.s.c.c.s.e.NativeEnvironmentRepository  : Adding property source: file:/var/folders/dk/48l9cm2x3vnfl5ymh6dtxpwc0000gn/T/config-repo-7195892194658362240/a-bootiful-client.properties

Let’s open our browser and visit http://localhost:8080/message. We see “Hello World”.

Now let’s change the message in a-bootiful-client.properties again and this time put, “Hi! :-)”.

After saving and doing a commit, we visit http://localhost:8888/a-bootiful-client/default to confirm our change.

Now we invoke the Spring Boot Actuator refersh endpoint to refresh our client:

curl -X POST http://localhost:8080/refresh

We visit http://localhost:8080/message and see our message “Hi! :-)” is displayed.

For more information on Spring Boot Actuator, see the tutorial Building Spring Boot RESTful Service + Spring Boot Actuator.

7. Conclusion

We just completed centralizing configuration of our services in Spring. We accomplished this by standing up a Spring Cloud Config Server and creating a client to consume the configuration on startup and then refresh the configuration without restarting.

Many other things can be done with Spring Cloud Config Server that we did not touch on, such as:

  • Have the Config Server register with the Discovery Service for Spring Cloud Netflix, Eureka Service Discovery or Spring Cloud Consul
  • Serve configuration in YAML or Properties format
  • Serve plain text configuration files
  • Embed the config server in an application

The full source code can be found on Github.

Intro to Redis with Spring Boot

1. Overview

In this article, we will review the basics of how to use Redis with Spring Boot through the Spring Data Redis library.

We will build an application that demonstrates how to perform CRUD operations Redis through a web interface. The full source code for this project is available on Github.

2. What is Redis?

Redis is an open source, in-memory key-value data store, used as a database, cache and message broker. In terms of implementation, Key Value stores represent one of the largest and oldest members in the NoSQL space. Redis supports data structures such as strings, hashes, lists, sets, and sorted sets with range queries.

The Spring Data Redis  framework makes it easy to write Spring applications that use the Redis key value store by providing an abstraction to the data store.

3. Setting Up A Redis Server

The server is available for free at http://redis.io/download.

If you use a Mac, you can install it with homebrew:

brew install redis

Then start the server:

mikes-MacBook-Air:~ mike$ redis-server
10699:C 23 Nov 08:35:58.306 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
10699:C 23 Nov 08:35:58.307 # Redis version=4.0.2, bits=64, commit=00000000, modified=0, pid=10699, just started
10699:C 23 Nov 08:35:58.307 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
10699:M 23 Nov 08:35:58.309 * Increased maximum number of open files to 10032 (it was originally set to 256).
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 4.0.2 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 10699
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

10699:M 23 Nov 08:35:58.312 # Server initialized
10699:M 23 Nov 08:35:58.312 * Ready to accept connections

 

4. Maven Dependencies

Let’s declare the necessary dependencies in our pom.xml for the example application we are building:

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-redis</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-thymeleaf</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>



5. Redis Configuration

We need to connect our application with the Redis server. To establish this connection, we are using Jedis, a Redis client implementation.

5.1 Config

Let’s start with the configuration bean definitions:

    @Bean
    JedisConnectionFactory jedisConnectionFactory() {
        return new JedisConnectionFactory();
    }

    @Bean
    public RedisTemplate<String, Object> redisTemplate() {
        final RedisTemplate<String, Object> template = new RedisTemplate<String, Object>();
        template.setConnectionFactory(jedisConnectionFactory());
        template.setValueSerializer(new GenericToStringSerializer<Object>(Object.class));
        return template;
    }

The JedisConnectionFactory is made into a bean so we can create a RedisTemplate to query data.

5.2 Message Publisher

Following the principles of SOLID, we create a MessagePublisher interface:

public interface MessagePublisher {

    void publish(final String message);
}

We implement the MessagePublisher interface to use the high-level RedisTemplate to publish the message since the RedisTemplate allows arbitrary objects to be passed in as messages:

@Service
public class MessagePublisherImpl implements MessagePublisher {
    
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    @Autowired
    private ChannelTopic topic;

    public MessagePublisherImpl() {
    }

    public MessagePublisherImpl(final RedisTemplate<String, Object> redisTemplate, final ChannelTopic topic) {
        this.redisTemplate = redisTemplate;
        this.topic = topic;
    }

    public void publish(final String message) {
        redisTemplate.convertAndSend(topic.getTopic(), message);
    }

}

We also define this as a bean in RedisConfig:

    @Bean
    MessagePublisher redisPublisher() {
        return new MessagePublisherImpl(redisTemplate(), topic());
    }

Message Listener

In order to subscribe to messages, we need to implement the MessageListener interface: each time a new message arrives, a callback gets invoked and the user code executed through a method named onMessage. This interface gives access to the message, the channel it has been received through, and any pattern used by the subscription to match the channel.

Thus, we create a service class to implement MessageSubscriber:

@Service
public class MessageSubscriber implements MessageListener {
    
    public static List<String> messageList = new ArrayList<String>();

    public void onMessage(final Message message, final byte[] pattern) {
        messageList.add(message.toString());
        System.out.println("Message received: " + new String(message.getBody()));
    }

}

We add a bean definition to RedisConfig:

    @Bean
    MessageListenerAdapter messageListener() {
        return new MessageListenerAdapter(new MessageSubscriber());
    }

6. RedisRepository

Now that we have configured the application to interact with the Redis server, we are going to prepare the application to take example data.

6.1 Model

For this example, we defining a Movie model with two fields:

private String id;
private String name;
//standard getters and setters

6.2 Repository interface

Unlike other Spring Data projects, Spring Data Redis does offer any features to build on top of the other Spring Data interfaces. This is odd for us who have experience with the other Spring Data projects.

Often there is no need to write an implementation of a repository interface with Spring Data projects. We simply just interact with the interface. Spring Data JPA provides numerous repository interfaces that can be extended to get features such as CRUD operations, derived queries, and paging.

So, unfortunately, we need to write our own interface and then define the methods:

public interface RedisRepository {

    Map<Object, Object> findAllMovies();

    void add(Movie movie);

    void delete(String id);

    Movie findMovie(String id);
    
}

6.3 Repository implementation

Our implementation class uses the redisTemplate defined in our configuration class RedisConfig.

We use the HashOperations template that Spring Data Redis offers:

@Repository
public class RedisRepositoryImpl implements RedisRepository {
    private static final String KEY = "Movie";
    
    private RedisTemplate<String, Object> redisTemplate;
    private HashOperations hashOperations;
    
    @Autowired
    public RedisRepositoryImpl(RedisTemplate<String, Object> redisTemplate){
        this.redisTemplate = redisTemplate;
    }

    @PostConstruct
    private void init(){
        hashOperations = redisTemplate.opsForHash();
    }
    
    public void add(final Movie movie) {
        hashOperations.put(KEY, movie.getId(), movie.getName());
    }

    public void delete(final String id) {
        hashOperations.delete(KEY, id);
    }
    
    public Movie findMovie(final String id){
        return (Movie) hashOperations.get(KEY, id);
    }
    
    public Map<Object, Object> findAllMovies(){
        return hashOperations.entries(KEY);
    }

  
}

Let’s take note of the init() method. In this method, we use a function named opsForHash(), which returns the operations performed on hash values bound to the given key. We then use the hashOps, which was defined in init(), for all our CRUD operations.

7. Web interface

In this section, we will review adding Redis CRUD operations capabilities to a web interface.

7.1 Add A Movie

We want to be able to add a Movie in our web page. The Key is the is the Movie id and the Value is the actual object. However, we will later address this so only the Movie name is shown as the value.

So, let’s add a form to a HTML document and assign appropriate names and ids :

      
<form id="addForm">
<div class="form-group">
                    <label for="keyInput">Movie ID (key)</label>
                    <input name="keyInput" id="keyInput" class="form-control"/>
                </div>
<div class="form-group">
                    <label for="valueInput">Movie Name (field of Movie object value)</label>
                    <input name="valueInput" id="valueInput" class="form-control"/>
                </div>
                <button class="btn btn-default" id="addButton">Add</button>
            </form>

Now we use JavaScript to persist the values on form submission:

$(document).ready(function() {
    var keyInput = $('#keyInput'),
        valueInput = $('#valueInput');

    refreshTable();
    $('#addForm').on('submit', function(event) {
        var data = {
            key: keyInput.val(),
            value: valueInput.val()
        };

        $.post('/add', data, function() {
            refreshTable();
            keyInput.val('');
            valueInput.val('');
            keyInput.focus();
        });
        event.preventDefault();
    });

    keyInput.focus();
});

We assign the @RequestMapping value for the POST request, request the key and value, create a Movie object, and save it to the repository:

    @RequestMapping(value = "/add", method = RequestMethod.POST)
    public ResponseEntity<String> add(
        @RequestParam String key,
        @RequestParam String value) {
        
        Movie movie = new Movie(key, value);
        
        redisRepository.add(movie);
        return new ResponseEntity<>(HttpStatus.OK);
    }

7.2 Viewing the content

Once a Movie object is added, we refresh the table to display an updated table. In our JavaScript code block for section 7.1, we called a JavaScript function called refreshTable(). This function performs a GET request to retrieve the current data in the repository:

function refreshTable() {
    $.get('/values', function(data) {
        var attr,
            mainTable = $('#mainTable tbody');
        mainTable.empty();
        for (attr in data) {
            if (data.hasOwnProperty(attr)) {
                mainTable.append(row(attr, data[attr]));
            }
        }
    });
}

The GET request is processed by a method named findAll() that retrieves all the Movie objects stored in the repository and then converts the datatype from Map<Object, Object> to Map<String, String>:

    @RequestMapping("/values")
    public @ResponseBody Map<String, String> findAll() {
        Map<Object, Object> aa = redisRepository.findAllMovies();
        Map<String, String> map = new HashMap<String, String>();
        for(Map.Entry<Object, Object> entry : aa.entrySet()){
            String key = (String) entry.getKey();
            map.put(key, aa.get(key).toString());
        }
        return map;
    }

7.3 Delete a Movie

We write Javascript to do a POST request to /delete, refresh the table, and set keyboard focus to key input:

function deleteKey(key) {
    $.post('/delete', {key: key}, function() {
        refreshTable();
        $('#keyInput').focus();
    });
}

We request the key and delete the object in the redisRepository based on this key:

    @RequestMapping(value = "/delete", method = RequestMethod.POST)
    public ResponseEntity<String> delete(@RequestParam String key) {
        redisRepository.delete(key);
        return new ResponseEntity<>(HttpStatus.OK);
    }

8. Demo

Here we added two movies:
added key value to redis spring boot application

Here we removed one movie:
delete key-value in redis spring boot

9. Conclusion

In this tutorial, we introduced Spring Data Redis and one way of connecting it to a web application to perform CRUD operations.

The source code for the example application is on Github.


AJAX with CKEditor in Spring Boot

1. Overview

In this article, we will cover how to use CKEditor with Spring Boot. In this tutorial, we will be importing an XML document with numerous data, program the ability to load a set of data to the CKEditor instance with a GET request, and do a POST request to save the CKEditor’s data.

Technologies we will be using include MongoDB, Thymeleaf, and Spring Batch.

The full source code for this tutorial is available on Github.

2. What is CKEditor?

CKEditor is a browser-based What-You-See-Is-What-You-Get (WYSIWYG) content editor.  CKEditor aims to bring to a web interface common word processor features found in desktop editing applications like Microsoft Word and OpenOffice.

CKEditor has numerous features for end users in regards to the user interface, inserting content, authoring content, and more.

There are different versions of CKEditor, but for this tutorial we are using CKEditor 4. To see a demo, visit: https://ckeditor.com/ckeditor-4/

3. The XML Document

As mentioned, we are uploading an XML document in this application. The XML data will be inserted into the database and used for the rest of the tutorial.

<?xml version="1.0"?>
<Music xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" id="MUS-1" style="1.1">
<status date="2017-11-07">draft</status>
 <title xmlns:xhtml="http://www.w3.org/1999/xhtml" >Guide to Music I Like - No Specific Genre</title>
  <description xmlns:xhtml="http://www.w3.org/1999/xhtml" >This guide presents a catalog of music that can be found on Spotify. 
    <html:br xmlns:html="http://www.w3.org/1999/xhtml"/>
    <html:br xmlns:html="http://www.w3.org/1999/xhtml"/>
    This is a very small sample of music found on Spotify and is no way to be considered comprehensive.
    </description>
    <songs>
    <song>
    <artist>
    Run the Jewels
    </artist>
    <song-title>Legend Has It</song-title>
    </song>
    <song>
    <artist>
    Kendrick Lamar
    </artist>
    <song-title>ELEMENT.</song-title>
    </song>
    <song>
    <artist>
    Weird Al Yankovic
    </artist>
    <song-title>NOW That's What I Call Polka!</song-title>
    </song>
    <song>
    <artist>
    Eiffel 65
    </artist>
    <song-title>Blue (Da Ba Dee) - DJ Ponte Ice Pop Radio</song-title>
    </song>
    <song>
    <artist>
    YTCracker
    </artist>
    <song-title>Hacker Music</song-title>
    </song>
    <song>
    <artist>
    MAN WITH A MISSION
    </artist>
    <song-title>
    Raise Your Flag
    </song-title>
    </song>
    <song>
    <artist>
    GZA, Method Man
    </artist>
    <song-title>
    Shadowboxin'
    </song-title>
    </song>
    </songs>
</Music>

4. Model

For the above XML code, we can model a Song like this:

public class SongModel {
    @Id
    private String id;
    @Indexed
    private String artist;
    @Indexed
    private String songTitle;
    @Indexed
    private Boolean updated;
    
    public Boolean getUpdated() {
        return updated;
    }
    public void setUpdated(Boolean updated) {
        this.updated = updated;
    }
    public String getArtist() {
        return artist;
    }
    public void setArtist(String artist) {
        this.artist = artist;
    }
    public String getSongTitle() {
        return songTitle;
    }
    public void setSongTitle(String songTitle) {
        this.songTitle = songTitle;
    }
    
    public String getId() {
        return id;
    }
    public void setId(String id) {
        this.id = id;
    }
    
    @JsonCreator
    public SongModel(
        @JsonProperty("artist") String artist,
        @JsonProperty("song-title") String songTitle){
        this.artist = artist;
        this.songTitle = songTitle;
    }
 
    @Override
    public String toString() {
      return "Person [id=" + id + ", artist=" + artist + ", song-title=" + songTitle + "]";
    }

}

For our application, we will be differentiating between an unmodified song and a song that has been modified in CKEditor with a separate Model and Repository.

Let’s now define what an Updated Song is:

public class UpdatedSong {
    
    @Id
    private String id;
    @Indexed
    private String artist;
    @Indexed
    private String songTitle;
    @Indexed
    private String html;
    @Indexed
    private String sid;
    
    public String getSid() {
        return sid;
    }
    public void setSid(String sid) {
        this.sid = sid;
    }
    
    public String getId() {
        return id;
    }
    public void setId(String id) {
        this.id = id;
    }
    public String getArtist() {
        return artist;
    }
    public void setArtist(String artist) {
        this.artist = artist;
    }
    public String getSongTitle() {
        return songTitle;
    }
    public void setSongTitle(String songTitle) {
        this.songTitle = songTitle;
    }
    public String getHtml() {
        return html;
    }
    public void setHtml(String html) {
        this.html = html;
    }

}

5. File Upload and Processing

As this article’s focus is on CKEditor and AJAX, we aren’t going to go into detail on the file upload and processing with Spring Batch. We have reviewed this process in depth in these prior posts, however:

6. Setting Data to CKEditor – GET Request

It is our goal to retrieve the data for an individual song and display that data in CKEditor. There are two issues to tackle: retrieving the data for an individual song and displaying it in CKEditor.

6.1 Client Side Code

In view.html, we use a table in Thymeleaf to iterate through each Song in the Song repository. To be able to retrieve the data from the server for a specific song, we pass in the Song’s id to a function.

Here’s the snippet of code that is responsible for calling the function that retrieves the data from the server and subsequently sets the data to the CKEditor instance:

<table class="table datatable">
<thead>
<tr>
<th>Artist</th>
<th>Song Title</th>
<th>Load</th>
</tr>
</thead>
<tbody>
<tr th:each="songList : ${songList}">
<td th:text="${songList.artist}">Text ...</td>
<td th:text="${songList.songTitle}">Text ...</td>
<td><button th:onclick="|getSong('${songList.id}')|" id="button" class="btn btn-primary btn-condensed">
<i class="glyphicon glyphicon-folder-open"></i>
</button></td>
</tr>
</tbody>
</table>

As we can see, the id of Song is essential for us to be able to retrieve the data.

In the getSong function, we use a deferred promise to ensure that data is set after the GET request:

		function getSong(song) {
			$.ajax({
				url : "/api/show/?sid=" + song,
				type : 'GET',
				dataType : 'text'
			}).then(function(data) {
				var length = data.length-2;
				var datacut = data.slice(9,length);
				CKEDITOR.instances.content.setData(datacut);

			});

			$("#form").attr("action", "/api/save/?sid=" + song);

		};

6.2 Server Side Code

getSong accepts a parameter named sid, which stands for Song id. sid is also a path variable in the @GetMapping. We treat sid as a String because this is the id of the Song from MongoDB.

We check if the Song has been modified and if so we retrieve the associated UpdatedSong entity. If not, we treat the Song differently. Ultimately, we return a simple POJO with a String for data named ResponseModel, however:

    @GetMapping(value={"/show/","/show/{sid}"})
    public ResponseEntity<?> getSong(@RequestParam String sid, Model model){
        ResponseModel response = new ResponseModel();
        System.out.println("SID :::::" + sid);
        ArrayList<String> musicText = new ArrayList<String>();
        if(sid!=null){
            String sidString = sid;
            SongModel songModel = songDAO.findOne(sidString);
            System.out.println("get status of boolean during get ::::::" + songModel.getUpdated());
            if(songModel.getUpdated()==false ){
                
                musicText.add(songModel.getArtist());
                musicText.add(songModel.getSongTitle());
                String filterText = format.changeJsonToHTML(musicText);
                response.setData(filterText);
                
            } else if(songModel.getUpdated()==true){
                UpdatedSong updated = updatedDAO.findBysid(sidString);
                String text = updated.getHtml();
                System.out.println("getting the updated text ::::::::" + text);
                response.setData(text);
            }
            
        }

        model.addAttribute("response", response);
        
        return ResponseEntity.ok(response);
    }

ResponseModel is a very simple POJO as mentioned:

public class ResponseModel {
    private String data;
    
    public ResponseModel(){
            
    }
    
    public ResponseModel(String data){
            this.data = data;
    }

    public String getData() {
            return data;
    }

    public void setData(String data) {
            this.data = data;
    }
}

 

7. Saving CKEditor Data – POST Request

POSTing the data isn’t much of a challenge; however, ensuring that the data is handled appropriately can be.

7.1 Client Side Code

As the CKEditor instance is a textarea within a form, we can trigger a function on a form submit:

$(document)
.ready(
function() {

// SUBMIT FORM
$("#form").submit(function(event) {
// Prevent the form from submitting via the browser.
event.preventDefault();
ajaxPost();
});

ajaxPost() retrieves the current data in the CKEditor and sets it to the variable formData:

function ajaxPost() {

// PREPARE FORM DATA
var formData = CKEDITOR.instances.content
.getData();

// DO POST
$
.ajax({
type : "POST",
contentType : "text/html",
url : $("#form").attr("action"),
data : formData,
dataType : 'text',
success : function(result) {

$("#postResultDiv")
.html(
"

"
+ "Post Successfully! "
+ "

");

console.log(result);
},
error : function(e) {
alert("Error!")
console.log("ERROR: ", e);
}
});

}

})

It is important to note:

  • contentType is “text/html”
  • dataType is “text”

Having the incorrect contentType or dataType can lead to errors or malformed data.

7.2 Server Side Code

We stated in our contentType for the POST request that the mediatype is “text/html”.  We need to specify in our mapping that this will be consumed. Therefore, we add consumes = MediaType.TEXT_HTML_VALUE with our @PostMapping.

Areas for us to note include:

  • @RequestBody String body is responsible for setting the variable body to the content of our request
  • We once again return ResponseModel, the simple POJO described earlier that contains our data
  • We treat a previously modified SongModel different than one that has not been modified before

Also, like the GET request, the sid allows us to deal with the correct Song:

 @PostMapping(value={"/save/","/save/[sid]"}, consumes = MediaType.TEXT_HTML_VALUE)
    public @ResponseBody ResponseModel saveSong( @RequestBody String body, @RequestParam String sid){
        ResponseModel response = new ResponseModel();
        response.setData(body);
        SongModel oldSong = songDAO.findOne(sid);
        String songTitle = oldSong.getSongTitle();
        String artistName = oldSong.getArtist();
        if(oldSong.getUpdated() == false){
            UpdatedSong updatedSong = new UpdatedSong();
            updatedSong.setArtist(artistName);
            updatedSong.setSongTitle(songTitle);
            updatedSong.setHtml(body);
            updatedSong.setSid(sid);
            oldSong.setUpdated(true);
            songDAO.save(oldSong);
            updatedDAO.insert(updatedSong);
            System.out.println("get status of boolean during post :::::" + oldSong.getUpdated());
        }else{
            UpdatedSong currentSong = updatedDAO.findBysid(sid);
            currentSong.setHtml(body);
            updatedDAO.save(currentSong);
        }        
        
        return response;
    }

8. Demo

We visit localhost:8080:

home page for example ckeditor spring boot application

We upload the provided music-example.xml file:

Screen shown after upload for example ckeditor application

We click “Load” for a song:

loaded content into ckeditor

We add content and click “Save”:

we add content and click save ckeditor

If you return to the saved content, you may see “\n”for line breaks. For the time being, discussing this is out of the scope for the tutorial.

9. Conclusion

In this tutorial, we covered how to load  data using a GET request with the object’s id, set the data to the CKEditor instance, and save the CKEditor’s data back to the database with a POST request. There’s extra code, such as using two different entities for the data (an original and a modified version), that isn’t necessary, but hopefully is instructive.

The full code can be found on Github.

Converting XML to JSON + Raw Use in MongoDB + Spring Batch

Overview

Why convert XML to JSON for raw use in MongoDB?

Since MongoDB uses JSON documents in order to store records, just as tables and rows store records in a relational database, we naturally need to convert our XML to JSON.

Some applications may need to store raw (unmodified) JSON because there is uncertainty in how the data will be structured.

There are hundreds of XML-based standards. If an application is to process XML files that do not follow the same standard, there is uncertainty in how the data will be structured.

Why use Spring Batch?

Spring Batch provides reusable functions that are essential in processing large volumes of records and other features that enable high-volume and high performance batch jobs. The Spring Website has documented Spring Batch well.

For another tutorial on Spring Batch, see my previous post on Processing CSVs with Spring Batch.

0 – Converting XML to JSON For Use In MongoDB With Spring Batch Example Application

The example application converts an XML document that is a “policy” for configuring a music playlist. This policy is intended to resemble real cyber security configuration documents. It is a short document but illustrates how you will search complex XML documents.

The approach we will be taking our tutorial is for handling XML files of varying style. We want to be able to handle the unexpected. This is why we are keeping the data “raw.”

View and Download the code from Github

1 – Project Structure

It is a typical Maven structure. We have one package for this example application. The XML file is in src/main/resources.
Project structure of converting xml to json with spring batch mongodb

2 – Project Dependencies

Besides our typical Spring Boot dependencies, we include dependencies for an embedded MongoDB database and for processing JSON.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>michaelcgood-spring-batch-mongodb</artifactId>
	<version>0.0.1</version>
	<packaging>jar</packaging>

	<name>michaelcgood-spring-batch-mongodb</name>
	<description>Michael C  Good - XML to JSON + MongoDB + Spring Batch Example</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.7.RELEASE</version>
		<relativePath /> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-batch</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>de.flapdoodle.embed</groupId>
			<artifactId>de.flapdoodle.embed.mongo</artifactId>
			<version>1.50.5</version>
		</dependency>
		<dependency>
			<groupId>cz.jirutka.spring</groupId>
			<artifactId>embedmongo-spring</artifactId>
			<version>RELEASE</version>
		</dependency>
		<dependency>
				<groupId>org.json</groupId>
				<artifactId>json</artifactId>
				<version>20170516</version>
			</dependency>

			<dependency>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-starter-data-mongodb</artifactId>
			</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>


</project>

3 – XML Document

This is the example policy document created for this tutorial. It’s structure is based on real cyber security policy documents.

  • Note the parent of the document is the Policy tag.
  • Important information lies within the Group tag.
  • Look at the values that reside within the tags, such as the id in Policy or the date within status.

There’s a lot of information condensed in this small document to consider. For instance, there is also the XML namespace (xmlns). We won’t touch on this in the rest of the tutorial, but depending on your goals it could be something to add logic for.

<?xml version="1.0"?>
<Policy  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" style="STY_1.1" id="NRD-1">
  <status date="2017-10-18">draft</status>
  <title xmlns:xhtml="http://www.w3.org/1999/xhtml">Guide to the Configuration of Music Playlist</title>
   <description xmlns:xhtml="http://www.w3.org/1999/xhtml" >This guide presents a catalog of relevant
    configuration settings for a playlist that I listen to while I work on software development.
    <html:br xmlns:html="http://www.w3.org/1999/xhtml"/>
    <html:br xmlns:html="http://www.w3.org/1999/xhtml"/>
    Providing myself with such guidance reminds me how to efficiently
    configure my playlist.  Lorem ipsum <html:i xmlns:html="http://www.w3.org/1999/xhtml">Lorem ipsum,</html:i> 
    and Lorem ipsum.  Some example
    <html:i xmlns:html="http://www.w3.org/1999/xhtml">Lorem ipsum</html:i>, which are Lorem ipsum.
  </description>
  <Group id="remediation_functions">
    <title xmlns:xhtml="http://www.w3.org/1999/xhtml" >Remediation functions used by the SCAP Security Guide Project</title>
    <description xmlns:xhtml="http://www.w3.org/1999/xhtml" >XCCDF form of the various remediation functions as used by
      remediation scripts from the SCAP Security Guide Project</description>
    <Value id="is_the_music_good" prohibitChanges="true" >
      <title xmlns:xhtml="http://www.w3.org/1999/xhtml" >Remediation function to fix bad playlist</title>
      <description xmlns:xhtml="http://www.w3.org/1999/xhtml" >Function to fix bad playlist.
      
        
       Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
       
       Lorem ipsum
       Lorem ipsum
       Lorem ipsum
       Lorem ipsum
      </description>
      <value>
        function fix_bad_playlist {
        
        # Load function arguments into local variables
       Lorem ipsum
       Lorem ipsum
       Lorem ipsum
        
        # Check sanity of the input
        if [ $# Lorem ipsum ]
        then
        echo "Usage: Lorem ipsum"
        echo "Aborting."
        exit 1
        fi
        
        }
      </value>
    </Value>
    </Group>
    </Policy>

4 – MongoDB Configuration

Below we specify that we are using an embedded MongoDB database, make it discoverable for a component scan that is bundled in the convenience annotation @SpringBootApplication, and specify that mongoTemplate will be a bean.

package com.michaelcgood;

import java.io.IOException;
import cz.jirutka.spring.embedmongo.EmbeddedMongoFactoryBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.*;
import com.mongodb.MongoClient;
 
 
@Configuration
public class MongoConfig {
 
    private static final String MONGO_DB_URL = "localhost";
    private static final String MONGO_DB_NAME = "embeded_db";
    @Bean
    public MongoTemplate mongoTemplate() throws IOException {
        EmbeddedMongoFactoryBean mongo = new EmbeddedMongoFactoryBean();
        mongo.setBindIp(MONGO_DB_URL);
        MongoClient mongoClient = mongo.getObject();
        MongoTemplate mongoTemplate = new MongoTemplate(mongoClient, MONGO_DB_NAME);
        return mongoTemplate;
    }
}



5 – Processing XML to JSON

step1() of our Spring Batch Job contains calls three methods to help process the XML to JSON. We will review each individually.

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .tasklet(new Tasklet() {
                    @Override
                    public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
                        
                        // get path of file in src/main/resources
                        Path xmlDocPath =  Paths.get(getFilePath());
                        
                        // process the file to json
                         String json = processXML2JSON(xmlDocPath);
                         
                         // insert json into mongodb
                         insertToMongo(json);
                        return RepeatStatus.FINISHED;
                    }
                }).build();
    }

5.1 – getFilePath()

This method simply gets the file path that is passed as a parameter to the method processXML2JSON.
Note:

  • ClassLoader is helping us locate the XML file in our resources folder.
 // no parameter method for creating the path to our xml file
    private String getFilePath(){
        
        String fileName = "FakePolicy.xml";
        ClassLoader classLoader = getClass().getClassLoader();
        File file = new File(classLoader.getResource(fileName).getFile());
        String xmlFilePath = file.getAbsolutePath();
        
        return xmlFilePath;
    }

5.2 – processXML2JSON(xmlDocPath)

The string returned by getFilePath is passed into this method as a parameter. A JSONOBject is created from a String of the XML file.

   // takes a parameter of xml path and returns json as a string
    private String processXML2JSON(Path xmlDocPath) throws JSONException {
        
        
        String XML_STRING = null;
        try {
            XML_STRING = Files.lines(xmlDocPath).collect(Collectors.joining("\n"));
        } catch (IOException e) {
            e.printStackTrace();
        }
        
        JSONObject xmlJSONObj = XML.toJSONObject(XML_STRING);
        String jsonPrettyPrintString = xmlJSONObj.toString(PRETTY_PRINT_INDENT_FACTOR);
        System.out.println("PRINTING STRING :::::::::::::::::::::" + jsonPrettyPrintString);
        
        return jsonPrettyPrintString;
    }

5.3 – insertToMongo(json)

We insert the parsed JSON into a MongoDB document. We then insert this document with the help of the @Autowired mongoTemplate into a collection named “foo”.

   // inserts to our mongodb
    private void insertToMongo(String jsonString){
        Document doc = Document.parse(jsonString);
        mongoTemplate.insert(doc, "foo");
    }

6 – Querying MongoDB

step2() of our Spring Batch Job contains our MongoDB queries.

  • mongoTemplate.collectionExists returns a Boolean value based on the existence of the collection.
  • mongoTemplate.getCollection(“foo”).find() returns all the documents within the collection.
  • alldocs.toArray() returns an array of DBObjects.
  • Then we call three methods that we will review individually below.
 public Step step2(){
        return stepBuilderFactory.get("step2")
            .tasklet(new Tasklet(){
            @Override
            public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception{
                // all printing out to console removed for post's brevity
                // checks if our collection exists
                Boolean doesexist = mongoTemplate.collectionExists("foo");
                
                // show all DBObjects in foo collection
                DBCursor alldocs = mongoTemplate.getCollection("foo").find();
                List<DBObject> dbarray = alldocs.toArray();
                
                // execute the three methods we defined for querying the foo collection
                String result = doCollect();
                String resultTwo = doCollectTwo();
                String resultThree = doCollectThree();
               
                return RepeatStatus.FINISHED;
            }
        }).build();
    }

6.1 – First query

The goal of this query is to find a document where style=”STY_1.1″. To accomplish this, we need to remember where style resides in the document. It is a child of Policy; therefore, we address it in the criteria as Policy.style.

The other goal of this query is to only return the id field of the Policy. It is also just a child of Policy.

The result is returned by calling this method: mongoTemplate.findOne(query, String.class, “foo”);. The output is a String, so the second parameter is String.class. The third parameter is our collection name.

    public String doCollect(){
        Query query = new Query();
        query.addCriteria(Criteria.where("Policy.style").is("STY_1.1")).fields().include("Policy.id");
        String result = mongoTemplate.findOne(query, String.class, "foo");
        return result;
    }

6.2 – Second query

The difference between the second query and the first query are the fields returned. In the second query, we return Value, which is a child of both Policy and Group.

    public String doCollectTwo(){
        Query query = new Query();
        query.addCriteria(Criteria.where("Policy.style").is("STY_1.1")).fields().include("Policy.Group.Value");
        String result = mongoTemplate.findOne(query, String.class, "foo");
        
        return result;
    }

6.3 – Third query

The criteria for the third query is different. We only want to return a document with the id “NRD-1” and a status date of “2017-10-18”. We only want to return two fields: title and description, which are both children of Value.

Referring to the XML document or the printed JSON in the demo below for further clarification on the queries.

    public String doCollectThree(){
        Query query = new Query();
        query.addCriteria(Criteria.where("Policy.id").is("NRD-1").and("Policy.status.date").is("2017-10-18")).fields().include("Policy.Group.Value.title").include("Policy.Group.Value.description");
        String result = mongoTemplate.findOne(query, String.class, "foo");
        
        return result;
    }

7 – Spring Batch Job

The Job begins with step1 and calls step2 next.

 @Bean
    public Job xmlToJsonToMongo() {
        return jobBuilderFactory.get("XML_Processor")
                .start(step1())
                .next(step2())
                .build();
    }

8 – @SpringBootApplication

This is a standard class with static void main and @SpringBootApplication.

package com.michaelcgood;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;

@SpringBootApplication
@EnableAutoConfiguration(exclude={DataSourceAutoConfiguration.class})
public class SpringBatchMongodb {

	public static void main(String[] args) {
		SpringApplication.run(SpringBatchMongodb.class, args);
	}
}

9 – Demo

9.1 – step1

The JSON is printed as a String. I have cut the output past description below because it is long.

Executing step: [step1]
PRINTING STRING :::::::::::::::::::::{"Policy": {
    "Group": {
        "Value": {
            "prohibitChanges": true,
            "description": {

9.2 – step2

I have cut the results to format the output for the blog post.

Executing step: [step2]

Checking if the collection exists

Status of collection returns :::::::::::::::::::::true

Show all objects

list of db objects returns:::::::::::::::::::::[{ "_id" : { "$oid" : "59e7c0324ad9510acf5773c0"} , [..]

Just return the id of Policy

RESULT:::::::::::::::::::::{ "_id" : { "$oid" : "59e7c0324ad9510acf5773c0"} , "Policy" : { "id" : "NRD-1"}}

To see the other results printed to the console, fork/download the code from Github and run the application.

10 – Conclusion

We have reviewed how to convert XML to JSON, store the JSON to MongoDB, and how to query the database for specific results.

Further reading:

The source code is on Github



Spring Batch CSV Processing

Overview

Topics we will be discussing include the essential concepts of batch processing with Spring Batch and how to import the data from a CSV into a database.

0 – Spring Batch CSV Processing Example Application

We are building an application that demonstrates the basics of Spring Batch for processing CSV files. Our demo application will allow us to process a CSV file that contains hundreds of records of Japanese anime titles.

0.1 – The CSV

I have downloaded the CSV we will be using from this Github repository, and it provides a pretty comprehensive list of animes.

Here is a screenshot of the CSV open in Microsoft Excel

Animes CSV screenshot

View and Download the code from Github

1 – Project Structure

Project structure of spring batch application

2 – Project Dependencies

Besides typical Spring Boot dependencies, we include spring-boot-starter-batch, which is the dependency for Spring Batch as the name suggests, and hsqldb for an in-memory database. We also include commons-lang3 for ToStringBuilder.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>michaelcgood-spring-batch-csv</artifactId>
	<version>0.0.1</version>
	<packaging>jar</packaging>

	<name>michaelcgood-spring-batch-csv</name>
	<description>Michael C  Good - Spring Batch CSV Example Application</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.7.RELEASE</version>
		<relativePath /> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-batch</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-jpa</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>

		<dependency>
			<groupId>org.hsqldb</groupId>
			<artifactId>hsqldb</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
		<dependency>
			<groupId>org.apache.commons</groupId>
			<artifactId>commons-lang3</artifactId>
			<version>3.6</version>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>


</project>

3 – Model

This is a POJO that models the fields of an anime. The fields are:

  • ID. For the sake of simplicity, we treat the ID as a String. However, this could be changed to another data type such as an Integer or Long.
  • Title. This is the title of the anime and it is appropriate for it to be a String.
  • Description. This is the description of the anime, which is longer than the title, and it can also be treated as a String.

What is important to note is our class constructor for the three fields: public AnimeDTO(String id, String title, String description). This will be used in our application. Also, as usual, we need to make a default constructor with no parameters or else Java will throw an error.

package com.michaelcgood;

import org.apache.commons.lang3.builder.ToStringBuilder;
/**
 * Contains the information of a single anime
 *
 * @author Michael C Good michaelcgood.com
 */

public class AnimeDTO {
	
	public String getId() {
		return id;
	}

	public void setId(String id) {
		this.id = id;
	}

	public String getTitle() {
		return title;
	}

	public void setTitle(String title) {
		this.title = title;
	}

	public String getDescription() {
		return description;
	}

	public void setDescription(String description) {
		this.description = description;
	}



	private String id;
	



	private String title;
	private String description;
	
	public AnimeDTO(){
		
	}
	
	public AnimeDTO(String id, String title, String description){
		this.id = id;
		this.title = title;
		this.description = title;
	}
	

	
	   @Override
	    public String toString() {
		   return new ToStringBuilder(this)
				   .append("id", this.id)
				   .append("title", this.title)
				   .append("description", this.description)
				   .toString();
	   }


}

4 – CSV File to Database Configuration

There is a lot going on in this class and it is not all written at once, so we are going to go through the code in steps. Visit Github to see the code in its entirety.

4.1 – Reader

As the Spring Batch documentation states FlatFileIteamReader will “read lines of data from a flat file that typically describe records with fields of data defined by fixed positions in the file or delimited by some special character (e.g. Comma)”.

We are dealing with a CSV, so of course the data is delimited by a comma, making this the perfect for use with our file.

   @Bean
    public FlatFileItemReader<AnimeDTO> csvAnimeReader(){
        FlatFileItemReader<AnimeDTO> reader = new FlatFileItemReader<AnimeDTO>();
        reader.setResource(new ClassPathResource("animescsv.csv"));
        reader.setLineMapper(new DefaultLineMapper<AnimeDTO>() {{
            setLineTokenizer(new DelimitedLineTokenizer() {{
                setNames(new String[] { "id", "title", "description" });
            }});
            setFieldSetMapper(new BeanWrapperFieldSetMapper<AnimeDTO>() {{
                setTargetType(AnimeDTO.class);
            }});
        }});
        return reader;
    }

Important points:

  • FlatFileItemReader is parameterized with a model. In our case, this is AnimeDTO.
  • FlatFileItemReader must set a resource. It uses setResource method. Here we set the resource to animescsv.csv
  • setLineMapper method converts Strings to objects representing the item. Our String will be an anime record consisting of an id, title, and description. This String is made into an object. Note that DefaultLineMapper is parameterized with our model, AnimeDTO.
  • However, LineMapper is given a raw line, which means there is work that needs to be done to map the fields appropriately. The line must be tokenized into a FieldSet, which DelimitedLineTokenizer takes care of. DelimitedLineTokenizer returns a FieldSet.
  • Now that we have a FieldSet, we need to map it. setFieldSetMapper is used for taking the FieldSet object and mapping its contents to a DTO, which is AnimeDTO in our case.
  • 4.2 – Processor

    If we want to transform the data before writing it to the database, an ItemProcessor is necessary. Our code does not actually apply any business logic to transform the data, but we allow for the capability to.

    4.2.1 – Processor in CsvFileToDatabaseConfig.Java

    csvAnimeProcessor returns a new instance of the AnimeProcessor object which we review below.

    	@Bean
    	ItemProcessor<AnimeDTO, AnimeDTO> csvAnimeProcessor() {
    		return new AnimeProcessor();
    	}
    

    4.2.2 – AnimeProcessor.Java

    If we wanted to apply business logic before writing to the database, you could manipulate the Strings before writing to the database. For instance, you could add toUpperCase() after getTitle to make the title upper case before writing to the database. However, I decided not to do that or apply any other business logic for this example processor, so no manipulation is being done. The Processor is here simply for demonstration.

    package com.michaelcgood;
    
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    
    import org.springframework.batch.item.ItemProcessor;
    
    public class AnimeProcessor implements ItemProcessor<AnimeDTO, AnimeDTO> {
    	
        private static final Logger log = LoggerFactory.getLogger(AnimeProcessor.class);
        
        @Override
        public AnimeDTO process(final AnimeDTO AnimeDTO) throws Exception {
        	
        	final String id = AnimeDTO.getId();
            final String title = AnimeDTO.getTitle();
            final String description = AnimeDTO.getDescription();
    
            final AnimeDTO transformedAnimeDTO = new AnimeDTO(id, title, description);
    
            log.info("Converting (" + AnimeDTO + ") into (" + transformedAnimeDTO + ")");
    
            return transformedAnimeDTO;
        }
    
    }
    
    

    4.3 – Writer

    The csvAnimeWriter method is responsible for actually writing the values into our database. Our database is an in-memory HSQLDB however this application allows us to easily swap out one database for another. The dataSource is autowired.

    	@Bean
    	public JdbcBatchItemWriter<AnimeDTO> csvAnimeWriter() {
    		 JdbcBatchItemWriter<AnimeDTO> excelAnimeWriter = new JdbcBatchItemWriter<AnimeDTO>();
    		 excelAnimeWriter.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<AnimeDTO>());
    		 excelAnimeWriter.setSql("INSERT INTO animes (id, title, description) VALUES (:id, :title, :description)");
    		 excelAnimeWriter.setDataSource(dataSource);
    	        return excelAnimeWriter;
    	}
    

    4.4 – Step

    A Step is a domain object that contains an independent, sequential phase of a batch job and contains all of the information needed to define and control the actual batch processing.

    Now that we’ve created the reader and processor for data we need to write it. For the reading, we’ve been using chunk-oriented processing, meaning we’ve been reading the data one at a time. Chunk-oriented processing also includes creating ‘chunks’ that will be written out, within a transaction boundary. For chunk-oriented processing, you set a commit interval and once the number of items read equals the commit interval that has been set, the entire chunk is written out via the ItemWriter, and the transaction is committed. We set the chunk interval size to 1.

    I suggest reading the Spring Batch documentation about chunk-oriented processing.

    Then the reader, processor, and writer call the methods we wrote.

    	@Bean
    	public Step csvFileToDatabaseStep() {
    		return stepBuilderFactory.get("csvFileToDatabaseStep")
    				.<AnimeDTO, AnimeDTO>chunk(1)
    				.reader(csvAnimeReader())
    				.processor(csvAnimeProcessor())
    				.writer(csvAnimeWriter())
    				.build();
    	}
    

    4.5 – Job

    A Job consists of Steps. We pass a parameter into the Job below because we want to track the completion of the Job.

    	@Bean
    	Job csvFileToDatabaseJob(JobCompletionNotificationListener listener) {
    		return jobBuilderFactory.get("csvFileToDatabaseJob")
    				.incrementer(new RunIdIncrementer())
    				.listener(listener)
    				.flow(csvFileToDatabaseStep())
    				.end()
    				.build();
    	}
    



    5 – Job Completion Notification Listener

    The class below autowires the JdbcTemplate because we’ve already set the dataSource and we want to easily make our query. The results of our are query are a list of AnimeDTO objects. For each object returned, we will create a message in our console to show that the item has been written to the database.

    package com.michaelcgood;
    
    import java.sql.ResultSet;
    import java.sql.SQLException;
    import java.util.List;
    
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    
    import org.springframework.batch.core.BatchStatus;
    import org.springframework.batch.core.JobExecution;
    import org.springframework.batch.core.listener.JobExecutionListenerSupport;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.jdbc.core.JdbcTemplate;
    import org.springframework.jdbc.core.RowMapper;
    import org.springframework.stereotype.Component;
    
    @Component
    public class JobCompletionNotificationListener extends JobExecutionListenerSupport {
    
    	private static final Logger log = LoggerFactory.getLogger(JobCompletionNotificationListener.class);
    
    	private final JdbcTemplate jdbcTemplate;
    
    	@Autowired
    	public JobCompletionNotificationListener(JdbcTemplate jdbcTemplate) {
    		this.jdbcTemplate = jdbcTemplate;
    	}
    
    	@Override
    	public void afterJob(JobExecution jobExecution) {
    		if(jobExecution.getStatus() == BatchStatus.COMPLETED) {
    			log.info("============ JOB FINISHED ============ Verifying the results....\n");
    
    			List<AnimeDTO> results = jdbcTemplate.query("SELECT id, title, description FROM animes", new RowMapper<AnimeDTO>() {
    				@Override
    				public AnimeDTO mapRow(ResultSet rs, int row) throws SQLException {
    					return new AnimeDTO(rs.getString(1), rs.getString(2), rs.getString(3));
    				}
    			});
    
    			for (AnimeDTO AnimeDTO : results) {
    				log.info("Discovered <" + AnimeDTO + "> in the database.");
    			}
    
    		}
    	}
    	
    }
    
    

    6 – SQL

    We need to create a schema for our database. As mentioned, we have made all fields Strings for ease of use, so we have made their data types VARCHAR.

    DROP TABLE animes IF EXISTS;
    CREATE TABLE animes  (
        id VARCHAR(10),
        title VARCHAR(400),
        description VARCHAR(999)
    );
    
    

    6 – Main

    This is a standard class with main(). As the Spring Documentation states, @SpringBootApplication is a convenience annotation that includes @Configuration, @EnableAutoConfiguration, @EnableWebMvc, and @ComponentScan.

    package com.michaelcgood;
    
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    
    @SpringBootApplication
    public class SpringBatchCsvApplication {
    
    	public static void main(String[] args) {
    		SpringApplication.run(SpringBatchCsvApplication.class, args);
    	}
    }
    
    

    7 – Demo

    7.1 – Converting

    The FieldSet is fed through the processor and “Converting” is printed to the console.
    Converting CSV to database in Spring Batch

    7.2 – Discovering New Items In Database

    When the Spring Batch Job is finished, we select all the records and print them out to the console individually.
    Discovering newly imported items in database in Spring Batch application

    7.3 – Batch Process Complete

    When the Batch Process is complete this is what is printed to the console.

    Job: [FlowJob: [name=csvFileToDatabaseJob]] completed with the following parameters: [{run.id=1, -spring.output.ansi.enabled=always}] and the following status: [COMPLETED]
    Started SpringBatchCsvApplication in 36.0 seconds (JVM running for 46.616)
    

    8 – Conclusion

    Spring Batch builds upon the POJO-based development approach and user-friendliness of the Spring Framework’s to make it easy for developers to create enterprise grade batch processing.

    The source code is on Github



Validation in Thymeleaf + Spring

Overview

Important topics we will be discussing are dealing with null values, empty strings, and validation of input so we do not enter invalid data into our database.

In dealing with null values, we touch on use of java.util.Optional which was introduced in Java 1.8.

0 – Spring Boot + Thymeleaf Example Form Validation Application

We are building a web application for a university that allows potential students to request information on their programs.

View and Download the code from Github

1 – Project Structure

Thymeleaf validation application project structure

2 – Project Dependencies

Besides our typical Spring Boot dependencies, we are using an embedded HSQLDB database and nekohtml for LEGACYHTML5 mode.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>michaelcgood-validation-thymeleaf</artifactId>
	<version>0.0.1</version>
	<packaging>jar</packaging>

	<name>michaelcgood-validation-thymeleaf</name>
	<description>Michael C  Good - Validation in Thymeleaf Example Application</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.7.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-jpa</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-thymeleaf</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>

		<dependency>
			<groupId>org.hsqldb</groupId>
			<artifactId>hsqldb</artifactId>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<!-- legacy html allow -->
		<dependency>
			<groupId>net.sourceforge.nekohtml</groupId>
			<artifactId>nekohtml</artifactId>
			<version>1.9.21</version>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>


</project>

3 – Model

In our model we define:

  • An autogenerated id field
  • A name field that cannot be null
  • That the name must be between 2 and 40 characters
  • An email field that is validated by the @Email annotation
  • A boolean field “openhouse” that allows a potential student to indicate if she wants to attend an open house
  • A boolean field “subscribe” for subscribing to email updates
  • A comments field that is optional, so there is no minimum character requirement but there is a maximum character requirement
package com.michaelcgood.model;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.validation.constraints.NotNull;
import javax.validation.constraints.Size;

import org.hibernate.validator.constraints.Email;

@Entity
public class Student {

	@Id
	@GeneratedValue(strategy = GenerationType.AUTO)
	private Long id;
	@NotNull
    @Size(min=2, max=40)
	private String name;
	@NotNull
	@Email
	private String email;
	private Boolean openhouse;
	private Boolean subscribe;
	 @Size(min=0, max=300)
	private String  comments;
	
	public Long getId() {
		return id;
	}
	public void setId(Long id) {
		this.id = id;
	}
	public String getName() {
		return name;
	}
	public void setName(String name) {
		this.name = name;
	}
	public String getEmail() {
		return email;
	}
	public void setEmail(String email) {
		this.email = email;
	}
	public Boolean getOpenhouse() {
		return openhouse;
	}
	public void setOpenhouse(Boolean openhouse) {
		this.openhouse = openhouse;
	}
	public Boolean getSubscribe() {
		return subscribe;
	}
	public void setSubscribe(Boolean subscribe) {
		this.subscribe = subscribe;
	}
	public String getComments() {
		return comments;
	}
	public void setComments(String comments) {
		this.comments = comments;
	}
	

}

4 – Repository

We define a repository.

package com.michaelcgood.dao;

import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;

import com.michaelcgood.model.Student;

@Repository
public interface StudentRepository extends JpaRepository<Student,Long> {

}

5 – Controller

We register StringTrimmerEditor to convert empty Strings to null values automatically.

When a user sends a POST request, we want to receive the value of that Student object, so we use @ModelAttribute to do just that.

To ensure that the user is sending values that are valid, we use the appropriately named @Valid annotation next.

BindingResult must follow next, or else the user is given an error page when submitting invalid data instead of remaining on the form page.

We use if…else to control what happens when a user submits a form. If the user submits invalid data, the user will remain on the current page and nothing more will occur on the server side. Otherwise, the application will consume the user’s data and the user can proceed.

At this point, it is kind of redundant to check if the student’s name is null, but we do. Then, we call the method checkNullString, which is defined below, to see if the comment field is an empty String or null.

package com.michaelcgood.controller;

import java.util.Optional;

import javax.validation.Valid;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.propertyeditors.StringTrimmerEditor;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.validation.BindingResult;
import org.springframework.web.bind.WebDataBinder;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.InitBinder;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.PostMapping;
import com.michaelcgood.dao.StudentRepository;
import com.michaelcgood.model.Student;

@Controller
public class StudentController {
	@InitBinder
	public void initBinder(WebDataBinder binder) {
	    binder.registerCustomEditor(String.class, new StringTrimmerEditor(true));
	}
	public String finalString = null;
	@Autowired
	private StudentRepository studentRepository;
	@PostMapping(value="/")
	public String addAStudent(@ModelAttribute @Valid Student newStudent, BindingResult bindingResult, Model model){
		if (bindingResult.hasErrors()) {
			System.out.println("BINDING RESULT ERROR");
			return "index";
		} else {
			model.addAttribute("student", newStudent);

			if (newStudent.getName() != null) {
				try {
					// check for comments and if not present set to 'none'
					String comments = checkNullString(newStudent.getComments());
					if (comments != "None") {
						System.out.println("nothing changes");
					} else {
						newStudent.setComments(comments);
					}
				} catch (Exception e) {

					System.out.println(e);

				}
				studentRepository.save(newStudent);
				System.out.println("new student added: " + newStudent);
			}

			return "thanks";
		}
	}
	
	@GetMapping(value="thanks")
	public String thankYou(@ModelAttribute Student newStudent, Model model){
		model.addAttribute("student",newStudent);
		
		return "thanks";
	}
	
	@GetMapping(value="/")
	public String viewTheForm(Model model){
		Student newStudent = new Student();
		model.addAttribute("student",newStudent);
		return "index";
	}
	
	public String checkNullString(String str){
		String endString = null;
		if(str == null || str.isEmpty()){
			System.out.println("yes it is empty");
			str = null;
			Optional<String> opt = Optional.ofNullable(str);
			endString = opt.orElse("None");
			System.out.println("endString : " + endString);
		}
		else{
			; //do nothing
		}
		
		
		return endString;
		
	}

}

Optional.ofNullable(str); means that the String will become the data type Optional, but the String may be a null value.

endString = opt.orElse(“None”); sets the String value to “None” if the variable opt is null.


6 – Thymeleaf Templates

As you saw in our Controller’s mapping above, there are two pages. The index.html is our main page that has the form for potential University students.

Our main object is Student, so of course our th:object refers to that. Our model’s fields respectively go into th:field.

We wrap our form’s inputs inside a table for formatting purposes.

Below each table cell (td) we have a conditional statement like this one: […]
th:if=”${#fields.hasErrors(‘name’)}” th:errors=”*{name}”
[…]

The above conditional statement means if the user inputs data into that field that doesn’t match the requirement we put for that field in our Student model and then submits the form, show the input requirements when the user is returned to this page.

index.html

<html xmlns="http://www.w3.org/1999/xhtml"
	xmlns:th="http://www.thymeleaf.org">

<head>
<!-- CSS INCLUDE -->
<link rel="stylesheet"
	href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
	integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u"
	crossorigin="anonymous"></link>

<!-- EOF CSS INCLUDE -->
</head>
<body>

	<!-- START PAGE CONTAINER -->
	<div class="container-fluid">
		<!-- PAGE TITLE -->
		<div class="page-title">
			<h2>
				<span class="fa fa-arrow-circle-o-left"></span> Request University
				Info
			</h2>
		</div>
		<!-- END PAGE TITLE -->
		<div class="column">
			<form action="#" th:action="@{/}" th:object="${student}"
				method="post">
				<table>
					<tr>
						<td>Name:</td>
						<td><input type="text" th:field="*{name}"></input></td>
						<td th:if="${#fields.hasErrors('name')}" th:errors="*{name}">Name
							Error</td>
					</tr>
					<tr>
						<td>Email:</td>
						<td><input type="text" th:field="*{email}"></input></td>
						<td th:if="${#fields.hasErrors('email')}" th:errors="*{email}">Email
							Error</td>
					</tr>
					<tr>
						<td>Comments:</td>
						<td><input type="text" th:field="*{comments}"></input></td>
					</tr>
					<tr>
						<td>Open House:</td>
						<td><input type="checkbox" th:field="*{openhouse}"></input></td>
				
					</tr>
					<tr>
						<td>Subscribe to updates:</td>
						<td><input type="checkbox" th:field="*{subscribe}"></input></td>
				
					</tr>
					<tr>
						<td>
							<button type="submit" class="btn btn-primary">Submit</button>
						</td>
					</tr>
				</table>
			</form>

		</div>
		<!-- END PAGE CONTENT -->
		<!-- END PAGE CONTAINER -->
	</div>
	<script src="https://code.jquery.com/jquery-1.11.1.min.js"
		integrity="sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE="
		crossorigin="anonymous"></script>


	<script
		src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"
		integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa"
		crossorigin="anonymous"></script>


</body>
</html>

Here we have the page that a user sees when they have successfully completed the form. We use th:textto show the user the text he or she input for that field.

thanks.html

<html xmlns="http://www.w3.org/1999/xhtml"
	xmlns:th="http://www.thymeleaf.org">

<head>
<!-- CSS INCLUDE -->
<link rel="stylesheet"
	href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
	integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u"
	crossorigin="anonymous"></link>

<!-- EOF CSS INCLUDE -->


</head>
<body>

	<!-- START PAGE CONTAINER -->
	<div class="container-fluid">

		<!-- PAGE TITLE -->
		<div class="page-title">
			<h2>
				<span class="fa fa-arrow-circle-o-left"></span> Thank you
			</h2>
		</div>
		<!-- END PAGE TITLE -->
		<div class="column">
			<table class="table datatable">
				<thead>
					<tr>
						<th>Name</th>
						<th>Email</th>
						<th>Open House</th>
						<th>Subscribe</th>
						<th>Comments</th>
					</tr>
				</thead>
				<tbody>
					<tr th:each="student : ${student}">
						<td th:text="${student.name}">Text ...</td>
						<td th:text="${student.email}">Text ...</td>
						<td th:text="${student.openhouse}">Text ...</td>
						<td th:text="${student.subscribe}">Text ...</td>
						<td th:text="${student.comments}">Text ...</td>
					</tr>
				</tbody>
			</table>
		</div>	
		</div>
		<!-- END PAGE CONTAINER -->
	</div>
		<script
  src="https://code.jquery.com/jquery-1.11.1.min.js"
  integrity="sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE="
  crossorigin="anonymous"></script>
 

	<script
		src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"
		integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa"
		crossorigin="anonymous"></script>

</body>
</html>

7 – Configuration

Using Spring Boot Starter and including Thymeleaf dependencies, you will automatically have a templates location of /templates/, and Thymeleaf just works out of the box. So most of these settings aren’t needed.

The one setting to take note of is LEGACYHTM5 which is provided by nekohtml. This allows us to use more casual HTML5 tags if we want to. Otherwise, Thymeleaf will be very strict and may not parse your HTML. For instance, if you do not close an input tag, Thymeleaf will not parse your HTML.

application.properties

#==================================
# = Thymeleaf configurations 
#==================================
spring.thymeleaf.check-template-location=true
spring.thymeleaf.prefix=classpath:/templates/
spring.thymeleaf.suffix=.html
spring.thymeleaf.content-type=text/html
spring.thymeleaf.cache=false
spring.thymeleaf.mode=LEGACYHTML5

server.contextPath=/

8 – Demo

Home page

Here we arrive on the home page.
Home page of example Thymeleaf validation application

Invalid data

I input invalid data into the name field and email field.

Invalid data in example Thymeleaf validation

Valid data with no comment

Now I put valid data in all fields, but do not provide a comment. It is not required to provide a comment. In our controller, we made all empty Strings null values. If the user did not provide a comment, the String value is made “None”.

Comments set to none Thymeleaf

9 – Conclusion

Wrap up

This demo application demonstrated how to valid user input in a Thymeleaf form.
In my opinion, Spring and Thymeleaf work well with javax.validation.constraints for validating user input.
The source code is on Github

Notes

Java 8’s Optional was sort of forced into this application for demonstration purposes, and I want to note it works more organically when using @RequestParam as shown in my PagingAndSortingRepository tutorial.

However, if you were not using Thymeleaf, you could have possibly made our not required fields Optional. Here Vlad Mihalcea discusses the best way to map Optional entity attribute with JPA and Hibernate.


Building Spring Boot RESTful Service + Spring Boot Actuator

Overview

What is REST?

REST(REpresentational State Transfer) is the architectural style the web is built on and has become a standard software design pattern used for web applications. The term Representational State Transfer was first used by Roy Fielding, the originator of REST and one of the principal authors of HTTP specification, in his doctoral dissertation.

There are many good references on REST including:

This tutorial is based on Building Rest Services with Spring and the beginning of the tutorial has a good overview of REST as well.

What is Spring Boot Actuator?

Spring Boot Actuator is a sub-project of Spring Boot. It adds several production grade services to your application with minimal effort on your part.

Definition of Actuator

An actuator is a component responsible for moving or controlling a system.
The term actuator is not limited to Spring Boot; however, that is our focus here.

After Actuator is configured in your Spring Boot application, it allows you to interact and monitor your application by invoking different technology agnostic endpoints exposed by Spring Boot Actuator such as application health, beans, loggers, mappings, and trace. More are listed in this Spring doc.

0 – Spring Boot RESTful Web Service with Actuator Example Application

We will build an example RESTful web application with Spring Boot and Actuator.

The application will be a “username tracker.” In this application, a person has one account and their account may have many usernames.

View and Download the code from Github

1 – Project Structure

As usual, we have a normal Maven project structure.
project structure of spring boot actuator rest example

2 – Project Dependencies

Besides typical Spring Boot dependencies, we are including HSQLDB for our embedded database and spring-boot-starter-actuator for all the Actuator dependencies.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>michaelcgood-springbootactuator</artifactId>
	<version>0.0.1</version>
	<packaging>jar</packaging>

	<name>Spring-Boot-Actuator-Example</name>
	<description>Michael C  Good - Spring Boot Actuator Example</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.6.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		 <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
			<groupId>org.hsqldb</groupId>
			<artifactId>hsqldb</artifactId>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-jpa</artifactId>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>


</project>

3 – Run the Empty Application

Although we haven’t written any code, we will run the Spring Boot application.

Go to your terminal and follow along with the commands.

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080
{"timestamp":1505235455245,"status":404,"error":"Not Found","message":"No message available","path":"/"}

We haven’t written any code yet, instead of a default container-generated HTML error response, Actuator produces you a JSON response from its /error endpoint.

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080/health
{"status":"UP"}

The Actuator /health endpoint will let you know if your application is up.

4 – Model

Now let’s define the fields of the models for our username tracker application.

  • As mentioned, a person has one account and may have many usernames. So we map Set with a @OneToMany annotation
  • A username model will have a password and a username of course
  • Our model will need an ID and we make it autogenerated
  • We make a class construction to define an account may be made with a username and a password. Because of this custom constructor we also need to make a default one with no parameters.

Account.java

package com.michaelcgood.model;

import java.util.HashSet;
import java.util.Set;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import javax.persistence.OneToMany;

import com.fasterxml.jackson.annotation.JsonIgnore;

@Entity
public class Account {

	public Set<Usernames> getUsernames() {
		return usernames;
	}

	public void setUsernames(Set<Usernames> usernames) {
		this.usernames = usernames;
	}

	public Long getId() {
		return id;
	}

	public void setId(Long id) {
		this.id = id;
	}

	public String getPassword() {
		return password;
	}

	public void setPassword(String password) {
		this.password = password;
	}

	public String getUsername() {
		return username;
	}

	public void setUsername(String username) {
		this.username = username;
	}

	@OneToMany(mappedBy= "account")
	private Set<Usernames> usernames = new HashSet<>();
	
	@Id
	@GeneratedValue
	private Long id;
	
	@JsonIgnore
	public String password;
	public String username;
	
	public Account(String name, String password) {
        this.username = name;
        this.password = password;
    }
	
	Account(){
		
	}
	
}

Usernames.java

  • As there is one account to many usernames, the reverse is true as well: there many usernames to one account. Therefore, we map Account with @ManyToOne annotation
  • To track a username we need: the url and the username
  • We once again define an autogenerated ID
  • We define a custom class constructor that requires account, url, and username parameter. Once again we need to define a default constructor method to avoid an error being thrown.
package com.michaelcgood.model;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import javax.persistence.ManyToOne;

import com.fasterxml.jackson.annotation.JsonIgnore;

@Entity
public class Usernames {
	
	@JsonIgnore
	@ManyToOne
	private Account account;
	
	public Account getAccount() {
		return account;
	}

	public void setAccount(Account account) {
		this.account = account;
	}

	public Long getId() {
		return id;
	}

	public void setId(Long id) {
		this.id = id;
	}

	public String getUrl() {
		return url;
	}

	public void setUrl(String url) {
		this.url = url;
	}

	public String getUsername() {
		return username;
	}

	public void setUsername(String username) {
		this.username = username;
	}

	@Id
	@GeneratedValue
	private Long id;
	
	public String url;
	public String username;
	
	Usernames(){
		
	}
	
	public Usernames(Account account, String url, String username){
		this.url=url;
		this.account=account;
		this.username=username;
	}
	


}

5 – Repository

We create a repository for both models and create search functions using derived queries.

AccountRepository.java

package com.michaelcgood.dao;

import java.util.Optional;

import org.springframework.data.jpa.repository.JpaRepository;

import com.michaelcgood.model.Account;

public interface AccountRepository extends JpaRepository<Account,Long> {

	Optional<Account> findByUsername(String username);
}

UsernamesRepository.java

package com.michaelcgood.dao;

import java.util.Collection;

import org.springframework.data.jpa.repository.JpaRepository;

import com.michaelcgood.model.Usernames;

public interface UsernamesRepository extends JpaRepository<Usernames,Long> {

	Collection<Usernames> findByAccountUsername(String username);
	
}



6 – Controller

In the controller, we define all the mapping that we will be using for our RESTful web service.

  • We annotate our controller with @RestController rather than @Controller. As stated in the javadoc, it is “a convenience annotation that is itself annotated with @Controller and @ResponseBody.”
  • We declare the variables for our UsernamesRepository and AccountRepository and make them final because we only want the value to be assigned once. We annotate them as @Autowired over the UsernamesRestController class constructor.
  • {userId} and {usernamesId} are path variables. That means these values are provided in a URL. This will be shown in our demo.
  • The Controller methods return POJOs (Plain Old Java Objects). Spring Boot automatically wires HttpMessageConverter to convert these generic objects to JSON.

UsernamesRestController.java

package com.michaelcgood.controller;

import java.net.URI;
import java.util.Collection;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.servlet.support.ServletUriComponentsBuilder;

import com.michaelcgood.dao.AccountRepository;
import com.michaelcgood.dao.UsernamesRepository;
import com.michaelcgood.model.Usernames;

@RestController
@RequestMapping("/{userId}/usernames")
public class UsernamesRestController {

	private final UsernamesRepository usernamesRepository;
	private final AccountRepository accountRepository;
	
	@Autowired
	UsernamesRestController(UsernamesRepository usernamesRepository, AccountRepository accountRepository){
		this.usernamesRepository = usernamesRepository;
		this.accountRepository = accountRepository;
	}
	
	@GetMapping
	Collection<Usernames> readUsernames (@PathVariable String userId){
		this.validateUser(userId);
		return this.usernamesRepository.findByAccountUsername(userId);
	}
	
	@PostMapping
	ResponseEntity<?> add(@PathVariable String userId,@RequestBody Usernames input){
		this.validateUser(userId);
		
		return this.accountRepository.findByUsername(userId)
				.map(account -> {
					Usernames result = usernamesRepository.save(new Usernames(account,input.url,input.username));
					
					URI url = ServletUriComponentsBuilder
							.fromCurrentRequest().path("/{id}")
							.buildAndExpand(result.getId()).toUri();
					
						return ResponseEntity.created(url).build();	
				})
				.orElse(ResponseEntity.noContent().build());
	}
	
	@GetMapping(value="{usernamesId}")
	Usernames readUsername(@PathVariable String userId, @PathVariable Long usernameId){
		this.validateUser(userId);
		return this.usernamesRepository.findOne(usernameId);
	}
	
	private void validateUser(String userId){
		this.accountRepository.findByUsername(userId).orElseThrow(
				() -> new UserNotFoundException(userId));
	}
	
	
}

UserNotFoundException.java

Here we define the custom exception we used in our Controller class to explain a user could not be found.

package com.michaelcgood.controller;

import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.ResponseStatus;

@ResponseStatus(HttpStatus.NOT_FOUND)
public class UserNotFoundException extends RuntimeException {

/**
	 * 
	 */
	private static final long serialVersionUID = 7537022054146700535L;

public UserNotFoundException(String userId){
	super("Sorry, we could not find user '" + userId +"'.");
}
	
	
}

7 – @SpringBootApplication

We use the CommandLineRunner to create accounts and insert usernames. Every account will have two usernames.

package com.michaelcgood;

import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

import com.michaelcgood.dao.AccountRepository;
import com.michaelcgood.dao.UsernamesRepository;
import com.michaelcgood.model.Account;
import com.michaelcgood.model.Usernames;

import java.util.Arrays;

@SpringBootApplication
public class SpringBootActuatorExampleApplication {

	public static void main(String[] args) {
		SpringApplication.run(SpringBootActuatorExampleApplication.class, args);
	}
	
	@Bean
	CommandLineRunner init(AccountRepository accountRepository,
			UsernamesRepository usernamesRepository) {
		return (evt) -> Arrays.asList(
				"ricksanchez,mortysmith,bethsmith,jerrysmith,summersmith,birdperson,squanchy,picklerick".split(","))
				.forEach(
						a -> {
							Account account = accountRepository.save(new Account(a,
									"password"));
							usernamesRepository.save(new Usernames(account,
									"http://example.com/login", a +"1"));
							usernamesRepository.save(new Usernames(account,
									"http://example2.com/login", "the_"+a));
						});
	}
}

8 – Configuration

It is stated in the Spring documentation:

By default all sensitive HTTP endpoints are secured such that only users that have an ACTUATOR role may access them. Security is enforced using the standard HttpServletRequest.isUserInRole method.

We haven’t set up any security and user roles, as this is just an example. So, for ease of demonstration, I will be disabling the security requirement. Otherwise, we will get an “unauthorized” error as of right now, like the one shown below.

{"timestamp":1505321635068,"status":401,"error":"Unauthorized","message":"Full authentication is required to access this resource.","path":"/beans"}

application.properties

Add this to your application.properties to disable the need for authentication.

management.security.enabled=false

9 – Demo

To retrieve the responses from the server, you could either visit the URL in your browser or use curl. For my demo, I am using curl.

REST queries for data in repositories

Query for usernames belonging to account jerrysmith.

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080/jerrysmith/usernames
[{"id":7,"url":"http://example.com/login","username":"jerrysmith1"},{"id":8,"url":"http://example2.com/login","username":"the_jerrysmith"}]

Query for usernames belonging to account picklerick

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080/picklerick/usernames
[{"id":15,"url":"http://example.com/login","username":"picklerick1"},{"id":16,"url":"http://example2.com/login","username":"the_picklerick"}]

Actuator queries

The response to this query is truncated because it is really, really long.

Beans

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080/beans
[{"context":"application","parent":null,"beans":[{"bean":"springBootActuatorExampleApplication","aliases":[],"scope":"singleton","type":"com.michaelcgood.SpringBootActuatorExampleApplication$$EnhancerBySpringCGLIB$$509f4984","resource":"null","dependencies":[]},{"bean":"org.springframework.boot.autoconfigure.internalCachingMetadataReaderFactory","aliases":[],"scope":"singleton","type":"org.springframework.core.type.classreading.CachingMetadataReaderFactory","resource":"null","dependencies":[]},{"bean":"usernamesRestController","aliases":[],"scope":"singleton","type":"com.michaelcgood.controller.UsernamesRestController","resource":"file [/Users/mike/javaSTS/Spring-Boot-Actuator-Example/target/classes/com/michaelcgood/controller/UsernamesRestController.class]","dependencies":["usernamesRepository","accountRepository"]},{"bean":"init","aliases":[],"scope":"singleton","type":"com.michaelcgood.SpringBootActuatorExampleApplication$$Lambda$11/889398176","resource":"com.michaelcgood.SpringBootActuatorExampleApplication",
[...]

Metrics

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080/metrics
{"mem":350557,"mem.free":208275,"processors":4,"instance.uptime":213550,"uptime":240641,"systemload.average":1.6552734375,"heap.committed":277504,"heap.init":131072,"heap.used":69228,"heap":1864192,"nonheap.committed":74624,"nonheap.init":2496,"nonheap.used":73062,"nonheap":0,"threads.peak":27,"threads.daemon":23,"threads.totalStarted":30,"threads":25,"classes":9791,"classes.loaded":9791,"classes.unloaded":0,"gc.ps_scavenge.count":11,"gc.ps_scavenge.time":139,"gc.ps_marksweep.count":2,"gc.ps_marksweep.time":148,"httpsessions.max":-1,"httpsessions.active":0,"datasource.primary.active":0,"datasource.primary.usage":0.0,"gauge.response.beans":14.0,"gauge.response.info":13.0,"counter.status.200.beans":2,"counter.status.200.info":1}

9 – Conclusion

Congratulations, you have created a RESTful Web Service that can be monitored with Actuator. REST is really the most organic way for different clients to communicate because it works due to HTTP.

The source code is on Github