Converting HTML to RichTextString for Apache POI

1. Overview

In this tutorial, we will be building an application that takes HTML as an input and creates a Microsoft Excel Workbook with a RichText representation of the HTML that was provided. To generate the Microsoft Excel Workbook, we will be using Apache POI. To analyze the HTML, we will be using Jericho.

The full source code for this tutorial is available on Github.

2. What is Jericho?

Jericho is a java library that allows analysis and manipulation of parts of an HTML document, including server-side tags, while reproducing verbatim any unrecognized or invalid HTML. It also provides high-level HTML form manipulation functions. It is an open source library released under the following licenses: Eclipse Public License (EPL)GNU Lesser General Public License (LGPL), and Apache License.

I found Jericho to be very easy to use for achieving my goal of converting HTML to RichText.

3. pom.xml

Here are the required dependencies for the application we are building. Please take note that for this application we have to use Java 9. This is because of a java.util.regex appendReplacement method we use that has only been available since Java 9.

<parent>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-parent</artifactId>
	<version>1.5.9.RELEASE</version>
	<relativePath /> <!-- lookup parent from repository -->
</parent>

<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
	<java.version>9</java.version>
</properties>

<dependencies>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-batch</artifactId>
	</dependency>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-thymeleaf</artifactId>
	</dependency>

	<dependency>
		<groupId>com.h2database</groupId>
		<artifactId>h2</artifactId>
		<scope>runtime</scope>
	</dependency>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-test</artifactId>
		<scope>test</scope>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
	<dependency>
		<groupId>org.apache.commons</groupId>
		<artifactId>commons-lang3</artifactId>
		<version>3.7</version>
	</dependency>
	<dependency>
		<groupId>org.springframework.batch</groupId>
		<artifactId>spring-batch-test</artifactId>
		<scope>test</scope>
	</dependency>
	<dependency>
		<groupId>org.apache.poi</groupId>
		<artifactId>poi</artifactId>
		<version>3.15</version>
	</dependency>

	<dependency>
		<groupId>org.apache.poi</groupId>
		<artifactId>poi-ooxml</artifactId>
		<version>3.15</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/net.htmlparser.jericho/jericho-html -->
	<dependency>
		<groupId>net.htmlparser.jericho</groupId>
		<artifactId>jericho-html</artifactId>
		<version>3.4</version>
	</dependency>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-configuration-processor</artifactId>
		<optional>true</optional>
	</dependency>
	<!-- legacy html allow -->
	<dependency>
		<groupId>net.sourceforge.nekohtml</groupId>
		<artifactId>nekohtml</artifactId>
	</dependency>
</dependencies>

4. Web page – Thymeleaf

We use Thymeleaf to create a basic webpage that has a form with a textarea. The source code for Thymeleaf page is available here on Github. This textarea could be replaced with a RichText Editor if we like, such as CKEditor. We just must be mindful to make the data for AJAX correct, using an appropriate setData method. There is a previous tutorial about CKeditor titled AJAX with CKEditor in Spring Boot.

5. Controller

In our controller, we Autowire JobLauncher and a Spring Batch job we are going to create called GenerateExcel. Autowiring these two classes allow us to run the Spring Batch Job GenerateExcel on demand when a POST request is sent to “/export”.

Another thing to note is that to ensure that the Spring Batch job will run more than once we include unique parameters with this code: addLong(“uniqueness”, System.nanoTime()).toJobParameters(). An error may occur if we do not include unique parameters because only unique JobInstances may be created and executed, and Spring Batch has no way of distinguishing between the first and second JobInstance otherwise.

@Controller
public class WebController {

    private String currentContent;

    @Autowired
    JobLauncher jobLauncher;
    
    @Autowired
    GenerateExcel exceljob; 

    @GetMapping("/")
    public ModelAndView getHome() {
        ModelAndView modelAndView = new ModelAndView("index");
        return modelAndView;

    }
    

    @PostMapping("/export")
    public String postTheFile(@RequestBody String body, RedirectAttributes redirectAttributes, Model model)
        throws IOException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException, JobParametersInvalidException {


        setCurrentContent(body);

        Job job = exceljob.ExcelGenerator();
        jobLauncher.run(job, new JobParametersBuilder().addLong("uniqueness", System.nanoTime()).toJobParameters()
            );

        return "redirect:/";
    }

    //standard getters and setters

}



6. Batch Job

In Step1 of our Batch job, we call the getCurrentContent() method to get the content that was passed into the Thymeleaf form, create a new XSSFWorkbook, specify an arbitrary Microsoft Excel Sheet tab name, and then pass all three variables into the createWorksheet method that we will be making in the next step of our tutorial :

@Configuration
@EnableBatchProcessing
@Lazy
public class GenerateExcel {
    
    List<String> docIds = new ArrayList<String>();

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Autowired
    WebController webcontroller;
    
    @Autowired
    CreateWorksheet createexcel;

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
            .tasklet(new Tasklet() {
                @Override
                public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception, JSONException {

                    String content = webcontroller.getCurrentContent();
                    
                    System.out.println("content is ::" + content);
                    Workbook wb = new XSSFWorkbook();
                    String tabName = "some";
                    createexcel.createWorkSheet(wb, content, tabName);

                    return RepeatStatus.FINISHED;
                }
            })
            .build();
    }

    @Bean
    public Job ExcelGenerator() {
        return jobBuilderFactory.get("ExcelGenerator")
            .start(step1())
            .build();

    }

}

We have covered Spring Batch in other tutorials such as Converting XML to JSON + Spring Batch and Spring Batch CSV Processing.

7. Excel Creation Service

We use a variety of classes to create our Microsoft Excel file. Order matters when dealing with converting HTML to RichText, so this will be a focus.

7.1 RichTextDetails

A class with two parameters: a String that will have our contents that will become RichText and a font map.

public class RichTextDetails {
    private String richText;
    private Map<Integer, Font> fontMap;
    //standard getters and setters
    @Override
    public int hashCode() {
     
        // The goal is to have a more efficient hashcode than standard one.
        return richText.hashCode();
    }

7.2 RichTextInfo

A POJO that will keep track of the location of the RichText and what not:

public class RichTextInfo {
    private int startIndex;
    private int endIndex;
    private STYLES fontStyle;
    private String fontValue;
    // standard getters and setters, and the like

7.3 Styles

A enum to contains HTML tags that we want to process. We can add to this as necessary:

public enum STYLES {
    BOLD("b"), 
    EM("em"), 
    STRONG("strong"), 
    COLOR("color"), 
    UNDERLINE("u"), 
    SPAN("span"), 
    ITALLICS("i"), 
    UNKNOWN("unknown"),
    PRE("pre");
    // standard getters and setters

7.4 TagInfo

A POJO to keep track of tag info:

public class TagInfo {
    private String tagName;
    private String style;
    private int tagType;
    // standard getters and setters

7.5 HTML to RichText

This is not a small class, so let’s break it down by method.

Essentially, we are surrounding any arbitrary HTML with a div tag, so we know what we are looking for. Then we look for all elements within the div tag, add each to an ArrayList of RichTextDetails , and then pass the whole ArrayList to the mergeTextDetails method. mergeTextDetails returns RichtextString, which is what we need to set a cell value:

   public RichTextString fromHtmlToCellValue(String html, Workbook workBook){
       Config.IsHTMLEmptyElementTagRecognised = true;
       
       Matcher m = HEAVY_REGEX.matcher(html);
       String replacedhtml =  m.replaceAll("");
       StringBuilder sb = new StringBuilder();
       sb.insert(0, "<div>");
       sb.append(replacedhtml);
       sb.append("</div>");
       String newhtml = sb.toString();
       Source source = new Source(newhtml);
       List<RichTextDetails> cellValues = new ArrayList<RichTextDetails>();
       for(Element el : source.getAllElements("div")){
           cellValues.add(createCellValue(el.toString(), workBook));
       }
       RichTextString cellValue = mergeTextDetails(cellValues);

       
       return cellValue;
   }

As we saw above, we pass an ArrayList of RichTextDetails in this method. Jericho has a setting that takes boolean value to recognize empty tag elements such as
: Config.IsHTMLEmptyElementTagRecognised. This can be important when dealing with online rich text editors, so we set this to true. Because we need to keep track of the order of the elements, we use a LinkedHashMap instead of a HashMap.

    private static RichTextString mergeTextDetails(List<RichTextDetails> cellValues) {
        Config.IsHTMLEmptyElementTagRecognised = true;
        StringBuilder textBuffer = new StringBuilder();
        Map<Integer, Font> mergedMap = new LinkedHashMap<Integer, Font>(550, .95f);
        int currentIndex = 0;
        for (RichTextDetails richTextDetail : cellValues) {
            //textBuffer.append(BULLET_CHARACTER + " ");
            currentIndex = textBuffer.length();
            for (Entry<Integer, Font> entry : richTextDetail.getFontMap()
                .entrySet()) {
                mergedMap.put(entry.getKey() + currentIndex, entry.getValue());
            }
            textBuffer.append(richTextDetail.getRichText())
                .append(NEW_LINE);
        }

        RichTextString richText = new XSSFRichTextString(textBuffer.toString());
        for (int i = 0; i < textBuffer.length(); i++) {
            Font currentFont = mergedMap.get(i);
            if (currentFont != null) {
                richText.applyFont(i, i + 1, currentFont);
            }
        }
        return richText;
    }

As mentioned above, we are using Java 9 in order to use StringBuilder with the java.util.regex.Matcher.appendReplacement. Why? Well that’s because StringBuffer slower than StringBuilder for operations. StringBuffer functions are synchronized for thread safety and thus slower.

We are using Deque instead of Stack because a more complete and consistent set of LIFO stack operations is provided by the Deque interface:

    static RichTextDetails createCellValue(String html, Workbook workBook) {
        Config.IsHTMLEmptyElementTagRecognised  = true;
        Source source = new Source(html);
        Map<String, TagInfo> tagMap = new LinkedHashMap<String, TagInfo>(550, .95f);
        for (Element e : source.getChildElements()) {
            getInfo(e, tagMap);
        }

        StringBuilder sbPatt = new StringBuilder();
        sbPatt.append("(").append(StringUtils.join(tagMap.keySet(), "|")).append(")");
        String patternString = sbPatt.toString();
        Pattern pattern = Pattern.compile(patternString);
        Matcher matcher = pattern.matcher(html);

        StringBuilder textBuffer = new StringBuilder();
        List<RichTextInfo> textInfos = new ArrayList<RichTextInfo>();
        ArrayDeque<RichTextInfo> richTextBuffer = new ArrayDeque<RichTextInfo>();
        while (matcher.find()) {
            matcher.appendReplacement(textBuffer, "");
            TagInfo currentTag = tagMap.get(matcher.group(1));
            if (START_TAG == currentTag.getTagType()) {
                richTextBuffer.push(getRichTextInfo(currentTag, textBuffer.length(), workBook));
            } else {
                if (!richTextBuffer.isEmpty()) {
                    RichTextInfo info = richTextBuffer.pop();
                    if (info != null) {
                        info.setEndIndex(textBuffer.length());
                        textInfos.add(info);
                    }
                }
            }
        }
        matcher.appendTail(textBuffer);
        Map<Integer, Font> fontMap = buildFontMap(textInfos, workBook);

        return new RichTextDetails(textBuffer.toString(), fontMap);
    }

We can see where RichTextInfo comes in to use here:

    private static Map<Integer, Font> buildFontMap(List<RichTextInfo> textInfos, Workbook workBook) {
        Map<Integer, Font> fontMap = new LinkedHashMap<Integer, Font>(550, .95f);

        for (RichTextInfo richTextInfo : textInfos) {
            if (richTextInfo.isValid()) {
                for (int i = richTextInfo.getStartIndex(); i < richTextInfo.getEndIndex(); i++) {
                    fontMap.put(i, mergeFont(fontMap.get(i), richTextInfo.getFontStyle(), richTextInfo.getFontValue(), workBook));
                }
            }
        }

        return fontMap;
    }

Where we use STYLES enum:

    private static Font mergeFont(Font font, STYLES fontStyle, String fontValue, Workbook workBook) {
        if (font == null) {
            font = workBook.createFont();
        }

        switch (fontStyle) {
        case BOLD:
        case EM:
        case STRONG:
            font.setBoldweight(Font.BOLDWEIGHT_BOLD);
            break;
        case UNDERLINE:
            font.setUnderline(Font.U_SINGLE);
            break;
        case ITALLICS:
            font.setItalic(true);
            break;
        case PRE:
            font.setFontName("Courier New");
        case COLOR:
            if (!isEmpty(fontValue)) {

                font.setColor(IndexedColors.BLACK.getIndex());
            }
            break;
        default:
            break;
        }

        return font;
    }

We are making use of the TagInfo class to track the current tag:

    private static RichTextInfo getRichTextInfo(TagInfo currentTag, int startIndex, Workbook workBook) {
        RichTextInfo info = null;
        switch (STYLES.fromValue(currentTag.getTagName())) {
        case SPAN:
            if (!isEmpty(currentTag.getStyle())) {
                for (String style : currentTag.getStyle()
                    .split(";")) {
                    String[] styleDetails = style.split(":");
                    if (styleDetails != null && styleDetails.length > 1) {
                        if ("COLOR".equalsIgnoreCase(styleDetails[0].trim())) {
                            info = new RichTextInfo(startIndex, -1, STYLES.COLOR, styleDetails[1]);
                        }
                    }
                }
            }
            break;
        default:
            info = new RichTextInfo(startIndex, -1, STYLES.fromValue(currentTag.getTagName()));
            break;
        }
        return info;
    }

We process the HTML tags:

    private static void getInfo(Element e, Map<String, TagInfo> tagMap) {
        tagMap.put(e.getStartTag()
            .toString(),
            new TagInfo(e.getStartTag()
                .getName(), e.getAttributeValue("style"), START_TAG));
        if (e.getChildElements()
            .size() > 0) {
            List<Element> children = e.getChildElements();
            for (Element child : children) {
                getInfo(child, tagMap);
            }
        }
        if (e.getEndTag() != null) {
            tagMap.put(e.getEndTag()
                .toString(),
                new TagInfo(e.getEndTag()
                    .getName(), END_TAG));
        } else {
            // Handling self closing tags
            tagMap.put(e.getStartTag()
                .toString(),
                new TagInfo(e.getStartTag()
                    .getName(), END_TAG));
        }
    }

7.6 Create Worksheet

Using StringBuilder, I create a String that is going to written to FileOutPutStream. In a real application this should be user defined. I appended my folder path and filename on two different lines. Please change the file path to your own.

sheet.createRow(0) creates a row on the very first line and dataRow.createCell(0) creates a cell in column A of the row.

public void createWorkSheet(Workbook wb, String content, String tabName) {
        StringBuilder sbFileName = new StringBuilder();
        sbFileName.append("/Users/mike/javaSTS/michaelcgood-apache-poi-richtext/");
        sbFileName.append("myfile.xlsx");
        String fileMacTest = sbFileName.toString();
        try {
            this.fileOut = new FileOutputStream(fileMacTest);
        } catch (FileNotFoundException ex) {
            Logger.getLogger(CreateWorksheet.class.getName())
                .log(Level.SEVERE, null, ex);
        }

        Sheet sheet = wb.createSheet(tabName); // Create new sheet w/ Tab name

        sheet.setZoom(85); // Set sheet zoom: 85%
        

        // content rich text
        RichTextString contentRich = null;
        if (content != null) {
            contentRich = htmlToExcel.fromHtmlToCellValue(content, wb);
        }


        // begin insertion of values into cells
        Row dataRow = sheet.createRow(0);
        Cell A = dataRow.createCell(0); // Row Number
        A.setCellValue(contentRich);
        sheet.autoSizeColumn(0);
        
        
        try {
            /////////////////////////////////
            // Write the output to a file
            wb.write(fileOut);
            fileOut.close();
        } catch (IOException ex) {
            Logger.getLogger(CreateWorksheet.class.getName())
                .log(Level.SEVERE, null, ex);
        }


    }

8. Demo

We visit localhost:8080.

We input some text with some HTML:
example of input for apache poi richtext app

We open up our excel file and see the RichText we created:
showing richtext in excel from apache poi app

9. Conclusion

We can see it is not trivial to convert HTML to Apache POI’s RichTextString class; however, for business applications converting HTML to RichTextString can be essential because readability is important in Microsoft Excel files. There’s likely room to improve upon the performance of the application we build, but we covered the foundation of building such an application .

The full source code is available on Github.



Intro to Redis with Spring Boot

1. Overview

In this article, we will review the basics of how to use Redis with Spring Boot through the Spring Data Redis library.

We will build an application that demonstrates how to perform CRUD operations Redis through a web interface. The full source code for this project is available on Github.

2. What is Redis?

Redis is an open source, in-memory key-value data store, used as a database, cache and message broker. In terms of implementation, Key Value stores represent one of the largest and oldest members in the NoSQL space. Redis supports data structures such as strings, hashes, lists, sets, and sorted sets with range queries.

The Spring Data Redis  framework makes it easy to write Spring applications that use the Redis key value store by providing an abstraction to the data store.

3. Setting Up A Redis Server

The server is available for free at http://redis.io/download.

If you use a Mac, you can install it with homebrew:

brew install redis

Then start the server:

mikes-MacBook-Air:~ mike$ redis-server
10699:C 23 Nov 08:35:58.306 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
10699:C 23 Nov 08:35:58.307 # Redis version=4.0.2, bits=64, commit=00000000, modified=0, pid=10699, just started
10699:C 23 Nov 08:35:58.307 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
10699:M 23 Nov 08:35:58.309 * Increased maximum number of open files to 10032 (it was originally set to 256).
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 4.0.2 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 10699
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

10699:M 23 Nov 08:35:58.312 # Server initialized
10699:M 23 Nov 08:35:58.312 * Ready to accept connections

 

4. Maven Dependencies

Let’s declare the necessary dependencies in our pom.xml for the example application we are building:

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-redis</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-thymeleaf</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>



5. Redis Configuration

We need to connect our application with the Redis server. To establish this connection, we are using Jedis, a Redis client implementation.

5.1 Config

Let’s start with the configuration bean definitions:

    @Bean
    JedisConnectionFactory jedisConnectionFactory() {
        return new JedisConnectionFactory();
    }

    @Bean
    public RedisTemplate<String, Object> redisTemplate() {
        final RedisTemplate<String, Object> template = new RedisTemplate<String, Object>();
        template.setConnectionFactory(jedisConnectionFactory());
        template.setValueSerializer(new GenericToStringSerializer<Object>(Object.class));
        return template;
    }

The JedisConnectionFactory is made into a bean so we can create a RedisTemplate to query data.

5.2 Message Publisher

Following the principles of SOLID, we create a MessagePublisher interface:

public interface MessagePublisher {

    void publish(final String message);
}

We implement the MessagePublisher interface to use the high-level RedisTemplate to publish the message since the RedisTemplate allows arbitrary objects to be passed in as messages:

@Service
public class MessagePublisherImpl implements MessagePublisher {
    
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    @Autowired
    private ChannelTopic topic;

    public MessagePublisherImpl() {
    }

    public MessagePublisherImpl(final RedisTemplate<String, Object> redisTemplate, final ChannelTopic topic) {
        this.redisTemplate = redisTemplate;
        this.topic = topic;
    }

    public void publish(final String message) {
        redisTemplate.convertAndSend(topic.getTopic(), message);
    }

}

We also define this as a bean in RedisConfig:

    @Bean
    MessagePublisher redisPublisher() {
        return new MessagePublisherImpl(redisTemplate(), topic());
    }

Message Listener

In order to subscribe to messages, we need to implement the MessageListener interface: each time a new message arrives, a callback gets invoked and the user code executed through a method named onMessage. This interface gives access to the message, the channel it has been received through, and any pattern used by the subscription to match the channel.

Thus, we create a service class to implement MessageSubscriber:

@Service
public class MessageSubscriber implements MessageListener {
    
    public static List<String> messageList = new ArrayList<String>();

    public void onMessage(final Message message, final byte[] pattern) {
        messageList.add(message.toString());
        System.out.println("Message received: " + new String(message.getBody()));
    }

}

We add a bean definition to RedisConfig:

    @Bean
    MessageListenerAdapter messageListener() {
        return new MessageListenerAdapter(new MessageSubscriber());
    }

6. RedisRepository

Now that we have configured the application to interact with the Redis server, we are going to prepare the application to take example data.

6.1 Model

For this example, we defining a Movie model with two fields:

private String id;
private String name;
//standard getters and setters

6.2 Repository interface

Unlike other Spring Data projects, Spring Data Redis does offer any features to build on top of the other Spring Data interfaces. This is odd for us who have experience with the other Spring Data projects.

Often there is no need to write an implementation of a repository interface with Spring Data projects. We simply just interact with the interface. Spring Data JPA provides numerous repository interfaces that can be extended to get features such as CRUD operations, derived queries, and paging.

So, unfortunately, we need to write our own interface and then define the methods:

public interface RedisRepository {

    Map<Object, Object> findAllMovies();

    void add(Movie movie);

    void delete(String id);

    Movie findMovie(String id);
    
}

6.3 Repository implementation

Our implementation class uses the redisTemplate defined in our configuration class RedisConfig.

We use the HashOperations template that Spring Data Redis offers:

@Repository
public class RedisRepositoryImpl implements RedisRepository {
    private static final String KEY = "Movie";
    
    private RedisTemplate<String, Object> redisTemplate;
    private HashOperations hashOperations;
    
    @Autowired
    public RedisRepositoryImpl(RedisTemplate<String, Object> redisTemplate){
        this.redisTemplate = redisTemplate;
    }

    @PostConstruct
    private void init(){
        hashOperations = redisTemplate.opsForHash();
    }
    
    public void add(final Movie movie) {
        hashOperations.put(KEY, movie.getId(), movie.getName());
    }

    public void delete(final String id) {
        hashOperations.delete(KEY, id);
    }
    
    public Movie findMovie(final String id){
        return (Movie) hashOperations.get(KEY, id);
    }
    
    public Map<Object, Object> findAllMovies(){
        return hashOperations.entries(KEY);
    }

  
}

Let’s take note of the init() method. In this method, we use a function named opsForHash(), which returns the operations performed on hash values bound to the given key. We then use the hashOps, which was defined in init(), for all our CRUD operations.

7. Web interface

In this section, we will review adding Redis CRUD operations capabilities to a web interface.

7.1 Add A Movie

We want to be able to add a Movie in our web page. The Key is the is the Movie id and the Value is the actual object. However, we will later address this so only the Movie name is shown as the value.

So, let’s add a form to a HTML document and assign appropriate names and ids :

      
<form id="addForm">
<div class="form-group">
                    <label for="keyInput">Movie ID (key)</label>
                    <input name="keyInput" id="keyInput" class="form-control"/>
                </div>
<div class="form-group">
                    <label for="valueInput">Movie Name (field of Movie object value)</label>
                    <input name="valueInput" id="valueInput" class="form-control"/>
                </div>
                <button class="btn btn-default" id="addButton">Add</button>
            </form>

Now we use JavaScript to persist the values on form submission:

$(document).ready(function() {
    var keyInput = $('#keyInput'),
        valueInput = $('#valueInput');

    refreshTable();
    $('#addForm').on('submit', function(event) {
        var data = {
            key: keyInput.val(),
            value: valueInput.val()
        };

        $.post('/add', data, function() {
            refreshTable();
            keyInput.val('');
            valueInput.val('');
            keyInput.focus();
        });
        event.preventDefault();
    });

    keyInput.focus();
});

We assign the @RequestMapping value for the POST request, request the key and value, create a Movie object, and save it to the repository:

    @RequestMapping(value = "/add", method = RequestMethod.POST)
    public ResponseEntity<String> add(
        @RequestParam String key,
        @RequestParam String value) {
        
        Movie movie = new Movie(key, value);
        
        redisRepository.add(movie);
        return new ResponseEntity<>(HttpStatus.OK);
    }

7.2 Viewing the content

Once a Movie object is added, we refresh the table to display an updated table. In our JavaScript code block for section 7.1, we called a JavaScript function called refreshTable(). This function performs a GET request to retrieve the current data in the repository:

function refreshTable() {
    $.get('/values', function(data) {
        var attr,
            mainTable = $('#mainTable tbody');
        mainTable.empty();
        for (attr in data) {
            if (data.hasOwnProperty(attr)) {
                mainTable.append(row(attr, data[attr]));
            }
        }
    });
}

The GET request is processed by a method named findAll() that retrieves all the Movie objects stored in the repository and then converts the datatype from Map<Object, Object> to Map<String, String>:

    @RequestMapping("/values")
    public @ResponseBody Map<String, String> findAll() {
        Map<Object, Object> aa = redisRepository.findAllMovies();
        Map<String, String> map = new HashMap<String, String>();
        for(Map.Entry<Object, Object> entry : aa.entrySet()){
            String key = (String) entry.getKey();
            map.put(key, aa.get(key).toString());
        }
        return map;
    }

7.3 Delete a Movie

We write Javascript to do a POST request to /delete, refresh the table, and set keyboard focus to key input:

function deleteKey(key) {
    $.post('/delete', {key: key}, function() {
        refreshTable();
        $('#keyInput').focus();
    });
}

We request the key and delete the object in the redisRepository based on this key:

    @RequestMapping(value = "/delete", method = RequestMethod.POST)
    public ResponseEntity<String> delete(@RequestParam String key) {
        redisRepository.delete(key);
        return new ResponseEntity<>(HttpStatus.OK);
    }

8. Demo

Here we added two movies:
added key value to redis spring boot application

Here we removed one movie:
delete key-value in redis spring boot

9. Conclusion

In this tutorial, we introduced Spring Data Redis and one way of connecting it to a web application to perform CRUD operations.

The source code for the example application is on Github.


AJAX with CKEditor in Spring Boot

1. Overview

In this article, we will cover how to use CKEditor with Spring Boot. In this tutorial, we will be importing an XML document with numerous data, program the ability to load a set of data to the CKEditor instance with a GET request, and do a POST request to save the CKEditor’s data.

Technologies we will be using include MongoDB, Thymeleaf, and Spring Batch.

The full source code for this tutorial is available on Github.

2. What is CKEditor?

CKEditor is a browser-based What-You-See-Is-What-You-Get (WYSIWYG) content editor.  CKEditor aims to bring to a web interface common word processor features found in desktop editing applications like Microsoft Word and OpenOffice.

CKEditor has numerous features for end users in regards to the user interface, inserting content, authoring content, and more.

There are different versions of CKEditor, but for this tutorial we are using CKEditor 4. To see a demo, visit: https://ckeditor.com/ckeditor-4/

3. The XML Document

As mentioned, we are uploading an XML document in this application. The XML data will be inserted into the database and used for the rest of the tutorial.

<?xml version="1.0"?>
<Music xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" id="MUS-1" style="1.1">
<status date="2017-11-07">draft</status>
 <title xmlns:xhtml="http://www.w3.org/1999/xhtml" >Guide to Music I Like - No Specific Genre</title>
  <description xmlns:xhtml="http://www.w3.org/1999/xhtml" >This guide presents a catalog of music that can be found on Spotify. 
    <html:br xmlns:html="http://www.w3.org/1999/xhtml"/>
    <html:br xmlns:html="http://www.w3.org/1999/xhtml"/>
    This is a very small sample of music found on Spotify and is no way to be considered comprehensive.
    </description>
    <songs>
    <song>
    <artist>
    Run the Jewels
    </artist>
    <song-title>Legend Has It</song-title>
    </song>
    <song>
    <artist>
    Kendrick Lamar
    </artist>
    <song-title>ELEMENT.</song-title>
    </song>
    <song>
    <artist>
    Weird Al Yankovic
    </artist>
    <song-title>NOW That's What I Call Polka!</song-title>
    </song>
    <song>
    <artist>
    Eiffel 65
    </artist>
    <song-title>Blue (Da Ba Dee) - DJ Ponte Ice Pop Radio</song-title>
    </song>
    <song>
    <artist>
    YTCracker
    </artist>
    <song-title>Hacker Music</song-title>
    </song>
    <song>
    <artist>
    MAN WITH A MISSION
    </artist>
    <song-title>
    Raise Your Flag
    </song-title>
    </song>
    <song>
    <artist>
    GZA, Method Man
    </artist>
    <song-title>
    Shadowboxin'
    </song-title>
    </song>
    </songs>
</Music>

4. Model

For the above XML code, we can model a Song like this:

public class SongModel {
    @Id
    private String id;
    @Indexed
    private String artist;
    @Indexed
    private String songTitle;
    @Indexed
    private Boolean updated;
    
    public Boolean getUpdated() {
        return updated;
    }
    public void setUpdated(Boolean updated) {
        this.updated = updated;
    }
    public String getArtist() {
        return artist;
    }
    public void setArtist(String artist) {
        this.artist = artist;
    }
    public String getSongTitle() {
        return songTitle;
    }
    public void setSongTitle(String songTitle) {
        this.songTitle = songTitle;
    }
    
    public String getId() {
        return id;
    }
    public void setId(String id) {
        this.id = id;
    }
    
    @JsonCreator
    public SongModel(
        @JsonProperty("artist") String artist,
        @JsonProperty("song-title") String songTitle){
        this.artist = artist;
        this.songTitle = songTitle;
    }
 
    @Override
    public String toString() {
      return "Person [id=" + id + ", artist=" + artist + ", song-title=" + songTitle + "]";
    }

}

For our application, we will be differentiating between an unmodified song and a song that has been modified in CKEditor with a separate Model and Repository.

Let’s now define what an Updated Song is:

public class UpdatedSong {
    
    @Id
    private String id;
    @Indexed
    private String artist;
    @Indexed
    private String songTitle;
    @Indexed
    private String html;
    @Indexed
    private String sid;
    
    public String getSid() {
        return sid;
    }
    public void setSid(String sid) {
        this.sid = sid;
    }
    
    public String getId() {
        return id;
    }
    public void setId(String id) {
        this.id = id;
    }
    public String getArtist() {
        return artist;
    }
    public void setArtist(String artist) {
        this.artist = artist;
    }
    public String getSongTitle() {
        return songTitle;
    }
    public void setSongTitle(String songTitle) {
        this.songTitle = songTitle;
    }
    public String getHtml() {
        return html;
    }
    public void setHtml(String html) {
        this.html = html;
    }

}

5. File Upload and Processing

As this article’s focus is on CKEditor and AJAX, we aren’t going to go into detail on the file upload and processing with Spring Batch. We have reviewed this process in depth in these prior posts, however:

6. Setting Data to CKEditor – GET Request

It is our goal to retrieve the data for an individual song and display that data in CKEditor. There are two issues to tackle: retrieving the data for an individual song and displaying it in CKEditor.

6.1 Client Side Code

In view.html, we use a table in Thymeleaf to iterate through each Song in the Song repository. To be able to retrieve the data from the server for a specific song, we pass in the Song’s id to a function.

Here’s the snippet of code that is responsible for calling the function that retrieves the data from the server and subsequently sets the data to the CKEditor instance:

<table class="table datatable">
<thead>
<tr>
<th>Artist</th>
<th>Song Title</th>
<th>Load</th>
</tr>
</thead>
<tbody>
<tr th:each="songList : ${songList}">
<td th:text="${songList.artist}">Text ...</td>
<td th:text="${songList.songTitle}">Text ...</td>
<td><button th:onclick="|getSong('${songList.id}')|" id="button" class="btn btn-primary btn-condensed">
<i class="glyphicon glyphicon-folder-open"></i>
</button></td>
</tr>
</tbody>
</table>

As we can see, the id of Song is essential for us to be able to retrieve the data.

In the getSong function, we use a deferred promise to ensure that data is set after the GET request:

		function getSong(song) {
			$.ajax({
				url : "/api/show/?sid=" + song,
				type : 'GET',
				dataType : 'text'
			}).then(function(data) {
				var length = data.length-2;
				var datacut = data.slice(9,length);
				CKEDITOR.instances.content.setData(datacut);

			});

			$("#form").attr("action", "/api/save/?sid=" + song);

		};

6.2 Server Side Code

getSong accepts a parameter named sid, which stands for Song id. sid is also a path variable in the @GetMapping. We treat sid as a String because this is the id of the Song from MongoDB.

We check if the Song has been modified and if so we retrieve the associated UpdatedSong entity. If not, we treat the Song differently. Ultimately, we return a simple POJO with a String for data named ResponseModel, however:

    @GetMapping(value={"/show/","/show/{sid}"})
    public ResponseEntity<?> getSong(@RequestParam String sid, Model model){
        ResponseModel response = new ResponseModel();
        System.out.println("SID :::::" + sid);
        ArrayList<String> musicText = new ArrayList<String>();
        if(sid!=null){
            String sidString = sid;
            SongModel songModel = songDAO.findOne(sidString);
            System.out.println("get status of boolean during get ::::::" + songModel.getUpdated());
            if(songModel.getUpdated()==false ){
                
                musicText.add(songModel.getArtist());
                musicText.add(songModel.getSongTitle());
                String filterText = format.changeJsonToHTML(musicText);
                response.setData(filterText);
                
            } else if(songModel.getUpdated()==true){
                UpdatedSong updated = updatedDAO.findBysid(sidString);
                String text = updated.getHtml();
                System.out.println("getting the updated text ::::::::" + text);
                response.setData(text);
            }
            
        }

        model.addAttribute("response", response);
        
        return ResponseEntity.ok(response);
    }

ResponseModel is a very simple POJO as mentioned:

public class ResponseModel {
    private String data;
    
    public ResponseModel(){
            
    }
    
    public ResponseModel(String data){
            this.data = data;
    }

    public String getData() {
            return data;
    }

    public void setData(String data) {
            this.data = data;
    }
}

 

7. Saving CKEditor Data – POST Request

POSTing the data isn’t much of a challenge; however, ensuring that the data is handled appropriately can be.

7.1 Client Side Code

As the CKEditor instance is a textarea within a form, we can trigger a function on a form submit:

$(document)
.ready(
function() {

// SUBMIT FORM
$("#form").submit(function(event) {
// Prevent the form from submitting via the browser.
event.preventDefault();
ajaxPost();
});

ajaxPost() retrieves the current data in the CKEditor and sets it to the variable formData:

function ajaxPost() {

// PREPARE FORM DATA
var formData = CKEDITOR.instances.content
.getData();

// DO POST
$
.ajax({
type : "POST",
contentType : "text/html",
url : $("#form").attr("action"),
data : formData,
dataType : 'text',
success : function(result) {

$("#postResultDiv")
.html(
"

"
+ "Post Successfully! "
+ "

");

console.log(result);
},
error : function(e) {
alert("Error!")
console.log("ERROR: ", e);
}
});

}

})

It is important to note:

  • contentType is “text/html”
  • dataType is “text”

Having the incorrect contentType or dataType can lead to errors or malformed data.

7.2 Server Side Code

We stated in our contentType for the POST request that the mediatype is “text/html”.  We need to specify in our mapping that this will be consumed. Therefore, we add consumes = MediaType.TEXT_HTML_VALUE with our @PostMapping.

Areas for us to note include:

  • @RequestBody String body is responsible for setting the variable body to the content of our request
  • We once again return ResponseModel, the simple POJO described earlier that contains our data
  • We treat a previously modified SongModel different than one that has not been modified before

Also, like the GET request, the sid allows us to deal with the correct Song:

 @PostMapping(value={"/save/","/save/[sid]"}, consumes = MediaType.TEXT_HTML_VALUE)
    public @ResponseBody ResponseModel saveSong( @RequestBody String body, @RequestParam String sid){
        ResponseModel response = new ResponseModel();
        response.setData(body);
        SongModel oldSong = songDAO.findOne(sid);
        String songTitle = oldSong.getSongTitle();
        String artistName = oldSong.getArtist();
        if(oldSong.getUpdated() == false){
            UpdatedSong updatedSong = new UpdatedSong();
            updatedSong.setArtist(artistName);
            updatedSong.setSongTitle(songTitle);
            updatedSong.setHtml(body);
            updatedSong.setSid(sid);
            oldSong.setUpdated(true);
            songDAO.save(oldSong);
            updatedDAO.insert(updatedSong);
            System.out.println("get status of boolean during post :::::" + oldSong.getUpdated());
        }else{
            UpdatedSong currentSong = updatedDAO.findBysid(sid);
            currentSong.setHtml(body);
            updatedDAO.save(currentSong);
        }        
        
        return response;
    }

8. Demo

We visit localhost:8080:

home page for example ckeditor spring boot application

We upload the provided music-example.xml file:

Screen shown after upload for example ckeditor application

We click “Load” for a song:

loaded content into ckeditor

We add content and click “Save”:

we add content and click save ckeditor

If you return to the saved content, you may see “\n”for line breaks. For the time being, discussing this is out of the scope for the tutorial.

9. Conclusion

In this tutorial, we covered how to load  data using a GET request with the object’s id, set the data to the CKEditor instance, and save the CKEditor’s data back to the database with a POST request. There’s extra code, such as using two different entities for the data (an original and a modified version), that isn’t necessary, but hopefully is instructive.

The full code can be found on Github.

Converting XML to JSON + Raw Use in MongoDB + Spring Batch

Overview

Why convert XML to JSON for raw use in MongoDB?

Since MongoDB uses JSON documents in order to store records, just as tables and rows store records in a relational database, we naturally need to convert our XML to JSON.

Some applications may need to store raw (unmodified) JSON because there is uncertainty in how the data will be structured.

There are hundreds of XML-based standards. If an application is to process XML files that do not follow the same standard, there is uncertainty in how the data will be structured.

Why use Spring Batch?

Spring Batch provides reusable functions that are essential in processing large volumes of records and other features that enable high-volume and high performance batch jobs. The Spring Website has documented Spring Batch well.

For another tutorial on Spring Batch, see my previous post on Processing CSVs with Spring Batch.

0 – Converting XML to JSON For Use In MongoDB With Spring Batch Example Application

The example application converts an XML document that is a “policy” for configuring a music playlist. This policy is intended to resemble real cyber security configuration documents. It is a short document but illustrates how you will search complex XML documents.

The approach we will be taking our tutorial is for handling XML files of varying style. We want to be able to handle the unexpected. This is why we are keeping the data “raw.”

View and Download the code from Github

1 – Project Structure

It is a typical Maven structure. We have one package for this example application. The XML file is in src/main/resources.
Project structure of converting xml to json with spring batch mongodb

2 – Project Dependencies

Besides our typical Spring Boot dependencies, we include dependencies for an embedded MongoDB database and for processing JSON.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>michaelcgood-spring-batch-mongodb</artifactId>
	<version>0.0.1</version>
	<packaging>jar</packaging>

	<name>michaelcgood-spring-batch-mongodb</name>
	<description>Michael C  Good - XML to JSON + MongoDB + Spring Batch Example</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.7.RELEASE</version>
		<relativePath /> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-batch</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>de.flapdoodle.embed</groupId>
			<artifactId>de.flapdoodle.embed.mongo</artifactId>
			<version>1.50.5</version>
		</dependency>
		<dependency>
			<groupId>cz.jirutka.spring</groupId>
			<artifactId>embedmongo-spring</artifactId>
			<version>RELEASE</version>
		</dependency>
		<dependency>
				<groupId>org.json</groupId>
				<artifactId>json</artifactId>
				<version>20170516</version>
			</dependency>

			<dependency>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-starter-data-mongodb</artifactId>
			</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>


</project>

3 – XML Document

This is the example policy document created for this tutorial. It’s structure is based on real cyber security policy documents.

  • Note the parent of the document is the Policy tag.
  • Important information lies within the Group tag.
  • Look at the values that reside within the tags, such as the id in Policy or the date within status.

There’s a lot of information condensed in this small document to consider. For instance, there is also the XML namespace (xmlns). We won’t touch on this in the rest of the tutorial, but depending on your goals it could be something to add logic for.

<?xml version="1.0"?>
<Policy  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" style="STY_1.1" id="NRD-1">
  <status date="2017-10-18">draft</status>
  <title xmlns:xhtml="http://www.w3.org/1999/xhtml">Guide to the Configuration of Music Playlist</title>
   <description xmlns:xhtml="http://www.w3.org/1999/xhtml" >This guide presents a catalog of relevant
    configuration settings for a playlist that I listen to while I work on software development.
    <html:br xmlns:html="http://www.w3.org/1999/xhtml"/>
    <html:br xmlns:html="http://www.w3.org/1999/xhtml"/>
    Providing myself with such guidance reminds me how to efficiently
    configure my playlist.  Lorem ipsum <html:i xmlns:html="http://www.w3.org/1999/xhtml">Lorem ipsum,</html:i> 
    and Lorem ipsum.  Some example
    <html:i xmlns:html="http://www.w3.org/1999/xhtml">Lorem ipsum</html:i>, which are Lorem ipsum.
  </description>
  <Group id="remediation_functions">
    <title xmlns:xhtml="http://www.w3.org/1999/xhtml" >Remediation functions used by the SCAP Security Guide Project</title>
    <description xmlns:xhtml="http://www.w3.org/1999/xhtml" >XCCDF form of the various remediation functions as used by
      remediation scripts from the SCAP Security Guide Project</description>
    <Value id="is_the_music_good" prohibitChanges="true" >
      <title xmlns:xhtml="http://www.w3.org/1999/xhtml" >Remediation function to fix bad playlist</title>
      <description xmlns:xhtml="http://www.w3.org/1999/xhtml" >Function to fix bad playlist.
      
        
       Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
       
       Lorem ipsum
       Lorem ipsum
       Lorem ipsum
       Lorem ipsum
      </description>
      <value>
        function fix_bad_playlist {
        
        # Load function arguments into local variables
       Lorem ipsum
       Lorem ipsum
       Lorem ipsum
        
        # Check sanity of the input
        if [ $# Lorem ipsum ]
        then
        echo "Usage: Lorem ipsum"
        echo "Aborting."
        exit 1
        fi
        
        }
      </value>
    </Value>
    </Group>
    </Policy>

4 – MongoDB Configuration

Below we specify that we are using an embedded MongoDB database, make it discoverable for a component scan that is bundled in the convenience annotation @SpringBootApplication, and specify that mongoTemplate will be a bean.

package com.michaelcgood;

import java.io.IOException;
import cz.jirutka.spring.embedmongo.EmbeddedMongoFactoryBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.*;
import com.mongodb.MongoClient;
 
 
@Configuration
public class MongoConfig {
 
    private static final String MONGO_DB_URL = "localhost";
    private static final String MONGO_DB_NAME = "embeded_db";
    @Bean
    public MongoTemplate mongoTemplate() throws IOException {
        EmbeddedMongoFactoryBean mongo = new EmbeddedMongoFactoryBean();
        mongo.setBindIp(MONGO_DB_URL);
        MongoClient mongoClient = mongo.getObject();
        MongoTemplate mongoTemplate = new MongoTemplate(mongoClient, MONGO_DB_NAME);
        return mongoTemplate;
    }
}



5 – Processing XML to JSON

step1() of our Spring Batch Job contains calls three methods to help process the XML to JSON. We will review each individually.

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .tasklet(new Tasklet() {
                    @Override
                    public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
                        
                        // get path of file in src/main/resources
                        Path xmlDocPath =  Paths.get(getFilePath());
                        
                        // process the file to json
                         String json = processXML2JSON(xmlDocPath);
                         
                         // insert json into mongodb
                         insertToMongo(json);
                        return RepeatStatus.FINISHED;
                    }
                }).build();
    }

5.1 – getFilePath()

This method simply gets the file path that is passed as a parameter to the method processXML2JSON.
Note:

  • ClassLoader is helping us locate the XML file in our resources folder.
 // no parameter method for creating the path to our xml file
    private String getFilePath(){
        
        String fileName = "FakePolicy.xml";
        ClassLoader classLoader = getClass().getClassLoader();
        File file = new File(classLoader.getResource(fileName).getFile());
        String xmlFilePath = file.getAbsolutePath();
        
        return xmlFilePath;
    }

5.2 – processXML2JSON(xmlDocPath)

The string returned by getFilePath is passed into this method as a parameter. A JSONOBject is created from a String of the XML file.

   // takes a parameter of xml path and returns json as a string
    private String processXML2JSON(Path xmlDocPath) throws JSONException {
        
        
        String XML_STRING = null;
        try {
            XML_STRING = Files.lines(xmlDocPath).collect(Collectors.joining("\n"));
        } catch (IOException e) {
            e.printStackTrace();
        }
        
        JSONObject xmlJSONObj = XML.toJSONObject(XML_STRING);
        String jsonPrettyPrintString = xmlJSONObj.toString(PRETTY_PRINT_INDENT_FACTOR);
        System.out.println("PRINTING STRING :::::::::::::::::::::" + jsonPrettyPrintString);
        
        return jsonPrettyPrintString;
    }

5.3 – insertToMongo(json)

We insert the parsed JSON into a MongoDB document. We then insert this document with the help of the @Autowired mongoTemplate into a collection named “foo”.

   // inserts to our mongodb
    private void insertToMongo(String jsonString){
        Document doc = Document.parse(jsonString);
        mongoTemplate.insert(doc, "foo");
    }

6 – Querying MongoDB

step2() of our Spring Batch Job contains our MongoDB queries.

  • mongoTemplate.collectionExists returns a Boolean value based on the existence of the collection.
  • mongoTemplate.getCollection(“foo”).find() returns all the documents within the collection.
  • alldocs.toArray() returns an array of DBObjects.
  • Then we call three methods that we will review individually below.
 public Step step2(){
        return stepBuilderFactory.get("step2")
            .tasklet(new Tasklet(){
            @Override
            public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception{
                // all printing out to console removed for post's brevity
                // checks if our collection exists
                Boolean doesexist = mongoTemplate.collectionExists("foo");
                
                // show all DBObjects in foo collection
                DBCursor alldocs = mongoTemplate.getCollection("foo").find();
                List<DBObject> dbarray = alldocs.toArray();
                
                // execute the three methods we defined for querying the foo collection
                String result = doCollect();
                String resultTwo = doCollectTwo();
                String resultThree = doCollectThree();
               
                return RepeatStatus.FINISHED;
            }
        }).build();
    }

6.1 – First query

The goal of this query is to find a document where style=”STY_1.1″. To accomplish this, we need to remember where style resides in the document. It is a child of Policy; therefore, we address it in the criteria as Policy.style.

The other goal of this query is to only return the id field of the Policy. It is also just a child of Policy.

The result is returned by calling this method: mongoTemplate.findOne(query, String.class, “foo”);. The output is a String, so the second parameter is String.class. The third parameter is our collection name.

    public String doCollect(){
        Query query = new Query();
        query.addCriteria(Criteria.where("Policy.style").is("STY_1.1")).fields().include("Policy.id");
        String result = mongoTemplate.findOne(query, String.class, "foo");
        return result;
    }

6.2 – Second query

The difference between the second query and the first query are the fields returned. In the second query, we return Value, which is a child of both Policy and Group.

    public String doCollectTwo(){
        Query query = new Query();
        query.addCriteria(Criteria.where("Policy.style").is("STY_1.1")).fields().include("Policy.Group.Value");
        String result = mongoTemplate.findOne(query, String.class, "foo");
        
        return result;
    }

6.3 – Third query

The criteria for the third query is different. We only want to return a document with the id “NRD-1” and a status date of “2017-10-18”. We only want to return two fields: title and description, which are both children of Value.

Referring to the XML document or the printed JSON in the demo below for further clarification on the queries.

    public String doCollectThree(){
        Query query = new Query();
        query.addCriteria(Criteria.where("Policy.id").is("NRD-1").and("Policy.status.date").is("2017-10-18")).fields().include("Policy.Group.Value.title").include("Policy.Group.Value.description");
        String result = mongoTemplate.findOne(query, String.class, "foo");
        
        return result;
    }

7 – Spring Batch Job

The Job begins with step1 and calls step2 next.

 @Bean
    public Job xmlToJsonToMongo() {
        return jobBuilderFactory.get("XML_Processor")
                .start(step1())
                .next(step2())
                .build();
    }

8 – @SpringBootApplication

This is a standard class with static void main and @SpringBootApplication.

package com.michaelcgood;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;

@SpringBootApplication
@EnableAutoConfiguration(exclude={DataSourceAutoConfiguration.class})
public class SpringBatchMongodb {

	public static void main(String[] args) {
		SpringApplication.run(SpringBatchMongodb.class, args);
	}
}

9 – Demo

9.1 – step1

The JSON is printed as a String. I have cut the output past description below because it is long.

Executing step: [step1]
PRINTING STRING :::::::::::::::::::::{"Policy": {
    "Group": {
        "Value": {
            "prohibitChanges": true,
            "description": {

9.2 – step2

I have cut the results to format the output for the blog post.

Executing step: [step2]

Checking if the collection exists

Status of collection returns :::::::::::::::::::::true

Show all objects

list of db objects returns:::::::::::::::::::::[{ "_id" : { "$oid" : "59e7c0324ad9510acf5773c0"} , [..]

Just return the id of Policy

RESULT:::::::::::::::::::::{ "_id" : { "$oid" : "59e7c0324ad9510acf5773c0"} , "Policy" : { "id" : "NRD-1"}}

To see the other results printed to the console, fork/download the code from Github and run the application.

10 – Conclusion

We have reviewed how to convert XML to JSON, store the JSON to MongoDB, and how to query the database for specific results.

Further reading:

The source code is on Github



Spring Batch CSV Processing

Overview

Topics we will be discussing include the essential concepts of batch processing with Spring Batch and how to import the data from a CSV into a database.

0 – Spring Batch CSV Processing Example Application

We are building an application that demonstrates the basics of Spring Batch for processing CSV files. Our demo application will allow us to process a CSV file that contains hundreds of records of Japanese anime titles.

0.1 – The CSV

I have downloaded the CSV we will be using from this Github repository, and it provides a pretty comprehensive list of animes.

Here is a screenshot of the CSV open in Microsoft Excel

Animes CSV screenshot

View and Download the code from Github

1 – Project Structure

Project structure of spring batch application

2 – Project Dependencies

Besides typical Spring Boot dependencies, we include spring-boot-starter-batch, which is the dependency for Spring Batch as the name suggests, and hsqldb for an in-memory database. We also include commons-lang3 for ToStringBuilder.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>michaelcgood-spring-batch-csv</artifactId>
	<version>0.0.1</version>
	<packaging>jar</packaging>

	<name>michaelcgood-spring-batch-csv</name>
	<description>Michael C  Good - Spring Batch CSV Example Application</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.7.RELEASE</version>
		<relativePath /> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-batch</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-jpa</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>

		<dependency>
			<groupId>org.hsqldb</groupId>
			<artifactId>hsqldb</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
		<dependency>
			<groupId>org.apache.commons</groupId>
			<artifactId>commons-lang3</artifactId>
			<version>3.6</version>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>


</project>

3 – Model

This is a POJO that models the fields of an anime. The fields are:

  • ID. For the sake of simplicity, we treat the ID as a String. However, this could be changed to another data type such as an Integer or Long.
  • Title. This is the title of the anime and it is appropriate for it to be a String.
  • Description. This is the description of the anime, which is longer than the title, and it can also be treated as a String.

What is important to note is our class constructor for the three fields: public AnimeDTO(String id, String title, String description). This will be used in our application. Also, as usual, we need to make a default constructor with no parameters or else Java will throw an error.

package com.michaelcgood;

import org.apache.commons.lang3.builder.ToStringBuilder;
/**
 * Contains the information of a single anime
 *
 * @author Michael C Good michaelcgood.com
 */

public class AnimeDTO {
	
	public String getId() {
		return id;
	}

	public void setId(String id) {
		this.id = id;
	}

	public String getTitle() {
		return title;
	}

	public void setTitle(String title) {
		this.title = title;
	}

	public String getDescription() {
		return description;
	}

	public void setDescription(String description) {
		this.description = description;
	}



	private String id;
	



	private String title;
	private String description;
	
	public AnimeDTO(){
		
	}
	
	public AnimeDTO(String id, String title, String description){
		this.id = id;
		this.title = title;
		this.description = title;
	}
	

	
	   @Override
	    public String toString() {
		   return new ToStringBuilder(this)
				   .append("id", this.id)
				   .append("title", this.title)
				   .append("description", this.description)
				   .toString();
	   }


}

4 – CSV File to Database Configuration

There is a lot going on in this class and it is not all written at once, so we are going to go through the code in steps. Visit Github to see the code in its entirety.

4.1 – Reader

As the Spring Batch documentation states FlatFileIteamReader will “read lines of data from a flat file that typically describe records with fields of data defined by fixed positions in the file or delimited by some special character (e.g. Comma)”.

We are dealing with a CSV, so of course the data is delimited by a comma, making this the perfect for use with our file.

   @Bean
    public FlatFileItemReader<AnimeDTO> csvAnimeReader(){
        FlatFileItemReader<AnimeDTO> reader = new FlatFileItemReader<AnimeDTO>();
        reader.setResource(new ClassPathResource("animescsv.csv"));
        reader.setLineMapper(new DefaultLineMapper<AnimeDTO>() {{
            setLineTokenizer(new DelimitedLineTokenizer() {{
                setNames(new String[] { "id", "title", "description" });
            }});
            setFieldSetMapper(new BeanWrapperFieldSetMapper<AnimeDTO>() {{
                setTargetType(AnimeDTO.class);
            }});
        }});
        return reader;
    }

Important points:

  • FlatFileItemReader is parameterized with a model. In our case, this is AnimeDTO.
  • FlatFileItemReader must set a resource. It uses setResource method. Here we set the resource to animescsv.csv
  • setLineMapper method converts Strings to objects representing the item. Our String will be an anime record consisting of an id, title, and description. This String is made into an object. Note that DefaultLineMapper is parameterized with our model, AnimeDTO.
  • However, LineMapper is given a raw line, which means there is work that needs to be done to map the fields appropriately. The line must be tokenized into a FieldSet, which DelimitedLineTokenizer takes care of. DelimitedLineTokenizer returns a FieldSet.
  • Now that we have a FieldSet, we need to map it. setFieldSetMapper is used for taking the FieldSet object and mapping its contents to a DTO, which is AnimeDTO in our case.
  • 4.2 – Processor

    If we want to transform the data before writing it to the database, an ItemProcessor is necessary. Our code does not actually apply any business logic to transform the data, but we allow for the capability to.

    4.2.1 – Processor in CsvFileToDatabaseConfig.Java

    csvAnimeProcessor returns a new instance of the AnimeProcessor object which we review below.

    	@Bean
    	ItemProcessor<AnimeDTO, AnimeDTO> csvAnimeProcessor() {
    		return new AnimeProcessor();
    	}
    

    4.2.2 – AnimeProcessor.Java

    If we wanted to apply business logic before writing to the database, you could manipulate the Strings before writing to the database. For instance, you could add toUpperCase() after getTitle to make the title upper case before writing to the database. However, I decided not to do that or apply any other business logic for this example processor, so no manipulation is being done. The Processor is here simply for demonstration.

    package com.michaelcgood;
    
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    
    import org.springframework.batch.item.ItemProcessor;
    
    public class AnimeProcessor implements ItemProcessor<AnimeDTO, AnimeDTO> {
    	
        private static final Logger log = LoggerFactory.getLogger(AnimeProcessor.class);
        
        @Override
        public AnimeDTO process(final AnimeDTO AnimeDTO) throws Exception {
        	
        	final String id = AnimeDTO.getId();
            final String title = AnimeDTO.getTitle();
            final String description = AnimeDTO.getDescription();
    
            final AnimeDTO transformedAnimeDTO = new AnimeDTO(id, title, description);
    
            log.info("Converting (" + AnimeDTO + ") into (" + transformedAnimeDTO + ")");
    
            return transformedAnimeDTO;
        }
    
    }
    
    

    4.3 – Writer

    The csvAnimeWriter method is responsible for actually writing the values into our database. Our database is an in-memory HSQLDB however this application allows us to easily swap out one database for another. The dataSource is autowired.

    	@Bean
    	public JdbcBatchItemWriter<AnimeDTO> csvAnimeWriter() {
    		 JdbcBatchItemWriter<AnimeDTO> excelAnimeWriter = new JdbcBatchItemWriter<AnimeDTO>();
    		 excelAnimeWriter.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<AnimeDTO>());
    		 excelAnimeWriter.setSql("INSERT INTO animes (id, title, description) VALUES (:id, :title, :description)");
    		 excelAnimeWriter.setDataSource(dataSource);
    	        return excelAnimeWriter;
    	}
    

    4.4 – Step

    A Step is a domain object that contains an independent, sequential phase of a batch job and contains all of the information needed to define and control the actual batch processing.

    Now that we’ve created the reader and processor for data we need to write it. For the reading, we’ve been using chunk-oriented processing, meaning we’ve been reading the data one at a time. Chunk-oriented processing also includes creating ‘chunks’ that will be written out, within a transaction boundary. For chunk-oriented processing, you set a commit interval and once the number of items read equals the commit interval that has been set, the entire chunk is written out via the ItemWriter, and the transaction is committed. We set the chunk interval size to 1.

    I suggest reading the Spring Batch documentation about chunk-oriented processing.

    Then the reader, processor, and writer call the methods we wrote.

    	@Bean
    	public Step csvFileToDatabaseStep() {
    		return stepBuilderFactory.get("csvFileToDatabaseStep")
    				.<AnimeDTO, AnimeDTO>chunk(1)
    				.reader(csvAnimeReader())
    				.processor(csvAnimeProcessor())
    				.writer(csvAnimeWriter())
    				.build();
    	}
    

    4.5 – Job

    A Job consists of Steps. We pass a parameter into the Job below because we want to track the completion of the Job.

    	@Bean
    	Job csvFileToDatabaseJob(JobCompletionNotificationListener listener) {
    		return jobBuilderFactory.get("csvFileToDatabaseJob")
    				.incrementer(new RunIdIncrementer())
    				.listener(listener)
    				.flow(csvFileToDatabaseStep())
    				.end()
    				.build();
    	}
    



    5 – Job Completion Notification Listener

    The class below autowires the JdbcTemplate because we’ve already set the dataSource and we want to easily make our query. The results of our are query are a list of AnimeDTO objects. For each object returned, we will create a message in our console to show that the item has been written to the database.

    package com.michaelcgood;
    
    import java.sql.ResultSet;
    import java.sql.SQLException;
    import java.util.List;
    
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    
    import org.springframework.batch.core.BatchStatus;
    import org.springframework.batch.core.JobExecution;
    import org.springframework.batch.core.listener.JobExecutionListenerSupport;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.jdbc.core.JdbcTemplate;
    import org.springframework.jdbc.core.RowMapper;
    import org.springframework.stereotype.Component;
    
    @Component
    public class JobCompletionNotificationListener extends JobExecutionListenerSupport {
    
    	private static final Logger log = LoggerFactory.getLogger(JobCompletionNotificationListener.class);
    
    	private final JdbcTemplate jdbcTemplate;
    
    	@Autowired
    	public JobCompletionNotificationListener(JdbcTemplate jdbcTemplate) {
    		this.jdbcTemplate = jdbcTemplate;
    	}
    
    	@Override
    	public void afterJob(JobExecution jobExecution) {
    		if(jobExecution.getStatus() == BatchStatus.COMPLETED) {
    			log.info("============ JOB FINISHED ============ Verifying the results....\n");
    
    			List<AnimeDTO> results = jdbcTemplate.query("SELECT id, title, description FROM animes", new RowMapper<AnimeDTO>() {
    				@Override
    				public AnimeDTO mapRow(ResultSet rs, int row) throws SQLException {
    					return new AnimeDTO(rs.getString(1), rs.getString(2), rs.getString(3));
    				}
    			});
    
    			for (AnimeDTO AnimeDTO : results) {
    				log.info("Discovered <" + AnimeDTO + "> in the database.");
    			}
    
    		}
    	}
    	
    }
    
    

    6 – SQL

    We need to create a schema for our database. As mentioned, we have made all fields Strings for ease of use, so we have made their data types VARCHAR.

    DROP TABLE animes IF EXISTS;
    CREATE TABLE animes  (
        id VARCHAR(10),
        title VARCHAR(400),
        description VARCHAR(999)
    );
    
    

    6 – Main

    This is a standard class with main(). As the Spring Documentation states, @SpringBootApplication is a convenience annotation that includes @Configuration, @EnableAutoConfiguration, @EnableWebMvc, and @ComponentScan.

    package com.michaelcgood;
    
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    
    @SpringBootApplication
    public class SpringBatchCsvApplication {
    
    	public static void main(String[] args) {
    		SpringApplication.run(SpringBatchCsvApplication.class, args);
    	}
    }
    
    

    7 – Demo

    7.1 – Converting

    The FieldSet is fed through the processor and “Converting” is printed to the console.
    Converting CSV to database in Spring Batch

    7.2 – Discovering New Items In Database

    When the Spring Batch Job is finished, we select all the records and print them out to the console individually.
    Discovering newly imported items in database in Spring Batch application

    7.3 – Batch Process Complete

    When the Batch Process is complete this is what is printed to the console.

    Job: [FlowJob: [name=csvFileToDatabaseJob]] completed with the following parameters: [{run.id=1, -spring.output.ansi.enabled=always}] and the following status: [COMPLETED]
    Started SpringBatchCsvApplication in 36.0 seconds (JVM running for 46.616)
    

    8 – Conclusion

    Spring Batch builds upon the POJO-based development approach and user-friendliness of the Spring Framework’s to make it easy for developers to create enterprise grade batch processing.

    The source code is on Github



Validation in Thymeleaf + Spring

Overview

Important topics we will be discussing are dealing with null values, empty strings, and validation of input so we do not enter invalid data into our database.

In dealing with null values, we touch on use of java.util.Optional which was introduced in Java 1.8.

0 – Spring Boot + Thymeleaf Example Form Validation Application

We are building a web application for a university that allows potential students to request information on their programs.

View and Download the code from Github

1 – Project Structure

Thymeleaf validation application project structure

2 – Project Dependencies

Besides our typical Spring Boot dependencies, we are using an embedded HSQLDB database and nekohtml for LEGACYHTML5 mode.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>michaelcgood-validation-thymeleaf</artifactId>
	<version>0.0.1</version>
	<packaging>jar</packaging>

	<name>michaelcgood-validation-thymeleaf</name>
	<description>Michael C  Good - Validation in Thymeleaf Example Application</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.7.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-jpa</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-thymeleaf</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>

		<dependency>
			<groupId>org.hsqldb</groupId>
			<artifactId>hsqldb</artifactId>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<!-- legacy html allow -->
		<dependency>
			<groupId>net.sourceforge.nekohtml</groupId>
			<artifactId>nekohtml</artifactId>
			<version>1.9.21</version>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>


</project>

3 – Model

In our model we define:

  • An autogenerated id field
  • A name field that cannot be null
  • That the name must be between 2 and 40 characters
  • An email field that is validated by the @Email annotation
  • A boolean field “openhouse” that allows a potential student to indicate if she wants to attend an open house
  • A boolean field “subscribe” for subscribing to email updates
  • A comments field that is optional, so there is no minimum character requirement but there is a maximum character requirement
package com.michaelcgood.model;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.validation.constraints.NotNull;
import javax.validation.constraints.Size;

import org.hibernate.validator.constraints.Email;

@Entity
public class Student {

	@Id
	@GeneratedValue(strategy = GenerationType.AUTO)
	private Long id;
	@NotNull
    @Size(min=2, max=40)
	private String name;
	@NotNull
	@Email
	private String email;
	private Boolean openhouse;
	private Boolean subscribe;
	 @Size(min=0, max=300)
	private String  comments;
	
	public Long getId() {
		return id;
	}
	public void setId(Long id) {
		this.id = id;
	}
	public String getName() {
		return name;
	}
	public void setName(String name) {
		this.name = name;
	}
	public String getEmail() {
		return email;
	}
	public void setEmail(String email) {
		this.email = email;
	}
	public Boolean getOpenhouse() {
		return openhouse;
	}
	public void setOpenhouse(Boolean openhouse) {
		this.openhouse = openhouse;
	}
	public Boolean getSubscribe() {
		return subscribe;
	}
	public void setSubscribe(Boolean subscribe) {
		this.subscribe = subscribe;
	}
	public String getComments() {
		return comments;
	}
	public void setComments(String comments) {
		this.comments = comments;
	}
	

}

4 – Repository

We define a repository.

package com.michaelcgood.dao;

import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;

import com.michaelcgood.model.Student;

@Repository
public interface StudentRepository extends JpaRepository<Student,Long> {

}

5 – Controller

We register StringTrimmerEditor to convert empty Strings to null values automatically.

When a user sends a POST request, we want to receive the value of that Student object, so we use @ModelAttribute to do just that.

To ensure that the user is sending values that are valid, we use the appropriately named @Valid annotation next.

BindingResult must follow next, or else the user is given an error page when submitting invalid data instead of remaining on the form page.

We use if…else to control what happens when a user submits a form. If the user submits invalid data, the user will remain on the current page and nothing more will occur on the server side. Otherwise, the application will consume the user’s data and the user can proceed.

At this point, it is kind of redundant to check if the student’s name is null, but we do. Then, we call the method checkNullString, which is defined below, to see if the comment field is an empty String or null.

package com.michaelcgood.controller;

import java.util.Optional;

import javax.validation.Valid;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.propertyeditors.StringTrimmerEditor;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.validation.BindingResult;
import org.springframework.web.bind.WebDataBinder;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.InitBinder;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.PostMapping;
import com.michaelcgood.dao.StudentRepository;
import com.michaelcgood.model.Student;

@Controller
public class StudentController {
	@InitBinder
	public void initBinder(WebDataBinder binder) {
	    binder.registerCustomEditor(String.class, new StringTrimmerEditor(true));
	}
	public String finalString = null;
	@Autowired
	private StudentRepository studentRepository;
	@PostMapping(value="/")
	public String addAStudent(@ModelAttribute @Valid Student newStudent, BindingResult bindingResult, Model model){
		if (bindingResult.hasErrors()) {
			System.out.println("BINDING RESULT ERROR");
			return "index";
		} else {
			model.addAttribute("student", newStudent);

			if (newStudent.getName() != null) {
				try {
					// check for comments and if not present set to 'none'
					String comments = checkNullString(newStudent.getComments());
					if (comments != "None") {
						System.out.println("nothing changes");
					} else {
						newStudent.setComments(comments);
					}
				} catch (Exception e) {

					System.out.println(e);

				}
				studentRepository.save(newStudent);
				System.out.println("new student added: " + newStudent);
			}

			return "thanks";
		}
	}
	
	@GetMapping(value="thanks")
	public String thankYou(@ModelAttribute Student newStudent, Model model){
		model.addAttribute("student",newStudent);
		
		return "thanks";
	}
	
	@GetMapping(value="/")
	public String viewTheForm(Model model){
		Student newStudent = new Student();
		model.addAttribute("student",newStudent);
		return "index";
	}
	
	public String checkNullString(String str){
		String endString = null;
		if(str == null || str.isEmpty()){
			System.out.println("yes it is empty");
			str = null;
			Optional<String> opt = Optional.ofNullable(str);
			endString = opt.orElse("None");
			System.out.println("endString : " + endString);
		}
		else{
			; //do nothing
		}
		
		
		return endString;
		
	}

}

Optional.ofNullable(str); means that the String will become the data type Optional, but the String may be a null value.

endString = opt.orElse(“None”); sets the String value to “None” if the variable opt is null.


6 – Thymeleaf Templates

As you saw in our Controller’s mapping above, there are two pages. The index.html is our main page that has the form for potential University students.

Our main object is Student, so of course our th:object refers to that. Our model’s fields respectively go into th:field.

We wrap our form’s inputs inside a table for formatting purposes.

Below each table cell (td) we have a conditional statement like this one: […]
th:if=”${#fields.hasErrors(‘name’)}” th:errors=”*{name}”
[…]

The above conditional statement means if the user inputs data into that field that doesn’t match the requirement we put for that field in our Student model and then submits the form, show the input requirements when the user is returned to this page.

index.html

<html xmlns="http://www.w3.org/1999/xhtml"
	xmlns:th="http://www.thymeleaf.org">

<head>
<!-- CSS INCLUDE -->
<link rel="stylesheet"
	href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
	integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u"
	crossorigin="anonymous"></link>

<!-- EOF CSS INCLUDE -->
</head>
<body>

	<!-- START PAGE CONTAINER -->
	<div class="container-fluid">
		<!-- PAGE TITLE -->
		<div class="page-title">
			<h2>
				<span class="fa fa-arrow-circle-o-left"></span> Request University
				Info
			</h2>
		</div>
		<!-- END PAGE TITLE -->
		<div class="column">
			<form action="#" th:action="@{/}" th:object="${student}"
				method="post">
				<table>
					<tr>
						<td>Name:</td>
						<td><input type="text" th:field="*{name}"></input></td>
						<td th:if="${#fields.hasErrors('name')}" th:errors="*{name}">Name
							Error</td>
					</tr>
					<tr>
						<td>Email:</td>
						<td><input type="text" th:field="*{email}"></input></td>
						<td th:if="${#fields.hasErrors('email')}" th:errors="*{email}">Email
							Error</td>
					</tr>
					<tr>
						<td>Comments:</td>
						<td><input type="text" th:field="*{comments}"></input></td>
					</tr>
					<tr>
						<td>Open House:</td>
						<td><input type="checkbox" th:field="*{openhouse}"></input></td>
				
					</tr>
					<tr>
						<td>Subscribe to updates:</td>
						<td><input type="checkbox" th:field="*{subscribe}"></input></td>
				
					</tr>
					<tr>
						<td>
							<button type="submit" class="btn btn-primary">Submit</button>
						</td>
					</tr>
				</table>
			</form>

		</div>
		<!-- END PAGE CONTENT -->
		<!-- END PAGE CONTAINER -->
	</div>
	<script src="https://code.jquery.com/jquery-1.11.1.min.js"
		integrity="sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE="
		crossorigin="anonymous"></script>


	<script
		src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"
		integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa"
		crossorigin="anonymous"></script>


</body>
</html>

Here we have the page that a user sees when they have successfully completed the form. We use th:textto show the user the text he or she input for that field.

thanks.html

<html xmlns="http://www.w3.org/1999/xhtml"
	xmlns:th="http://www.thymeleaf.org">

<head>
<!-- CSS INCLUDE -->
<link rel="stylesheet"
	href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
	integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u"
	crossorigin="anonymous"></link>

<!-- EOF CSS INCLUDE -->


</head>
<body>

	<!-- START PAGE CONTAINER -->
	<div class="container-fluid">

		<!-- PAGE TITLE -->
		<div class="page-title">
			<h2>
				<span class="fa fa-arrow-circle-o-left"></span> Thank you
			</h2>
		</div>
		<!-- END PAGE TITLE -->
		<div class="column">
			<table class="table datatable">
				<thead>
					<tr>
						<th>Name</th>
						<th>Email</th>
						<th>Open House</th>
						<th>Subscribe</th>
						<th>Comments</th>
					</tr>
				</thead>
				<tbody>
					<tr th:each="student : ${student}">
						<td th:text="${student.name}">Text ...</td>
						<td th:text="${student.email}">Text ...</td>
						<td th:text="${student.openhouse}">Text ...</td>
						<td th:text="${student.subscribe}">Text ...</td>
						<td th:text="${student.comments}">Text ...</td>
					</tr>
				</tbody>
			</table>
		</div>	
		</div>
		<!-- END PAGE CONTAINER -->
	</div>
		<script
  src="https://code.jquery.com/jquery-1.11.1.min.js"
  integrity="sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE="
  crossorigin="anonymous"></script>
 

	<script
		src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"
		integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa"
		crossorigin="anonymous"></script>

</body>
</html>

7 – Configuration

Using Spring Boot Starter and including Thymeleaf dependencies, you will automatically have a templates location of /templates/, and Thymeleaf just works out of the box. So most of these settings aren’t needed.

The one setting to take note of is LEGACYHTM5 which is provided by nekohtml. This allows us to use more casual HTML5 tags if we want to. Otherwise, Thymeleaf will be very strict and may not parse your HTML. For instance, if you do not close an input tag, Thymeleaf will not parse your HTML.

application.properties

#==================================
# = Thymeleaf configurations 
#==================================
spring.thymeleaf.check-template-location=true
spring.thymeleaf.prefix=classpath:/templates/
spring.thymeleaf.suffix=.html
spring.thymeleaf.content-type=text/html
spring.thymeleaf.cache=false
spring.thymeleaf.mode=LEGACYHTML5

server.contextPath=/

8 – Demo

Home page

Here we arrive on the home page.
Home page of example Thymeleaf validation application

Invalid data

I input invalid data into the name field and email field.

Invalid data in example Thymeleaf validation

Valid data with no comment

Now I put valid data in all fields, but do not provide a comment. It is not required to provide a comment. In our controller, we made all empty Strings null values. If the user did not provide a comment, the String value is made “None”.

Comments set to none Thymeleaf

9 – Conclusion

Wrap up

This demo application demonstrated how to valid user input in a Thymeleaf form.
In my opinion, Spring and Thymeleaf work well with javax.validation.constraints for validating user input.
The source code is on Github

Notes

Java 8’s Optional was sort of forced into this application for demonstration purposes, and I want to note it works more organically when using @RequestParam as shown in my PagingAndSortingRepository tutorial.

However, if you were not using Thymeleaf, you could have possibly made our not required fields Optional. Here Vlad Mihalcea discusses the best way to map Optional entity attribute with JPA and Hibernate.


Building Spring Boot RESTful Service + Spring Boot Actuator

Overview

What is REST?

REST(REpresentational State Transfer) is the architectural style the web is built on and has become a standard software design pattern used for web applications. The term Representational State Transfer was first used by Roy Fielding, the originator of REST and one of the principal authors of HTTP specification, in his doctoral dissertation.

There are many good references on REST including:

This tutorial is based on Building Rest Services with Spring and the beginning of the tutorial has a good overview of REST as well.

What is Spring Boot Actuator?

Spring Boot Actuator is a sub-project of Spring Boot. It adds several production grade services to your application with minimal effort on your part.

Definition of Actuator

An actuator is a component responsible for moving or controlling a system.
The term actuator is not limited to Spring Boot; however, that is our focus here.

After Actuator is configured in your Spring Boot application, it allows you to interact and monitor your application by invoking different technology agnostic endpoints exposed by Spring Boot Actuator such as application health, beans, loggers, mappings, and trace. More are listed in this Spring doc.

0 – Spring Boot RESTful Web Service with Actuator Example Application

We will build an example RESTful web application with Spring Boot and Actuator.

The application will be a “username tracker.” In this application, a person has one account and their account may have many usernames.

View and Download the code from Github

1 – Project Structure

As usual, we have a normal Maven project structure.
project structure of spring boot actuator rest example

2 – Project Dependencies

Besides typical Spring Boot dependencies, we are including HSQLDB for our embedded database and spring-boot-starter-actuator for all the Actuator dependencies.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>michaelcgood-springbootactuator</artifactId>
	<version>0.0.1</version>
	<packaging>jar</packaging>

	<name>Spring-Boot-Actuator-Example</name>
	<description>Michael C  Good - Spring Boot Actuator Example</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.6.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		 <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
			<groupId>org.hsqldb</groupId>
			<artifactId>hsqldb</artifactId>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-jpa</artifactId>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>


</project>

3 – Run the Empty Application

Although we haven’t written any code, we will run the Spring Boot application.

Go to your terminal and follow along with the commands.

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080
{"timestamp":1505235455245,"status":404,"error":"Not Found","message":"No message available","path":"/"}

We haven’t written any code yet, instead of a default container-generated HTML error response, Actuator produces you a JSON response from its /error endpoint.

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080/health
{"status":"UP"}

The Actuator /health endpoint will let you know if your application is up.

4 – Model

Now let’s define the fields of the models for our username tracker application.

  • As mentioned, a person has one account and may have many usernames. So we map Set with a @OneToMany annotation
  • A username model will have a password and a username of course
  • Our model will need an ID and we make it autogenerated
  • We make a class construction to define an account may be made with a username and a password. Because of this custom constructor we also need to make a default one with no parameters.

Account.java

package com.michaelcgood.model;

import java.util.HashSet;
import java.util.Set;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import javax.persistence.OneToMany;

import com.fasterxml.jackson.annotation.JsonIgnore;

@Entity
public class Account {

	public Set<Usernames> getUsernames() {
		return usernames;
	}

	public void setUsernames(Set<Usernames> usernames) {
		this.usernames = usernames;
	}

	public Long getId() {
		return id;
	}

	public void setId(Long id) {
		this.id = id;
	}

	public String getPassword() {
		return password;
	}

	public void setPassword(String password) {
		this.password = password;
	}

	public String getUsername() {
		return username;
	}

	public void setUsername(String username) {
		this.username = username;
	}

	@OneToMany(mappedBy= "account")
	private Set<Usernames> usernames = new HashSet<>();
	
	@Id
	@GeneratedValue
	private Long id;
	
	@JsonIgnore
	public String password;
	public String username;
	
	public Account(String name, String password) {
        this.username = name;
        this.password = password;
    }
	
	Account(){
		
	}
	
}

Usernames.java

  • As there is one account to many usernames, the reverse is true as well: there many usernames to one account. Therefore, we map Account with @ManyToOne annotation
  • To track a username we need: the url and the username
  • We once again define an autogenerated ID
  • We define a custom class constructor that requires account, url, and username parameter. Once again we need to define a default constructor method to avoid an error being thrown.
package com.michaelcgood.model;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import javax.persistence.ManyToOne;

import com.fasterxml.jackson.annotation.JsonIgnore;

@Entity
public class Usernames {
	
	@JsonIgnore
	@ManyToOne
	private Account account;
	
	public Account getAccount() {
		return account;
	}

	public void setAccount(Account account) {
		this.account = account;
	}

	public Long getId() {
		return id;
	}

	public void setId(Long id) {
		this.id = id;
	}

	public String getUrl() {
		return url;
	}

	public void setUrl(String url) {
		this.url = url;
	}

	public String getUsername() {
		return username;
	}

	public void setUsername(String username) {
		this.username = username;
	}

	@Id
	@GeneratedValue
	private Long id;
	
	public String url;
	public String username;
	
	Usernames(){
		
	}
	
	public Usernames(Account account, String url, String username){
		this.url=url;
		this.account=account;
		this.username=username;
	}
	


}

5 – Repository

We create a repository for both models and create search functions using derived queries.

AccountRepository.java

package com.michaelcgood.dao;

import java.util.Optional;

import org.springframework.data.jpa.repository.JpaRepository;

import com.michaelcgood.model.Account;

public interface AccountRepository extends JpaRepository<Account,Long> {

	Optional<Account> findByUsername(String username);
}

UsernamesRepository.java

package com.michaelcgood.dao;

import java.util.Collection;

import org.springframework.data.jpa.repository.JpaRepository;

import com.michaelcgood.model.Usernames;

public interface UsernamesRepository extends JpaRepository<Usernames,Long> {

	Collection<Usernames> findByAccountUsername(String username);
	
}



6 – Controller

In the controller, we define all the mapping that we will be using for our RESTful web service.

  • We annotate our controller with @RestController rather than @Controller. As stated in the javadoc, it is “a convenience annotation that is itself annotated with @Controller and @ResponseBody.”
  • We declare the variables for our UsernamesRepository and AccountRepository and make them final because we only want the value to be assigned once. We annotate them as @Autowired over the UsernamesRestController class constructor.
  • {userId} and {usernamesId} are path variables. That means these values are provided in a URL. This will be shown in our demo.
  • The Controller methods return POJOs (Plain Old Java Objects). Spring Boot automatically wires HttpMessageConverter to convert these generic objects to JSON.

UsernamesRestController.java

package com.michaelcgood.controller;

import java.net.URI;
import java.util.Collection;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.servlet.support.ServletUriComponentsBuilder;

import com.michaelcgood.dao.AccountRepository;
import com.michaelcgood.dao.UsernamesRepository;
import com.michaelcgood.model.Usernames;

@RestController
@RequestMapping("/{userId}/usernames")
public class UsernamesRestController {

	private final UsernamesRepository usernamesRepository;
	private final AccountRepository accountRepository;
	
	@Autowired
	UsernamesRestController(UsernamesRepository usernamesRepository, AccountRepository accountRepository){
		this.usernamesRepository = usernamesRepository;
		this.accountRepository = accountRepository;
	}
	
	@GetMapping
	Collection<Usernames> readUsernames (@PathVariable String userId){
		this.validateUser(userId);
		return this.usernamesRepository.findByAccountUsername(userId);
	}
	
	@PostMapping
	ResponseEntity<?> add(@PathVariable String userId,@RequestBody Usernames input){
		this.validateUser(userId);
		
		return this.accountRepository.findByUsername(userId)
				.map(account -> {
					Usernames result = usernamesRepository.save(new Usernames(account,input.url,input.username));
					
					URI url = ServletUriComponentsBuilder
							.fromCurrentRequest().path("/{id}")
							.buildAndExpand(result.getId()).toUri();
					
						return ResponseEntity.created(url).build();	
				})
				.orElse(ResponseEntity.noContent().build());
	}
	
	@GetMapping(value="{usernamesId}")
	Usernames readUsername(@PathVariable String userId, @PathVariable Long usernameId){
		this.validateUser(userId);
		return this.usernamesRepository.findOne(usernameId);
	}
	
	private void validateUser(String userId){
		this.accountRepository.findByUsername(userId).orElseThrow(
				() -> new UserNotFoundException(userId));
	}
	
	
}

UserNotFoundException.java

Here we define the custom exception we used in our Controller class to explain a user could not be found.

package com.michaelcgood.controller;

import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.ResponseStatus;

@ResponseStatus(HttpStatus.NOT_FOUND)
public class UserNotFoundException extends RuntimeException {

/**
	 * 
	 */
	private static final long serialVersionUID = 7537022054146700535L;

public UserNotFoundException(String userId){
	super("Sorry, we could not find user '" + userId +"'.");
}
	
	
}

7 – @SpringBootApplication

We use the CommandLineRunner to create accounts and insert usernames. Every account will have two usernames.

package com.michaelcgood;

import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

import com.michaelcgood.dao.AccountRepository;
import com.michaelcgood.dao.UsernamesRepository;
import com.michaelcgood.model.Account;
import com.michaelcgood.model.Usernames;

import java.util.Arrays;

@SpringBootApplication
public class SpringBootActuatorExampleApplication {

	public static void main(String[] args) {
		SpringApplication.run(SpringBootActuatorExampleApplication.class, args);
	}
	
	@Bean
	CommandLineRunner init(AccountRepository accountRepository,
			UsernamesRepository usernamesRepository) {
		return (evt) -> Arrays.asList(
				"ricksanchez,mortysmith,bethsmith,jerrysmith,summersmith,birdperson,squanchy,picklerick".split(","))
				.forEach(
						a -> {
							Account account = accountRepository.save(new Account(a,
									"password"));
							usernamesRepository.save(new Usernames(account,
									"http://example.com/login", a +"1"));
							usernamesRepository.save(new Usernames(account,
									"http://example2.com/login", "the_"+a));
						});
	}
}

8 – Configuration

It is stated in the Spring documentation:

By default all sensitive HTTP endpoints are secured such that only users that have an ACTUATOR role may access them. Security is enforced using the standard HttpServletRequest.isUserInRole method.

We haven’t set up any security and user roles, as this is just an example. So, for ease of demonstration, I will be disabling the security requirement. Otherwise, we will get an “unauthorized” error as of right now, like the one shown below.

{"timestamp":1505321635068,"status":401,"error":"Unauthorized","message":"Full authentication is required to access this resource.","path":"/beans"}

application.properties

Add this to your application.properties to disable the need for authentication.

management.security.enabled=false

9 – Demo

To retrieve the responses from the server, you could either visit the URL in your browser or use curl. For my demo, I am using curl.

REST queries for data in repositories

Query for usernames belonging to account jerrysmith.

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080/jerrysmith/usernames
[{"id":7,"url":"http://example.com/login","username":"jerrysmith1"},{"id":8,"url":"http://example2.com/login","username":"the_jerrysmith"}]

Query for usernames belonging to account picklerick

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080/picklerick/usernames
[{"id":15,"url":"http://example.com/login","username":"picklerick1"},{"id":16,"url":"http://example2.com/login","username":"the_picklerick"}]

Actuator queries

The response to this query is truncated because it is really, really long.

Beans

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080/beans
[{"context":"application","parent":null,"beans":[{"bean":"springBootActuatorExampleApplication","aliases":[],"scope":"singleton","type":"com.michaelcgood.SpringBootActuatorExampleApplication$$EnhancerBySpringCGLIB$$509f4984","resource":"null","dependencies":[]},{"bean":"org.springframework.boot.autoconfigure.internalCachingMetadataReaderFactory","aliases":[],"scope":"singleton","type":"org.springframework.core.type.classreading.CachingMetadataReaderFactory","resource":"null","dependencies":[]},{"bean":"usernamesRestController","aliases":[],"scope":"singleton","type":"com.michaelcgood.controller.UsernamesRestController","resource":"file [/Users/mike/javaSTS/Spring-Boot-Actuator-Example/target/classes/com/michaelcgood/controller/UsernamesRestController.class]","dependencies":["usernamesRepository","accountRepository"]},{"bean":"init","aliases":[],"scope":"singleton","type":"com.michaelcgood.SpringBootActuatorExampleApplication$$Lambda$11/889398176","resource":"com.michaelcgood.SpringBootActuatorExampleApplication",
[...]

Metrics

mikes-MacBook-Air:Spring-Boot-Actuator-Example mike$ curl localhost:8080/metrics
{"mem":350557,"mem.free":208275,"processors":4,"instance.uptime":213550,"uptime":240641,"systemload.average":1.6552734375,"heap.committed":277504,"heap.init":131072,"heap.used":69228,"heap":1864192,"nonheap.committed":74624,"nonheap.init":2496,"nonheap.used":73062,"nonheap":0,"threads.peak":27,"threads.daemon":23,"threads.totalStarted":30,"threads":25,"classes":9791,"classes.loaded":9791,"classes.unloaded":0,"gc.ps_scavenge.count":11,"gc.ps_scavenge.time":139,"gc.ps_marksweep.count":2,"gc.ps_marksweep.time":148,"httpsessions.max":-1,"httpsessions.active":0,"datasource.primary.active":0,"datasource.primary.usage":0.0,"gauge.response.beans":14.0,"gauge.response.info":13.0,"counter.status.200.beans":2,"counter.status.200.info":1}

9 – Conclusion

Congratulations, you have created a RESTful Web Service that can be monitored with Actuator. REST is really the most organic way for different clients to communicate because it works due to HTTP.

The source code is on Github



PagingAndSortingRepository – How to Use With Thymeleaf

For this tutorial, I will demonstrate how to display a list of a business’ clients in Thymeleaf with pagination.

View and Download the code from Github

1 – Project Structure

We have a normal Maven project structure.
Maven structure of PagingAndSortingRepository Example

2 – Project Dependencies

Besides the normal Spring dependencies, we add Thymeleaf and hsqldb because we are using an embedded database.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>michaelcgood-pagingandsorting</artifactId>
	<version>0.0.1</version>
	<packaging>jar</packaging>

	<name>PagingAndSortingRepositoryExample</name>
	<description>Michael C  Good - PagingAndSortingRepository</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.6.RELEASE</version>
		<relativePath /> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-thymeleaf</artifactId>
		</dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.hsqldb</groupId>
			<artifactId>hsqldb</artifactId>
			<scope>runtime</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-jpa</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>


</project>

3 – Models

We define the following fields for a client:

  • a unique identifier
  • name of the client
  • an address of the client
  • the amount owed on the current invoice

The getters and setters are quickly generated in Spring Tool Suite.
The @Entity annotation is needed for registering this model to @SpringBootApplication.

ClientModel.java

package com.michaelcgood.model;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;

@Entity
public class ClientModel {
	
	@Id
	@GeneratedValue(strategy = GenerationType.AUTO)
	private Long id;
	public Long getId() {
		return id;
	}
	public void setId(Long id) {
		this.id = id;
	}
	public String getName() {
		return name;
	}
	public void setName(String name) {
		this.name = name;
	}
	public String getAddress() {
		return address;
	}
	public void setAddress(String address) {
		this.address = address;
	}
	public Integer getCurrentInvoice() {
		return currentInvoice;
	}
	public void setCurrentInvoice(Integer currentInvoice) {
		this.currentInvoice = currentInvoice;
	}
	private String name;
	private String address;
	private Integer currentInvoice;

}

The PagerModel is just a POJO (Plain Old Java Object), unlike the ClientModel. There are no imports, hence no annotations. This PagerModel is purely just used for helping with the pagination on our webpage. Revisit this model once you read the Thymeleaf template and see the demo pictures. The PagerModel makes more sense when you think about it in context.

PagerModel.java

package com.michaelcgood.model;

public class PagerModel {
	private int buttonsToShow = 5;

	private int startPage;

	private int endPage;

	public PagerModel(int totalPages, int currentPage, int buttonsToShow) {

		setButtonsToShow(buttonsToShow);

		int halfPagesToShow = getButtonsToShow() / 2;

		if (totalPages <= getButtonsToShow()) {
			setStartPage(1);
			setEndPage(totalPages);

		} else if (currentPage - halfPagesToShow <= 0) {
			setStartPage(1);
			setEndPage(getButtonsToShow());

		} else if (currentPage + halfPagesToShow == totalPages) {
			setStartPage(currentPage - halfPagesToShow);
			setEndPage(totalPages);

		} else if (currentPage + halfPagesToShow > totalPages) {
			setStartPage(totalPages - getButtonsToShow() + 1);
			setEndPage(totalPages);

		} else {
			setStartPage(currentPage - halfPagesToShow);
			setEndPage(currentPage + halfPagesToShow);
		}

	}

	public int getButtonsToShow() {
		return buttonsToShow;
	}

	public void setButtonsToShow(int buttonsToShow) {
		if (buttonsToShow % 2 != 0) {
			this.buttonsToShow = buttonsToShow;
		} else {
			throw new IllegalArgumentException("Must be an odd value!");
		}
	}

	public int getStartPage() {
		return startPage;
	}

	public void setStartPage(int startPage) {
		this.startPage = startPage;
	}

	public int getEndPage() {
		return endPage;
	}

	public void setEndPage(int endPage) {
		this.endPage = endPage;
	}

	@Override
	public String toString() {
		return "Pager [startPage=" + startPage + ", endPage=" + endPage + "]";
	}

}

4 – Repository

The PagingAndSortingRepository is an extension of the CrudRepository. The only difference is that it allows you to do pagination of entities. Notice that we annotate the interface with @Repository to make it visible to @SpringBootApplication.

ClientRepository.java

package com.michaelcgood.dao;

import org.springframework.data.repository.PagingAndSortingRepository;
import org.springframework.stereotype.Repository;

import com.michaelcgood.model.ClientModel;

@Repository
public interface ClientRepository extends PagingAndSortingRepository<ClientModel,Long> {

}

5 – Controller

We define some variables in the beginning of the class. We only want to show 3 page buttons at time. The initial page is the first page of results, the initial amount of items on the page is 5, and the user has the ability to have either 5 or 10 results per page.

We add some example values to our repository with the addtorepository() method, which is defined further below in this class. With the addtorepository method(), we add several “clients” to our repository, and many of them are hat companies because I ran out of ideas.

ModelAndView is used here rather than Model. ModelAndView is used instead because it is a container for both a ModelMap and a view object. It allows the controller to return both as a single value. This is desired for what we are doing.

ClientController.java

package com.michaelcgood.controller;

import java.util.Optional;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.servlet.ModelAndView;
import com.michaelcgood.model.PagerModel;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;

import com.michaelcgood.dao.ClientRepository;
import com.michaelcgood.model.ClientModel;

@Controller
public class ClientController {
	
	private static final int BUTTONS_TO_SHOW = 3;
	private static final int INITIAL_PAGE = 0;
	private static final int INITIAL_PAGE_SIZE = 5;
	private static final int[] PAGE_SIZES = { 5, 10};
	@Autowired
	ClientRepository clientrepository;
	
	@GetMapping("/")
	public ModelAndView homepage(@RequestParam("pageSize") Optional<Integer> pageSize,
			@RequestParam("page") Optional<Integer> page){
		
		if(clientrepository.count()!=0){
			;//pass
		}else{
			addtorepository();
		}
		
		ModelAndView modelAndView = new ModelAndView("index");
		//
		// Evaluate page size. If requested parameter is null, return initial
		// page size
		int evalPageSize = pageSize.orElse(INITIAL_PAGE_SIZE);
		// Evaluate page. If requested parameter is null or less than 0 (to
		// prevent exception), return initial size. Otherwise, return value of
		// param. decreased by 1.
		int evalPage = (page.orElse(0) < 1) ? INITIAL_PAGE : page.get() - 1;
		// print repo
		System.out.println("here is client repo " + clientrepository.findAll());
		Page<ClientModel> clientlist = clientrepository.findAll(new PageRequest(evalPage, evalPageSize));
		System.out.println("client list get total pages" + clientlist.getTotalPages() + "client list get number " + clientlist.getNumber());
		PagerModel pager = new PagerModel(clientlist.getTotalPages(),clientlist.getNumber(),BUTTONS_TO_SHOW);
		// add clientmodel
		modelAndView.addObject("clientlist",clientlist);
		// evaluate page size
		modelAndView.addObject("selectedPageSize", evalPageSize);
		// add page sizes
		modelAndView.addObject("pageSizes", PAGE_SIZES);
		// add pager
		modelAndView.addObject("pager", pager);
		return modelAndView;
		
	}
	
public void addtorepository(){
		
		//below we are adding clients to our repository for the sake of this example
				ClientModel widget = new ClientModel();
				widget.setAddress("123 Fake Street");
				widget.setCurrentInvoice(10000);
				widget.setName("Widget Inc");
				
				clientrepository.save(widget);
				
				//next client
				ClientModel foo = new ClientModel();
				foo.setAddress("456 Attorney Drive");
				foo.setCurrentInvoice(20000);
				foo.setName("Foo LLP");
				
				clientrepository.save(foo);
				
				//next client
				ClientModel bar = new ClientModel();
				bar.setAddress("111 Bar Street");
				bar.setCurrentInvoice(30000);
				bar.setName("Bar and Food");
				clientrepository.save(bar);
				
				//next client
				ClientModel dog = new ClientModel();
				dog.setAddress("222 Dog Drive");
				dog.setCurrentInvoice(40000);
				dog.setName("Dog Food and Accessories");
				clientrepository.save(dog);
				
				//next client
				ClientModel cat = new ClientModel();
				cat.setAddress("333 Cat Court");
				cat.setCurrentInvoice(50000);
				cat.setName("Cat Food");
				clientrepository.save(cat);
				
				//next client
				ClientModel hat = new ClientModel();
				hat.setAddress("444 Hat Drive");
				hat.setCurrentInvoice(60000);
				hat.setName("The Hat Shop");
				clientrepository.save(hat);
				
				//next client
				ClientModel hatB = new ClientModel();
				hatB.setAddress("445 Hat Drive");
				hatB.setCurrentInvoice(60000);
				hatB.setName("The Hat Shop B");
				clientrepository.save(hatB);
				
				//next client
				ClientModel hatC = new ClientModel();
				hatC.setAddress("446 Hat Drive");
				hatC.setCurrentInvoice(60000);
				hatC.setName("The Hat Shop C");
				clientrepository.save(hatC);
				
				//next client
				ClientModel hatD = new ClientModel();
				hatD.setAddress("446 Hat Drive");
				hatD.setCurrentInvoice(60000);
				hatD.setName("The Hat Shop D");
				clientrepository.save(hatD);
				
				//next client
				ClientModel hatE = new ClientModel();
				hatE.setAddress("447 Hat Drive");
				hatE.setCurrentInvoice(60000);
				hatE.setName("The Hat Shop E");
				clientrepository.save(hatE);
				
				//next client
				ClientModel hatF = new ClientModel();
				hatF.setAddress("448 Hat Drive");
				hatF.setCurrentInvoice(60000);
				hatF.setName("The Hat Shop F");
				clientrepository.save(hatF);
				
	}
	
}

6 – Thymeleaf Template

In Thymeleaf template, the two most important things to note are:

  • Thymeleaf Standard Dialect
  • Javascript

Like in a CrudRepository, we iterate through the PagingAndSortingRepository with th:each=”clientlist : ${clientlist}”. Except instead of each item in the repository being an Iterable, the item is a Page.

With select class=”form-control pagination” id=”pageSizeSelect”, we are allowing the user to pick their page size of either 5 or 10. We defined these values in our Controller.

Next is the code that allows the user to browse the various pages. This is where our PagerModel comes in to use.

The changePageAndSize() function is the JavaScript function that will update the page size when the user changes it.

<html xmlns="http://www.w3.org/1999/xhtml"
	xmlns:th="http://www.thymeleaf.org">

<head>
<!-- CSS INCLUDE -->
<link rel="stylesheet"
	href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
	integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u"
	crossorigin="anonymous"></link>

<!-- EOF CSS INCLUDE -->
<style>
.pagination-centered {
	text-align: center;
}

.disabled {
	pointer-events: none;
	opacity: 0.5;
}

.pointer-disabled {
	pointer-events: none;
}
</style>

</head>
<body>

	<!-- START PAGE CONTAINER -->
	<div class="container-fluid">
		<!-- START PAGE SIDEBAR -->
		<!-- commented out     <div th:replace="fragments/header :: header">&nbsp;</div> -->
		<!-- END PAGE SIDEBAR -->
		<!-- PAGE TITLE -->
		<div class="page-title">
			<h2>
				<span class="fa fa-arrow-circle-o-left"></span> Client Viewer
			</h2>
		</div>
		<!-- END PAGE TITLE -->
		<div class="row">
			<table class="table datatable">
				<thead>
					<tr>
						<th>Name</th>
						<th>Address</th>
						<th>Load</th>
					</tr>
				</thead>
				<tbody>
					<tr th:each="clientlist : ${clientlist}">
						<td th:text="${clientlist.name}">Text ...</td>
						<td th:text="${clientlist.address}">Text ...</td>
						<td><button type="button"
								class="btn btn-primary btn-condensed">
								<i class="glyphicon glyphicon-folder-open"></i>
							</button></td>
					</tr>
				</tbody>
			</table>
			<div class="row">
				<div class="form-group col-md-1">
					<select class="form-control pagination" id="pageSizeSelect">
						<option th:each="pageSize : ${pageSizes}" th:text="${pageSize}"
							th:value="${pageSize}"
							th:selected="${pageSize} == ${selectedPageSize}"></option>
					</select>
				</div>
				<div th:if="${clientlist.totalPages != 1}"
					class="form-group col-md-11 pagination-centered">
					<ul class="pagination">
						<li th:class="${clientlist.number == 0} ? disabled"><a
							class="pageLink"
							th:href="@{/(pageSize=${selectedPageSize}, page=1)}">&laquo;</a>
						</li>
						<li th:class="${clientlist.number == 0} ? disabled"><a
							class="pageLink"
							th:href="@{/(pageSize=${selectedPageSize}, page=${clientlist.number})}">&larr;</a>
						</li>
						<li
							th:class="${clientlist.number == (page - 1)} ? 'active pointer-disabled'"
							th:each="page : ${#numbers.sequence(pager.startPage, pager.endPage)}">
							<a class="pageLink"
							th:href="@{/(pageSize=${selectedPageSize}, page=${page})}"
							th:text="${page}"></a>
						</li>
						<li
							th:class="${clientlist.number + 1 == clientlist.totalPages} ? disabled">
							<a class="pageLink"
							th:href="@{/(pageSize=${selectedPageSize}, page=${clientlist.number + 2})}">&rarr;</a>
						</li>
						<li
							th:class="${clientlist.number + 1 == clientlist.totalPages} ? disabled">
							<a class="pageLink"
							th:href="@{/(pageSize=${selectedPageSize}, page=${clientlist.totalPages})}">&raquo;</a>
						</li>
					</ul>
				</div>
			</div>
		</div>
		<!-- END PAGE CONTENT -->
		<!-- END PAGE CONTAINER -->
	</div>
		<script
  src="https://code.jquery.com/jquery-1.11.1.min.js"
  integrity="sha256-VAvG3sHdS5LqTT+5A/aeq/bZGa/Uj04xKxY8KM/w9EE="
  crossorigin="anonymous"></script>
 

	<script
		src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"
		integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa"
		crossorigin="anonymous"></script>
	<script th:inline="javascript">
		/*<![CDATA[*/
		$(document).ready(function() {
	changePageAndSize();
});

function changePageAndSize() {
	$('#pageSizeSelect').change(function(evt) {
		window.location.replace("/?pageSize=" + this.value + "&page=1");
	});
}
		/*]]>*/
	</script>

</body>
</html>

7 – Configuration

The below properties can be changed based on your preferences but were what I wanted for my environment.

application.properties

#==================================
# = Thymeleaf configurations 
#==================================
spring.thymeleaf.check-template-location=true
spring.thymeleaf.prefix=classpath:/templates/
spring.thymeleaf.suffix=.html
spring.thymeleaf.content-type=text/html
spring.thymeleaf.cache=false
server.contextPath=/

8 – Demo

This is the homepage.

Homepage PartingAndSortingRepository Example

This is the second page.
Second Page PagingAndSortingRepository

I can change the amount of items on the page to 10.
Change page size to 10 PagingAndSortingRepository

The source code is on Github

Using MySQL JDBC Driver With Spring Boot

In this article, I will show you how to connect a MySQL database with your Spring Boot application.

All the code is available on Github

Tools used in this article include:

  • Spring Boot 1.5.6 Release
  • MySQL 5.7.X
  • Maven
  • Java 8
  • Spring Data JPA

1 – Project Structure

The project structure is a typical Maven structure.

Project structure Maven

2 – Project Dependencies

Please note that the parent needs to declared. If you are using Spring Tool Suite, you can click new “Spring Starter Project” and it will populate this for you.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.michaelcgood</groupId>
	<artifactId>mysql-jdbc</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>

	<name>mysql-jdbc-driver</name>
	<description>mysql jdbc driver example</description>

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>1.5.6.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter</artifactId>
		</dependency>

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
			<dependency>
      <groupId>mysql</groupId>
      <artifactId>mysql-connector-java</artifactId>
    </dependency>
    	<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-jpa</artifactId>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>


</project>

3 – Model

For this example application, our application will be “tracking” the last security audit of systems within a network. As this example application is meant to be simple, there will be minimal fields for the model.

Please note that there is a built in System class in the Java library. For this reason, I would avoid using System.java as a class name for a real application.

System.java

package com.michaelcgood.model;

import java.util.Date;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;

@Entity
public class System {

	private String name;
	private Date lastaudit;
	public Date getLastaudit() {
		return lastaudit;
	}
	public void setLastaudit(Date lastaudit) {
		this.lastaudit = lastaudit;
	}
	@Id
	@GeneratedValue(strategy = GenerationType.AUTO)
	@Column(name="id")
	private long id;
	public long getId() {
		return id;
	}
	public void setId(long id) {
		this.id = id;
	}
	public String getName() {
		return name;
	}
	public void setName(String name) {
		this.name = name;
	}
	
	public String toString(){
		return id+" | " + name+ " | "+ lastaudit;
	}
	
}

4 – Repository

This is a simple CrudRepository, which is an interface that allows us to do CRUD (Create, Read, Update, Delete) operations.

SystemRepository.java

package com.michaelcgood.dao;

import org.springframework.data.repository.CrudRepository;
import org.springframework.stereotype.Repository;
import com.michaelcgood.model.System;

@Repository
public interface SystemRepository extends CrudRepository<System,Long> {
	

}

5 – Database Initialization

Spring Boot enables the dataSource initializer by default and loads SQL scripts (schema.sql and data.sql) from the root of the classpath.

5.1

Here we create the SQL file that our application will use for the Table schema.

Schema.sql

DROP TABLE IF EXISTS system;
CREATE TABLE system (
  id INT NOT NULL AUTO_INCREMENT,
  name VARCHAR(100) NOT NULL,
  lastaudit DATE NOT NULL,
  PRIMARY KEY (id));

5.2

We insert example values into our database.

Data.sql

INSERT INTO system(name,lastaudit)VALUES('Windows Server 2012 R2 ','2017-08-11');
INSERT INTO system(name,lastaudit)VALUES('RHEL 7','2017-07-21');
INSERT INTO system(name,lastaudit)VALUES('Solaris 11','2017-08-13');

5.3

This XML file is used to configure our logging.

logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <statusListener class="ch.qos.logback.core.status.NopStatusListener" />

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.classic.PatternLayout">
            <Pattern>
                %d{yyyy-MM-dd HH:mm:ss} %-5level %logger{36} - %msg%n
            </Pattern>
        </layout>
    </appender>

    <logger name="org.springframework.jdbc" level="error" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>

    <logger name="com.michaelcgood" level="error" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>

    <root level="error">
        <appender-ref ref="STDOUT"/>
    </root>

</configuration>

6 – Configuration

We configure our datasource and JPA settings.

application.properties

#==== connect to mysql ======#
spring.jpa.hibernate.ddl-auto=update
spring.datasource.url=jdbc:mysql://localhost:3306/mysqltutorial?useSSL=false
spring.datasource.username=root
spring.datasource.password=
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect



7 – @SpringBootApplication

CommandLineRunner is implemented in order to execute command line arguments for this example.

package com.michaelcgood.app;

import javax.sql.DataSource;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.domain.EntityScan;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;

import com.michaelcgood.dao.SystemRepository;

@SpringBootApplication
@EnableJpaRepositories("com.michaelcgood.dao")
@EntityScan("com.michaelcgood.model")
public class MysqlJdbcDriverApplication implements CommandLineRunner {

	@Autowired
	DataSource dataSource;

	@Autowired
	SystemRepository systemRepository;

	public static void main(String[] args) {
		SpringApplication.run(MysqlJdbcDriverApplication.class, args);
	}

	@Override
	public void run(String... args) throws Exception {
		System.out.println("Our DataSource is = " + dataSource);
		Iterable<com.michaelcgood.model.System> systemlist = systemRepository.findAll();
		for(com.michaelcgood.model.System systemmodel:systemlist){
			System.out.println("Here is a system: " + systemmodel.toString());
		}
		

	}

}

8 – Demo


  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.5.6.RELEASE)

Our DataSource is = org.apache.tomcat.jdbc.pool.DataSource@40f70521{ConnectionPool[defaultAutoCommit=null; defaultReadOnly=null; defaultTransactionIsolation=-1; defaultCatalog=null; driverClassName=com.mysql.jdbc.Driver; maxActive=100; maxIdle=100; minIdle=10; initialSize=10; maxWait=30000; testOnBorrow=true; testOnReturn=false; timeBetweenEvictionRunsMillis=5000; numTestsPerEvictionRun=0; minEvictableIdleTimeMillis=60000; testWhileIdle=false; testOnConnect=false; password=********; url=jdbc:mysql://localhost:3306/mysqltutorial?useSSL=false; username=root; validationQuery=SELECT 1; validationQueryTimeout=-1; validatorClassName=null; validationInterval=3000; accessToUnderlyingConnectionAllowed=true; removeAbandoned=false; removeAbandonedTimeout=60; logAbandoned=false; connectionProperties=null; initSQL=null; jdbcInterceptors=null; jmxEnabled=true; fairQueue=true; useEquals=true; abandonWhenPercentageFull=0; maxAge=0; useLock=false; dataSource=null; dataSourceJNDI=null; suspectTimeout=0; alternateUsernameAllowed=false; commitOnReturn=false; rollbackOnReturn=false; useDisposableConnectionFacade=true; logValidationErrors=false; propagateInterruptState=false; ignoreExceptionOnPreLoad=false; useStatementFacade=true; }
Here is a system: 1 | Windows Server 2012 R2  | 2017-08-11 00:00:00.0
Here is a system: 2 | RHEL 7 | 2017-07-21 00:00:00.0
Here is a system: 3 | Solaris 11 | 2017-08-13 00:00:00.0

The full code is on Github

Batch Updates with JdbcTemplate

There may come time when you are using JdbcTemplate and want to use a PreparedStatement for a batch update. In the example below, we will explore how to insert thousands of records into a MySQL database using batchUpdate.

First, we must configure the datasource to use in our application.properties.

spring.datasource.url=jdbc:mysql://localhost:3306/myurl
spring.datasource.username=root
spring.datasource.password=mypassword
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect

Now in order to use this datasource in our Service.class we need to make it Autowired. Below I place the @Autowired annotation above the declaration of the DataSource but depending on your preference it could also be placed over the method constructor.

@Autowired
private DataSource dataSource;
private JdbcTemplate jdbcTemplate;

public void setDataSource(DataSource dataSource) {
this.dataSource = dataSource;
this.jdbcTemplate = new JdbcTemplate(dataSource);
}

In my example below, I have the method set to Private because the method should not be used outside of the class. However, depending on your own needs, you may make this Public.

Here is the full code; however, below I will explain the code line by line.

	private void loadTheContent(String[] rows) throws SQLException {
		String Query = "INSERT INTO content (id, type, publisher, title) VALUES (?,?,?,?)";
		JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
		jdbcTemplate.batchUpdate(Query, new BatchPreparedStatementSetter() {
			@Override
			public void setValues(PreparedStatement ps, int i) throws SQLException {
				String[] col = new String[4];
				col = rows[i].split("-");
				ps.setString(1, col[0]);
				ps.setString(2, col[1]);
				ps.setString(3, col[2]);
				ps.setString(4, col[3]);
			}

			@Override
			public int getBatchSize() {
				return rows.length;
			}
		});
		return;
	}

An an array of Strings is passed into the loadTheContent method.

(String[] rows) 

In my example, they are from a processed XML file. Covering XML processing is out of scope for this post, but will be covered in the future.

In order to successfully execute the batch prepared statement, we need to know the batch size – meaning how many records will be processed.

getBatchSize()

This is a request for the batch size. With the code below

  return rows.length; 

we instruct the program to measure the size of the rows array in order to calculate the batch size.



Next the query is defined.

String Query = "INSERT INTO content (id, type, publisher, title) VALUES (?,?,?,?)";

Take note that the query String cannot have a keyword such as final next to it as it is not immutable. Also, as this is not using a NamedParameterJdbcTemplate, we must use ‘?’ placeholders for the values.

JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);

The above instantiation of jdbcTemplate allows us to execute the SQL to our correct database. Since dataSource is autowired, this will execute correctly.

Next we actually define the batchUpdate. You may notice that the batchUpdate looks somewhat similar to a regular jdbcTemplate update.

The BatchPreparedStatmentSetter provides an interface to set values on the PreparedStatement.

jdbcTemplate.batchUpdate(Query, new BatchPreparedStatementSetter() {
			@Override
			public void setValues(PreparedStatement ps, int i) throws SQLException {

The Query parameter is already defined as a String, which I discussed above.
The int i is the current row that is being processed.

With this in mind, we set values for each column in the record. Below you see that I create an Array of 4 values, since I know that each record has 4 columns.

	String[] col = new String[4];
	col = rows[i].split("-");
	ps.setString(1, col[0]);
	ps.setString(2, col[1]);
	ps.setString(3, col[2]);
	ps.setString(4, col[3]);

I define each value in the col array by splitting at the “-” character in rows[i]. Next we use each value in the col array to set the values for the SQL query.

This means that the ‘?’ placeholders below are replaced by col[0], etc.

String Query = "INSERT INTO content (id, type, publisher, title) VALUES (?,?,?,?)";

For reference on JdbcTemplate.batchUpdate: https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/jdbc/core/JdbcTemplate.html#batchUpdate-java.lang.String-org.springframework.jdbc.core.BatchPreparedStatementSetter-

If you have any questions, ask in the comments and I will try to help.