Get a Resource File's Contents

Here is a little snippet on how to get the contents of a resource file.


And for reference here are the imports used

import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;

Learning About Large Json Files

When connecting to different APIs you might come across some large datasets that might not fit in normal text editors.  This makes it difficult to learn about how the data is structured.

There exists a really cool tool called jq that makes it a little easier to see the data structure.  It takes a while to learn the syntax, but it is pretty powerful.  Below are a few of the commands I use regularly to learn how my data is structured.

Assume we have a file names largeResponse.json.  That has a similar structure like:

{
  "data_available": true,
  "query": "The query is here",
  "results": {
    "details": [
      {
        "id": 1,
        "data1": "data1",
        "data2": "data2"
      },
      {
        "id": 2,
        "data1": "data1",
        "data2": "data2"
      }
      ... Many more rows here
    ]
  }
}

Getting the keys

 jq 'keys' largeResponse.json

This command will list all of the keys of the root object.  We can also see the keys of child objects using

jq '.results | keys' largeResponse 

Assuming there is a child object of 'results'  this will show the list of the keys in the results object.

Get an object

jq '.results' largeResponse.json 

will show all of the results object, which for this case would be a a LOT of information.  However, for smaller objects, like the query, it could be very helpful.

Working with Arrays

With a large list of data it is useful is see a sample of the data.  

Length. We can see the length an array with something like this

jq '.results.details | length' largeResponse.json

Notice how details is an array.

Get a single Object.  We can get an item from an array with the index.

jq '.results.details[0]' largeResponse.json

This would return 

{
  "id": 1,
  "data1": "data1",
  "data2": "data2"
}

Get a range of objects.  jq can also get a range in an array.

jq '.results.details[0,10]' largeResponse.json

This would return the first ten objects in the details array.

Tons more

And of course you can a million other things that I haven't come close to exploring.  To see all that you can do take a look at the jq manual.

Misc

To print out multiple fields:

jq '.users[] | .first + " " + .last'

Java 8 Predicate Chaining

The other day I got it in my head that it would be fun to chain together Java 8's Predicate lambdas in an object-oriented sort of way.  Sort of like predicate1.and(predicate2).

It was a fun little exercise, but I don't think it would be incredibly useful, unless you are conditionally chaining them together in a for loop.  It really wouldnt be any different than using similar syntax as  predicate1 && predicate2.

But, with that being said, below is the PredicateChain class as well as the test that demonstrates how it could be used.

5 Ways to Reduce Risk in Software Projects

Software developers, and IT in general, have a reputation for being over budget, being late, and not even meeting the needs of the customer.  There are a million different reasons for that; such is the complex nature of software development. But, here are a few ways to reduce that complexity that leads to a failed project.

Increase Communication

There can be a lot of people from many parts of an organization involved in a software project and it might really be impossible to get everyone together all the time.  The risk is when ideas are communicated from the customer to one person who then has to translate those ideas to the people developing the software.  Just like in the game telephone ideas get altered and assumptions lost. 

When the developers and customers don't communicate you also lose the negotiation that might happen. This negotiation allows potential problems to be brought into the light before any code is actually written.  

Reduce Complexity

With increased communication you can also negotiate reducing the complexity in the requirements. If you have a complex business process you'll want to simplify it as much as possible.  Complexity goes hand in hand with problems, and not simple problems.  Problems as complex as the process itself so they're hard to track down.  

One of my favorite phrases is 'You ain't gonna need it' (YAGNI).  Half of the software developed will never be used, so cut out as much of it as possible before actually building anything.  

Get Feedback as Quickly as Possible

Once you have a good plan in place with the absolute minimum requirements needed its time to start building something.  This is where the assumptions that people have start to come out and things can go off course really quickly.  This is why it is imperative to get feedback as quickly as possible from the customer.  Let them see what you're doing and have them validate that it is correct and do it while it's still fresh on everyone's mind.  

One way to measure this is your cycle time.  How fast can you go from idea to a released feature.  The quicker you can do that the easier it will be to change what isn't going to work. 

Plan to Change

When managing a complex project there is only 1 reliable truth:  the plan is going to change.  You'll need to make sure that change is understood to be a good thing.  It will allow the project to deliver real value and be useful.

Change has a reputation for being bad.  For some projects that might be justified, but for complex projects, like software, you need to assume that your assumptions are wrong and you need to learn what is right along the way.  

Build in Quality

And lastly, don't compromise quality. It will be tempting with pressures building, but if you do compromise quality you run the risk of introducing dumb little bugs that have you face-palming yourself. 

You should be proud that you were part of a project and do everything you can in order for it to be the best you could possibly make it. 

Following these general principle will help wrangle the risk we all face whenever we start complex projects.  We'll never eliminate it all, but we can do our best to improve our chances to make it a successful project. 

SQL: Getting the Previous and Next Records

Every so often it's necessary to compare a record with a previous or next record, based on some kind of ordering.  Many newer versions of databases support this with LAG and LEAD functions.  That seems like the ideal way to handle the problem,  but us lucky few we don't have that option so we have to do it in a round about sort of way.

Let's assume we have this table:

CREATE TABLE EVENT_LOG (
  system_name varchar(256),
  event_date datetime
 );

And let's say we want to know the previous and next entries for each record based on the system_name. Here is the query we could use:

WITH NEW_LOG AS (
  select ROW_NUMBER() OVER (order by system_name, event_date) row_num
  , system_name, event_date
  FROM EVENT_LOG
)
SELECT n.system_name, prev.event_date prev,  n.event_date, nex.event_date next
FROM NEW_LOG n
LEFT JOIN NEW_LOG prev 
  ON prev.system_name = n.system_name and prev.row_num = n.row_num -1
LEFT JOIN NEW_LOG nex 
  ON nex.system_name = n.system_name and nex.row_num = n.row_num +1
order by n.system_name, n.event_date desc;

Or feel free to play around with this SQLFiddle.

Here is a good reference here as well: http://blog.sqlauthority.com/2013/09/22/sql-server-how-to-access-the-previous-row-and-next-row-value-in-select-statement/

Also, note I ran a very similar query as above on an older version of DB2 just fine.

Spring Data Custom Batch Save Repository

We use Spring Data JPA for our data access framework and generally we use JpaRepository to create our repos, which is a really handy way of doing it.  Spring uses the SimpleJpaRepository for its implementation and it does have a save that allows an Iterable, but if you look at the code it just loops over the entities and calls the save method on each one.

So, if we want to save, or update, in a batch we need to use something else.  Luckily, Spring JDBC has a pretty handy way of doing it and you can add custom repos to your repository interface.

Like so:

To see the collection util that splits out our collection into batches see Split a Java List Into a List of Sublists

Dynamic Where Clause in JPA

Every once in a while you'll come across a scenario where you want to dynamically add parameters to a query's where clause.  For dynamic filtering I created a query generator that will simply add a segment to the where clause and the parameter to the query, and it works pretty well.

However, there are times when the scenario is a bit more complex and you'll need some more advanced logic.  For example given this entity:


Let's say we're given a list of similar data objects that have an optional userId, the last name, and the last 4 digits of the ssn. We want to match these to our User object. The rules to match up to the User is thus: use the userId if there is one, but if not match with the lastName and the last 4 digits of the ssn.

The SQL Query would look something like this:
In order to generate a query we need some conditional logic to decide if we should add the userId to a list to query the :userIds parameter or if we should add an OR clause to find a match of lastname and ssn.

The Criteria API

I've generally tried to avoid using the criteria API for my dynamic queries because it is fairly verbose, but it is an effective tool for the complex logic we're looking at. Below is an example of using the criteria API to generate the query we need.

As you can see it is pretty verbose but the flexibility to basically do whatever we want is pretty powerful.