Problem Solving - For Reals

One of my favorite quotes is a, "A problem well understood, is a problem half solved."

My team and I have been reading through the book the Toyota Way by Jeffrey Liker.  One thing that always stands out to me is that Toyota doesn't get hung up on unnecessary tools and processes.  They rely on human persistence and ingenuity and allow technology to enhance those attributes.

This is reflected in their way of solving problems.

When problems come up they seek to understand what is the root cause.  An example that was given in the book is a problem of a puddle of oil on the floor.  A quick inspection might show that a machine is leaking oil, so the solution might be to replace the leaking gasket in the machine.  I think many people would leave it at that, myself included.  However, that isn't getting to the root cause of the problem.  You would need to ask why is that gasket leaking in the first place, and that would lead to an area that many people may not be familiar with.

The 5 Whys

One of the main tools used by Toyota is the 5 Whys.  Or basically you ask Why 5 times to dig into what the root cause may be.  Generally going 5 levels deep is sufficient to understand what the root cause may be.

So to use the book's example again:
Why is there oil on the floor?
 Because the machine is leaking oil
    Why?
    Because the gasket has deteriorated
        Why?
        Because we bought gaskets made of inferior material
            Why?
            Because we got a good deal on those gaskets
                Why?
                Because the purchasing agent gets evaluated on short-term cost savings

From this example we see that the problem of an oil puddle stems from a much deeper root problem. If we imagine it is a technician who is solving this problem, he/she may not be familiar with the details of the purchasing office, so going this deep will require stepping out of his/her area of expertise.  An uncomfortable prospect for many.

A Culture of Problem Solving

Because of the difficulty of analyzing problems so deeply, this sort of tenacity has to be built into an organization's DNA.  The culture of the organization needs to promote this deep understanding.  It might take extra time to analyze problems, but the organization will need to make sure that is allowed, even praised.  Especially when the pressure is on.  And of course with any culture shift it will take time.  Toyota has been working on it for decades after all.

Bash Function To Zip & Encrypt Text

Usage:

$ zipe text-to-encrypt file-name

Get a Resource File's Contents

Here is a little snippet on how to get the contents of a resource file.


And for reference here are the imports used

import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;

Learning About Large Json Files

When connecting to different APIs you might come across some large datasets that might not fit in normal text editors.  This makes it difficult to learn about how the data is structured.

There exists a really cool tool called jq that makes it a little easier to see the data structure.  It takes a while to learn the syntax, but it is pretty powerful.  Below are a few of the commands I use regularly to learn how my data is structured.

Assume we have a file names largeResponse.json.  That has a similar structure like:

{
  "data_available": true,
  "query": "The query is here",
  "results": {
    "details": [
      {
        "id": 1,
        "data1": "data1",
        "data2": "data2"
      },
      {
        "id": 2,
        "data1": "data1",
        "data2": "data2"
      }
      ... Many more rows here
    ]
  }
}

Getting the keys

 jq 'keys' largeResponse.json

This command will list all of the keys of the root object.  We can also see the keys of child objects using

jq '.results | keys' largeResponse 

Assuming there is a child object of 'results'  this will show the list of the keys in the results object.

Get an object

jq '.results' largeResponse.json 

will show all of the results object, which for this case would be a a LOT of information.  However, for smaller objects, like the query, it could be very helpful.

Working with Arrays

With a large list of data it is useful is see a sample of the data.  

Length. We can see the length an array with something like this

jq '.results.details | length' largeResponse.json

Notice how details is an array.

Get a single Object.  We can get an item from an array with the index.

jq '.results.details[0]' largeResponse.json

This would return 

{
  "id": 1,
  "data1": "data1",
  "data2": "data2"
}

Get a range of objects.  jq can also get a range in an array.

jq '.results.details[0,10]' largeResponse.json

This would return the first ten objects in the details array.

Tons more

And of course you can a million other things that I haven't come close to exploring.  To see all that you can do take a look at the jq manual.

Misc

To print out multiple fields:

jq '.users[] | .first + " " + .last'

Java 8 Predicate Chaining

The other day I got it in my head that it would be fun to chain together Java 8's Predicate lambdas in an object-oriented sort of way.  Sort of like predicate1.and(predicate2).

It was a fun little exercise, but I don't think it would be incredibly useful, unless you are conditionally chaining them together in a for loop.  It really wouldnt be any different than using similar syntax as  predicate1 && predicate2.

But, with that being said, below is the PredicateChain class as well as the test that demonstrates how it could be used.

5 Ways to Reduce Risk in Software Projects

Software developers, and IT in general, have a reputation for being over budget, being late, and not even meeting the needs of the customer.  There are a million different reasons for that; such is the complex nature of software development. But, here are a few ways to reduce that complexity that leads to a failed project.

Increase Communication

There can be a lot of people from many parts of an organization involved in a software project and it might really be impossible to get everyone together all the time.  The risk is when ideas are communicated from the customer to one person who then has to translate those ideas to the people developing the software.  Just like in the game telephone ideas get altered and assumptions lost. 

When the developers and customers don't communicate you also lose the negotiation that might happen. This negotiation allows potential problems to be brought into the light before any code is actually written.  

Reduce Complexity

With increased communication you can also negotiate reducing the complexity in the requirements. If you have a complex business process you'll want to simplify it as much as possible.  Complexity goes hand in hand with problems, and not simple problems.  Problems as complex as the process itself so they're hard to track down.  

One of my favorite phrases is 'You ain't gonna need it' (YAGNI).  Half of the software developed will never be used, so cut out as much of it as possible before actually building anything.  

Get Feedback as Quickly as Possible

Once you have a good plan in place with the absolute minimum requirements needed its time to start building something.  This is where the assumptions that people have start to come out and things can go off course really quickly.  This is why it is imperative to get feedback as quickly as possible from the customer.  Let them see what you're doing and have them validate that it is correct and do it while it's still fresh on everyone's mind.  

One way to measure this is your cycle time.  How fast can you go from idea to a released feature.  The quicker you can do that the easier it will be to change what isn't going to work. 

Plan to Change

When managing a complex project there is only 1 reliable truth:  the plan is going to change.  You'll need to make sure that change is understood to be a good thing.  It will allow the project to deliver real value and be useful.

Change has a reputation for being bad.  For some projects that might be justified, but for complex projects, like software, you need to assume that your assumptions are wrong and you need to learn what is right along the way.  

Build in Quality

And lastly, don't compromise quality. It will be tempting with pressures building, but if you do compromise quality you run the risk of introducing dumb little bugs that have you face-palming yourself. 

You should be proud that you were part of a project and do everything you can in order for it to be the best you could possibly make it. 

Following these general principle will help wrangle the risk we all face whenever we start complex projects.  We'll never eliminate it all, but we can do our best to improve our chances to make it a successful project. 

SQL: Getting the Previous and Next Records

Every so often it's necessary to compare a record with a previous or next record, based on some kind of ordering.  Many newer versions of databases support this with LAG and LEAD functions.  That seems like the ideal way to handle the problem,  but us lucky few we don't have that option so we have to do it in a round about sort of way.

Let's assume we have this table:

CREATE TABLE EVENT_LOG (
  system_name varchar(256),
  event_date datetime
 );

And let's say we want to know the previous and next entries for each record based on the system_name. Here is the query we could use:

WITH NEW_LOG AS (
  select ROW_NUMBER() OVER (order by system_name, event_date) row_num
  , system_name, event_date
  FROM EVENT_LOG
)
SELECT n.system_name, prev.event_date prev,  n.event_date, nex.event_date next
FROM NEW_LOG n
LEFT JOIN NEW_LOG prev 
  ON prev.system_name = n.system_name and prev.row_num = n.row_num -1
LEFT JOIN NEW_LOG nex 
  ON nex.system_name = n.system_name and nex.row_num = n.row_num +1
order by n.system_name, n.event_date desc;

Or feel free to play around with this SQLFiddle.

Here is a good reference here as well: http://blog.sqlauthority.com/2013/09/22/sql-server-how-to-access-the-previous-row-and-next-row-value-in-select-statement/

Also, note I ran a very similar query as above on an older version of DB2 just fine.