Friday, July 7, 2017

Friday Challenge #1: Data Access

Today, I'm going to do a whole different kind of post - a Friday Challenge! Here's how it works: I'll post a challenge, you take up the challenge (usually by writing some code) and post the results or a link to the resulting repository in the comments. Discussions ensue. We share information. I'll put one up every Friday if there's interest. It's fun and engaging, what can go wrong? With that, here we go:

First Friday Challenge.

Your Quest is to write data access code using 3 different "patterns". I'll try to be a language agnostic as possible, the one requirement is to use a SQL based database.


In software there are many ways to perform the same functional operation, some are better than others when it comes to change, re-use, and readability.


You have a `users` table with the following columns: id IDENTITY/AUTOINCREMENT INT, name VARCHAR(255). You have the user's id and need to lookup the name of the user.


Given the following users:

|1|Marquart Flitshema|
|2|Loriens Blakens|
|3|Ashima Kithra|

When I look up a user with the id 1,
Then I should get the name "Marquart Flitshema".

When I look up a user with the id 3,
Then I should get the name "Ashima Kithra".


Pattern 1: Everything in 1 class/method/function (this is actually an anti-pattern).

Pattern 2: Pass the SQL and some mapping class/method/function into a different construct (class/method/function) to handle execution. Could be a base class (abstract or not), could be a prototype, any other construct will do so long as it is distinct from the construct that handles the SQL string and the mapping.

Pattern 3: Pass an entire construct into a different construct which handles the execution (by passing the database connection into the first construct).

Each pattern should be consumed similar to the following depending on language:

Pattern 1:

DataBase.getUserName 1.

 Pattern 2:

  Users.GetName 1


  Users.Get 1
(getName (dbQuery getUser 1))

consider the advantages and disadvantages of both approaches.

Pattern 3:

Db.Query UsersQueries.NameQuery 1



again consider advantages and disadvantages

Those are just examples in various language formats or pseudo-language formats. I tried to be inclusive, though I'm not as well versed in some languages (for a bit of language fun look up DerpCode). Feel free to use due dilligence such as null checking, exception handling whatever else you would like, but try not to muddle up the point too much with all that for now...don't let it get you off track is what I mean.

The code doesn't need to work completely, but if consumed by other code it should work (hint: you could use Unit Tests to drive the code and mock out the database but there should be some sort of `connect`, `open`, `close` and whatever else your SQL DB of choice normally has in the mock).

So that's it, the first Friday Challenge...go forth on your quest! If you need any tips, explainations, etc...feel free to put those in a comment and I (or anyone else who feels enlightened to do so) will do our best to get back to you as soon as we can. You should create a repo in GitHub, Bitbucket, JsBin, *Bin, someother online platform or whatever you do then share your work in the comments. Please don't link to file downloads, dropbox, etc...those aren't really for sharing code the way your should be sharing it for this.

Wednesday, July 5, 2017

What is Dependency Injection and Why Does it Matter?

While DI may be ubiquitous among most developers, it recently occurred to me that there are entire batches of newly minted grads entering the industry who, by and large, would not be familiar with the concept. Don't worry, you aren't the only ones! Although this topic has been bludgeoned to death by bloggers, I somehow feel the need to bubble it up here as well - sorry, it's a fun topic for me and I strongly feel that it's necessary to understand the principle. So if you are reading this and thinking - here we go again! Just give it a glance. But if you're like "what's this cool thing about?" or maybe still foggy on the principle read on and see if this doesn't clear things up.

I like to think about Dependency Injection/Inversion (DI) like this: A consumer should not be responsible for producing a resource that it consumes, the execution context should do that.

For a concrete example, consider the need to check whether or not a user is currently enrolled for mentoring. I'm going to be abstract in terms of implementation in order to make things a bit more language and platform agnostic (aside from the example being a web UI)...

We have some web site for managing mentor opportunities. Users who are mentors can navigate to the mentor page if they are mentors. If not a mentor, they cannot go to that page. Also, if they are not a mentor, we want to see if they have enough XP to serve as a mentor - if they do, encourage them to sign up for mentoring.

There are a few scenarios at play here so let's iterate those with specific examples:


| user   | XP    | is mentor |
| Tyler | 564   | yes           |
| Julie  | 8274 | no            |
| Jeff   |  128   | no            |

Scenario: User is a mentor.

Given user "Tyler" is logged in
When the navigation is shown
Then the user should see the mentors link.

Given user "Tyler" is logged in

When "Tyler" goes to the Mentors page
Then the Mentors page should be shown.

Scenario: User is not a mentor and is not qualified to mentor.

Given user "Jeff " is logged in
When the navigation is shown
Then the user should not see the mentors link.

Given user "Jeff " is logged in

When"Jeff " goes to the Mentors page
Then the user should be redirected to the home page.

Scenario: User is not a mentor and is qualified to mentor.

Given user "Julie" is logged in
When the navigation is shown
Then the user should see the mentors link, but it should be disabled.

Given user "Julie" is logged in

When"Julie" goes to the Mentors page
Then the user should be redirected to sign up for mentoring.

Given user "Julie" is logged in
When messages are displayed
And the user has not accepted the offer to become a mentor
Then an invite to become a mentor should be included in the messages.

Scenario: User becomes a mentor.

Given user "Julie" is logged in
When the navigation is shown
Then the user should see the mentors link.

Given user "Julie" is logged in

When "Julie" goes to the Mentors page
Then the Mentors page should be shown.

That's a lot of description, but it's worth noting the various scenarios in this example. But now back to focusing on the DI part...

So we have a module (note: I'm using "module" to express a class or function or whatever the unit is called in whatever language you use) for each component of the website -  the navigation bar, the mentor page, the mentor signup page. Those modules each need information about the user - is the user a mentor and can they become a mentor. Let's make a User module - one that can be passed into each web module and which can answer those questions. . Let's see how that would look:

        isMentor() : boolean
        isQualifiedForMentor() : boolean
        becomeMentor() : boolean

        getLinks(user) : links



The User module would be passed to the web modules by the request context (usually in the web framework) - the same thing that initializes the given web module.

So, something in the web request process will authenticate the user, load the user data, and pass the user details to the web page module as a User module. That User will be used to answer questions for each module that depends on it.

Notice what we are not going to do here - we are NOT going to call the database from within authorizeUser(), getLinks(), or signup()...those are details those modules simply should not care about! What they care about is whether or not the user is a mentor or is qualified to be one. Even the Signup page doesn't care about HOW to sign up the user, it just needs to direct something else to do the actual signup.

A level deeper...

The User itself could have some dependencies. Let's say the User needs to send a message to some signup processor - perhaps to a message queue so that the signup processor can pick off the queue and process at it's own pace (so the user won't have to wait for processing to complete).

Let's say at first, it handles the signup right there in the web process. Now let's say the site scales massively and running the signup process within the web processes impacts all users of the site due to the additional load on the web servers - we would want to offload that process. When we use DI, we can change the way the User handles the signup without changing the consumer (the User module).

    internals/private members/curried functions:
    public function:
        becomeMentor() : boolean
             calls _mentorSignup -> signup(user_id) or signupForMentoring(user_id)

Whatever happens when a user signs up is passed into the user module by the module that constructs the entire web request handling process (or whatever consumes the User for that matter).

The MentorSignup dependency (the one User is depending on) can either handle the signup itself OR pass off the signup to some other process that runs asynchronously so that the current process can continue execution without delay. It can be changed without touching the User module or any module that consumes User (theoretically).

A goal in good system design is to produce product that can be changed at a minimal cost - because software is soft (malleable, changeable).

In the case of changing the MentorSignup module, it's an ideal situation to be able to change the implementation without changing the consumers. However, there is one problem - if the MentorSignup module is consumed in a way that the web modules need to be recompiled or the MentorSignup module would need to be replaced in those modules in some other way, then we haven't achieved the ultimate goal. We are coupled architecturally.

Instead, we could have some communication between the web process and the signup process using a common interchange protocol such as HTTP and a mechanism to tell the signup process that a specific user would like to sign up to become a mentor. That's getting in a bit deeper than DI so I'll back off a little...

So here's what we have now: web modules, a User, a MentorSignup module, and something to build it all up and pass those dependencies in (MentorSignup into User, User into web modules).

We can do some interesting things with that - we can create some fake User code that gives us pre-defined answers so that we can build and test the web modules without depending on all the details of how a User is signed up, how user data is stored and retrieved, etc. We've decoupled the web modules from the dependency on the user data.

How else could we do it? Let's say we started out with the user data in a db and coded everything into the web modules directly:

        lookupUserLinks ... open a db connection, run query, map results, close connection, use results

        authorizeUser(user) ... open db connection, run query, map results, close connection, use results

        authorizeUser(user) ... open db connection, run query, map results, close connection, use results
        signup(user) ... open db connection, excute email...send email...close db connection...close different db connection....execute another command...close different db connection...etc

For one thing, we have a lot more room for errors...then duplicating errors. Another, our web modules would be difficult to understand - what does all this db code have to do with redirecting a user??? Ok you let's pull that out an put it into a User module, call up the user module to run the code...

        lookupUserLinks ... create User module...use User module to query (get user)...use results

        authorizeUser(user) ... create User module...use User module to query (get user)...use results

        authorizeUser(user) ... create User module...use User module to query (get user)...use results
        signup(user) ... create User module...use User module to query...use results...create Email module...use Email module to create and send email

At least the "goo" is centralized and has some re-usability - we no longer have a copy and paste scenario. But now we're still coupled to the specific User module that the functions know about directly! So we can't change those without changing the web modules - we can't do cool things like swap the User module on the fly or in test scenarios!

Now we get into programming to abstractions...we've already abstracted the details somewhat, but we've got this wicked coupling to only one specific implementation. So let's level that up and program to an abstraction...

For the purpose of building the web module, we don't care about the details of how to get the user, we don't even care that the user was got at all! What we really care about is that we have something to tell us certain things about the user and something that will sign the user up for mentoring. That's the abstraction we care about. We'll get into the details of implementation later in the development process - regardless of what those are, this thing should still work as described above in those Scenarios.

An important note on abstractions - they can leak! While we may only care logically that the user is a mentor, we also care that finding out does not take ages...if it did, that would be a leak in the abstraction! We really want to find out almost immediately when that block of consuming code runs - therefore we may want to load up all the user data BEFORE the web page is shown to the user. And of course we know this needs to be done quickly as well so we'll have to consider that in the implementation of the User module.

Now that we've gotten this far and (hopefullly) you can see how DI and many other fundamentals (programming to abstractions) are important to producing good software, let's take a look at one of the primary benefits of having done so...testing.

Now that we've designed the system in a way that depends on abstractions whose implementations are injected, we can swap out the implementation. For each of the defined scenarios described waaay above in this post, we can create a User that gives the pre-defined responses and exercise the code without depending on complicated - one-time only, db interactive - test setups.

We can have a setup for the first Scenario similar to:

MentorUser : User
    isMentor(): true

NonMentorNotQualifiedUser : User
    isMentor(): false
    isQualifiedForMentor() : false

NonMentorQualifiedUser : User
    isMentor(): false
    isQualifiedForMentor() : true
    becomeMentor(): true

Well, there you have a whole set of subtypes all based on the User with only isMentor()! Of course each has its own's own domain...we could further break those down to become specific dependencies of each web module, but that'll be enough detail for now...

think about the DI principle and other scenarios where you may want to swap some dependency based on context...connected to internet v not connected, testing v production, logging per environment, exception handling per environment, multi-tennant apps...just to name a few.

If you've now come to understand the DI principle then you've gained some valuable XP! Next you'll probably want to exercise the idea...look for something soon that'll help with that (hint:subscribe now to see what happens on Friday)!

Friday, June 30, 2017

How to Write Effective Commit Messages

When committing code to a source code repository, it is generally expected that we write some kind of comment to say what the change is all about. As we scroll through the history of changes we can see the comments for each commit. While there are variants of how we write our comments, I've found that following a simple formula can produce comments that provide the most value.

There are several problems when we write vague or disparate comments - consider it a smell. Either the work being done is nebulous or too big to pin down as in "doing work" or "cleaning up" or "refactor". Or the commenter isn't communicating well or worse yet, has no connection to the output of their work - "made tables" or "wrote code for beta" or "found a better way to do it".

The reason each of these types of comments are issues is that they don't communicate well to others - including your future self. As we scroll through source history looking for some past breaking change, we'd like to see specifics about each change.

It's good if you write comments like this:

    "[Issue #174] adding user type so Admins can login."

It has the following structure:

    "[<issue>] <what> so <why>"

First, the issue call out is formatted in a parse-able way - one day you may want to run a query on all comments and consistent formatting will help big time! More importantly, it links the change to an actual issue for reference otherwise there is no further context for the change. In some version control systems/integrations, you can actually link check-ins to issues. If you've got that setup, it's not strictly necessary to include the issue in the comment.

Next is <what> you are doing. Be brief in the first line. Make this something everyone can understand - make it relevant to users even. In this way, updates can easily be conveyed to users at deployment time! You can put details on subsequent lines if they aren't already in the issue itself or if they would be relevant to further understanding the change - but do so sparingly...the code and comments in the issue should speak for the details already.

Following <what> is <why>. I really like to include the why part because it allows other developers and your future self to understand at a glance why you made that change. More importantly, it gives product owners/managers some context to the changes that are being made with the same at-a-glance view. Communication with those folks can be facilitated via a dump of changes since last time! Of course with the proper tracking system this isn't the only way...but it is a handy tool for prepping a release.

Consider - we write code for consumers and will have to communicate to consumers about changes, commit comments is a big part of the glue between the incoming work and the outgoing product. Take the time to make them clear and concise for a better overall product experience!

Monday, May 15, 2017

Netflix Your Life

There we were, sitting at lunch talking about the "time v dollars" problem. The conversation wandered between alternatives to "trading time for dollars" and "working from wherever". As we tried to focus in on the ultimate work-life balance equation, my friend came out with phrase "Netflix your life". Hit it right on the nose!

If you were ever around when live television was a thing that people did, you know that you watched what was on - when it was on. In other put everything else in life on hold to watch your shows when they were on. And of course this meant sitting through commercials; or most likely, timing snacks and bathrooms with commercials. For that time, TV owned your life. And, their goal was to keep you glued to the TV for as long as possible so they could sell ad time at a higher rate (more viewers == more targets == more revenue) - the ratings game.

FFWD to modern times - aside from the VCR - there was no other way to do it, you were on Network Time. Nowadays, we have a bajillion ways to watch what we want, when we want - were in full-control of our viewing time. We can even watch at 2x speed to cram more video into one sitting. And...its portable - so we can watch wherever.

How did this change the game? We became more discerning with what we watch. TV got much much better. And producers can measure exactly what people watch. Furthermore, it should work better for the networks since they get more viewers - albeit more competition.

If we are able to work whenever and wherever - we have Netflixed our lives.

Monday, May 1, 2017

Quick Math on Time

Let's imagine we have some arbitrary process that takes about 20 mins per day to accomplish. And let's say it is done 5 days a week. That's 100 mins per week, about 400 mins per month, and 5200 mins per year.

I wish I could say that was 52 hours, but we're not quite vibing on metric time in this world so I'll have to stick with our whateverwegot standard time: It's about 86 hours per year...more than 2 weeks vacation for most people, an entire paycheck, a whole Sprint plus get the picture! How much does it cost?

Of course that part depends a lot on what the activity is. And valuation has the same dependency. You just can't automate a lot of processes away, but when you can automate and turning a 20 min/day task into a 1 minute task (or better yet 1hr/year for configuration) you better believe it can make a difference!

Let's put some $$$ to it...well, there's not easy answer here...but lets say its a skilled knowledge worker making $100/hr...easy to see that's over $8k/yr. And if more than one person does the same process that of course becomes a multiplier. A mid-size company might be losing a whole year or more on 20 minutes a day. Let's also suppose this process could be automated in maybe 8-16 hours.

How much does a meeting cost? Taking a guesstimates at some meetings I've attended where about 10 mid-tiers, 6 upper-mids, 15 lower-tiers (talking salaries here folks) met for an hour each month...that's equivalent of 31 person-hours (and then some considering the disruption, in which case we're talking 40 person-hours). Plus the space and resources used...range of $3000+ per meeting...or over $36k/year.

Automate or cut the fat...Kaizen is the practice of reducing (eliminating) waste. One way to cut waste is to automate, another is to eliminate, or lean-up your process. If you have an opportunity to do either, you stand to win big by freeing up resources so they can spend more time on more important things - like going to the beach, surfing, mountain biking...or what I mean is uhm...delivering more value, yeah that's what I meant.

Could that meeting have be cut down somehow? Probably not totally, but with some impact analysis maybe could save some people the trouble of having to attend. Would it be worth addressing? Probably not in this case. Point of this exercise is to illustrate that all waste cannot be reasonably eliminated. Plus there is a cultural value in the meeting as it served a secondary function to create some form of direction to an otherwise disparate departments around a common goal (as well as moderate hegemony).

Friday, April 21, 2017

Beyond Kanban

In the last lesson learned, we saw that Kanban is easy. It's easy because it only describes how work flows - there's a whole depth to it with value stream mapping, prediction, planning and measurement but that's really not that complicated either as we'll see soon enough. What Kanban cannot solve is how the work is understood.

In manufacturing, the origins of the culture that spawned Kanban, the product is always the same...therefore (much like a McDonald's) there are finite, well defined steps that procude a specified outcome everytime (within some degree of acceptance). In software, there's the same sort of problem but the product is different every time...

Since the product is different every time (if it wasn't we wouldn't need to build something new) the "specified" differs every time as does the "acceptance" and the "degree". Stephen Connolly eloquently describes this issue in this blog post

As Stephen describes by way of the problem with the shields on the USS Enterprise, the understanding of the feature will need to be clarified with the upstream party. Add a button to the screen, make it do something. Also make sure shields can be activated in other ways - different interfaces. the upstream party - shield system engineers (Scotty?) needs to make sure those shields can be activated from multiple sources. Now the shield becomes downstream for the activation call. Shield engineer specifies interaction protocols maybe defines some security practices - who makes that decision? Now it might be a deeper matter of verifying who is calling to raise the it Borg Captain Picard? Real Captain Picard? What's the override protocol? Does it need to change for this? Now it gets complex...what was a button is now a security and safety concern since were opening up access to a critical safety system...

Guess what? This is what Lean Change Management is all about - managing this sort of change without invoking filibuster (or bike-shedding, if you will). Map impact of the change, radiate change info, talk to people about the change, make changes small so they succeed. But I digress...

Point is, when we go beyond the initial ask (the happy path) and we have different requirements for each change, as Stephen points out, things need to be explicit. Kanban doesn't specify how this is done in software, but other processes do - to a degree.

In TPS (Toyota Production System) all the details are worked out well before the manufacturing process starts. In Software, that's generally not a good way to work. I recommend taking a bit from Scrum and breaking down the details at the start of some Kanban there is no iteration, but there is a work week so why not use that? Start going through the feature stack each week start. Fill in details about the top features and break down the features into work stacks with details. If the stack starts running low mid-week, put in a kanban for feature breakdown. Stop the line, breakdown features and reiterate. This way there is continual flow.

One thing this does is force small changes. Small changes are critical to agility, they are mutually inclusive. You have to be able to ship and switch gears to be agile - by definition.

On a personal note, I've always found the Sprint (of the Scrum process) awkward due to the fixed time box. Because of that fixed time box, we're forced to jam ill-fitting work into that box. Often we over-scope because we're inherently optimistic - to a fault. It makes more sense to me to make landmarks out of shippable features or feature sets. Estimation comes from measurement of past sizing...instead of asking "how long will this take to develop" we use past data on "how long will something this size/complexity take to deliver", a more important prediction by far. It's also easier to measure - when did we start and when did we release it and actually start to realize its value (lead-time). That's what really matters anyways.

Thursday, April 20, 2017

How to Kanban

Kanban - unless you've been in a prison camp for the past few years, you've at least heard about it. It's origins are in manufacturing - where they have to repeat the same task over and over (major implications). It's been ported to software development. The basics of the entire TPS (Toyota Production System) are about reducing waste and Kanban is one small part of that entire principle - Kaizen (reducing waste/continuous improvement) is one of the other important principles that make the TPS successful. In order to truly port those principles into Software Development, some of those principles need to be applied differently (in a software environment sort of way) in order to optimize throughput and reduce waste (the main goals of the TPS).

The start is the all ways. Start building your process with the end in mind and with the realization that YOUR process will depend on YOUR environment. Perhaps your environment benefits greatly when there is waste or when things take longer to accomplish, IDK. I'm only taking the happy path into consideration here - where your goals are to reduce waste and increase throughput (without reducing quality or increasing costs).

Most commonly, in software, the application of MVP (Minimum Viable Product) is used to reduce waste and that is a good start. Sometimes, and probably not rarely, important parts (artifacts) are cut out of the process in the guise of reducing waste. It's been covered already so I won't go there. I will focus right now on the kanban itself...the card.

A kanban is a card that represents a work order from downstream. Kanban move upstream and results move downstream. The kanban is a visual, tangible representation of something that a downstream step needs in order to do their work to deliver working product. All the way downstream sits the consumer. In software, data flow one way, kanban flow the other way.

The Kanban board is a visual way to see what's going on, but its located centrally somewhere so everyone can see what work is in process. This works when the team is co-located, but can easily fall apart when distributed. Each area should generally have its own board. These boards represent requests for work. Downstream should have a prioritized queue of all the work, those cards should be maintained by whomever is directing or requesting the product (the program manager, "the customer", the product manager, the BA whatever). The thing is, kanban move upstream in a pull system...usually starting at the end of the dataflow...the end-result.

For each work request, you need more than one for the board and one for the person doing the work. The one for the board is so that everyone else knows where something is...when its being worked on, the person/pair/team doing the work is indicated by placing their card on the work card...they get a copy of the work card.

Lets take a quick walkthrough of this process in action -
Lets say were doing a User Management thing. First story is add users - "Add Users Page" kanban goes to UI team. "User Persistence" goes to the backend team just before the UI team really needs persistence. Integration happens, goes out for testing. QA almost immediately sends a card to the UI team "View Users" because they can't see the users they just added. UI team puts a card in to the backend team "Get Users", works on view while waiting for data. UI team gets back the "Get Users" card and integrates. .QA gets the "View Users" card and checks it out. Now someone has the idea to edit users so those cards flow up and the feature flows down. Disable user becomes a thing and so on and so forth - cards flow upstream, features flow downstream.

That's it. That's all Kanban is. It sort of solves some things, but really there's a lot more to the whole TPS system that is of GREAT value to software teams if one has the impetus to use them...continuous improvement, Gemba (go to where the work is)...Gemba is a good one. It's just walking over to someone's desk and seeing what they're doing and how they work - understand their work and challenges they face. Pretty easy right!

Here's a good one - Stop the Line! If something isn't working right, stop the line and get it fixed - crash the problem, meaning everyone get to it and anyone has helpful ideas please let them be known.

Continuous improvement - always look for little gains in efficiency and reducing waste. Maybe its wasteful to have 4 manual steps in some process that could be automated and would reduce process time 80%. If that process is repeated 3x per day that's a lot of waste to reduce.

The moral? Doing Kanban is about an entire cultural alignment around Lean practices. It's pretty easy to do with some basic principles...and some obvious benefits...once you understand the nature of it.

The next daily lesson learned goes deeper into some details about the details.