Tuesday, February 6, 2018

Web Basics - TLS/SSL https

We've been looking at the basics of the internet. If you've been wondering about how it all works or are interested in web programming, you need to know the things in this series of posts.

Today's topic is TLS - Transport Layer Security. The transport layer is essentially the connection itself. The web can be divided into a model with 4 layers - two of which we've been talking about: application (HTTP) and transport (TCP and UDP). The other two are "network goo" that we really don't interact with directly. They're important, don't get me wrong, just not important to this series.

As we saw in the last post on TCP, your information is flying around the world at light speed. With the right equipment and wrongful intent, someone looking to make a buck could easily tap into your data in transit (that's what we call it when its on the move) and sell your information (usually a big batch of information) to someone who will exploit it to steal money. That is, unless its scrambled before it's sent, then unscrambled on the receiving side. Enter encryption.


Encryption



The newest big business buzz of currency - crypto-currency -  is all possible because of encryption (that's the crypto- part). It's built on the premise of uniquely encoding a "block-chain" and adding that to the existing chain to make it more valuable.

Encryption took off during WWII because radio transmissions were used by all of the militaries participating in that war. As we know, anyone can tune into radio frequencies and listen in (we can also listen to the radio transmissions of the cosmos - all the way back to the beginning of our universe!). Unless you can send a message in a way that only the receiver knows how to understand, you're toast! Every one of your moves will be known. It would be like playing chess while thinking your whole strategy out loud - you just can't win that way!

So they encoded the messages. With the messages encoded only those listeners with the decoding sequence would be able to understand. The U.S. got really good at cracking the code - which was one of the main reasons why the Allies won. Another was the perseverance and sacrifice of millions of lives of Russian soldiers. And the third was massive industrialization in the U.S. - both automated and manual industry.

History lesson aside, encryption has been used to protect privacy long before the internet. In modern times, it is used to protect data both in transit and at rest (in a database or on a hard-drive). TLS represents encryption in transit. SSL (Secure Sockets Layer) is the outdated predecessor to TLS - it's been deprecated by the authorities on internet security (the IETF*) as of June 2015.

TCP establishes a connection to communicate between two servers. TLS secures that connection by ensuring that all information transmitted through it is encrypted. The mechanisms for applying this encryption involve a certificate.

Certificates



Certificates operate on a trust basis. There are companies that issue certificates (issuer). Those companies are called certificate authorities (CA) and your computer has their root certificate pre-installed. If you are securing your server, you would purchase a certificate from one of those companies. Your URL would then be registered to that certificate. You install the certificate on your web server. When https requests are made to your server, the requester gets a copy of your certificate. Your certificate is used to establish your authenticity. It's kind of like a driver's license, passport, or other form of id.



If you are the requestor, your browser will check the certificate's signature against the signature you have on the root certificate of the issuer. The domain in the URL also has to match the domain name on the certificate you receive from the server. If there is a match, the server has been Authenticated. Once the Authenticity of the server has been established, your computer and the server will generate an encryption key for the session. All of the information shared between you and server will be encrypted and decrypted on either end using that key.

Issues

This site is not https, but it's readonly - you don't exchange any sensitive information. Be careful when you have sites that require sending sensitive info and there is no https or it is mis-configured.
This site is configured for https.


This is how most of your information is secured on the internet today - provided you and the server are using https properly. Often we see misconfigurations on servers or servers that still support unencrypted http connections (http without TLS). There are also different versions of TLS which creates more configuration issues. The best you can do is pay attention to what your browser is telling you and think a bit about what kind of information you are willing to compromise - and remember some hackers are fairly sophisticated and can piece information together from multiple sources if you are a specific target (e.g. have a lot of money or power or work for a target organization/industry).

TLS works well to protect us when configured properly, but we should still remain vigilant. It can be easy to think that https solves all of our internet security problems, but there are other ways that hackers will try to pwn you.

Continued Learning



Encryption is a vast subject in and of itself. It comes in many flavors and varieties. There are one-way and two-way hashing algorithms, asymmetric and symmetric keys, private-private and private-public keys. And it all involves some pretty intense mathematics. Crypto-masters are a rare breed but the work they do is vital to our lifeblood - secure data!


IETF - Internet Task Force



OWASP is the go-to for internet security - they have great info about TLS



Some certificate authorities along with more details are listed on WikiPedia here:


Wikipedia has a lot on the subject of TLS in general:



Thursday, February 1, 2018

Web Basics - The Internet - TCP/IP

Yesterday, I posted about the basics of HTTP. Of all the message protocols, it is the most common one used on the web. If you are in IT or use the internet - basically if you have a pulse - and you are curious about how the internet works...or, I don't know...want to learn web programming, then you should also learn about TCP/IP. That's TCP over IP.

But First - UDP


Our journey though the internet begins at your computer. Let's imagine that it's only connected to one other computer through a wire. That's the simplest kind of network. Your computer is networked with another. Both computers have a network interface - which is a card or microchip or something on your computer that knows how to connect to the outside world. Think of this device as a telephone.


The telephone allows you to connect with another person at a distance. You can talk and listen. A network interface card (NIC) can send and receive data (as packets) over a network. Your computer talks into its NIC, and its NIC uses the network to talk to other computers...just like your phone!

With the 2 computer network, the NICs in both computers will need to know how to exchange data. The simplest way is to just send the data to the other side. This is called UDP (User Datagram Protocol). This would work well in a simple network like two computers over a single wire. But lets imagine those computers are very far apart - one in New York and another in Los Angeles. And let's imagine they're connected over a telephone wire. And let's imagine they're sending data over that wire.

TCP


Since those old phone wires weren't designed for data - they were designed for talking - some of the data might get lost along the way. Luckily there's TCP (Transmission Control Protocol). TCP is a network protocol where the receiver tells the sender that it got the message and how much of the message it got. If it didn't get all of it, the sender sends the missing pieces. Sending this way takes longer, but it makes sure the right message was received.

Protocols are there to make sure we get the right message. They exist in everyday life, we call them manners - the pleases and thank you that ensure clear communications. The army has protocols for sending messages over radio. Imagine if a platoon heard the wrong thing over the radio - it could march straight into enemy hands! There are even communications protocols for Presidents of the United States of America! Those protocols keep the people of the world informed without going into a frenzied panic - or marching straight into enemy hands!

IP


The internet is a wide network - it spans the globe. Even into near-earth orbit. The ISS and its occupants tweet from the space station! This web of computers is very complex - it has smaller networks within and it has branched off networks. It exists over wires and radio signals. Each connection has an address. The address schema (a schema is a way to organize something) is called IP. Don't confuse this IP with Intellectual Property - also technology related. This IP is Internet Protocol.

IP is how the systems that run the network know how to find your computer. It's how your computer knows how to find NASA.gov. It's how your computer sometimes knows how to find your printer...and sometimes when it doesn't.

Your computer can use IP to talk to itself. There's a special address for this: 127.0.0.1 there's also a special name for it (called an alias): localhost. Your computer is also configured with an IP address for the internet - the network gateway. That IP address is where your NIC sends all of your requests that are bound for the internet. There could be other devices connected to your network - like your printer, other computers, servers, storage, your home, TV, etc. Those have their own IP address on your network. They can communicate within your internal network using that IP address. They can also communicate on the internet through the same internet gateway as your computer!

Ports


Along with the IP address, computers listen for incoming calls on Ports. Ports allow one computer to have many "conversations" at the same time. Ports can be numbered 0 to 65535 (that's 16-bits unsigned). Some common ports are:
80 - http
443 - https
There are many other common ports for things like ftp, ssh, and mail.
When your NIC sends a message, it opens up a port. It uses that port to send and receive messages to and from the other side.
What the receiver sees is your internet gateway - your router. Actually the last router in your network before your message gets sent into the great beyond of the internet. After that, it gets passed around the internet routers to your destination - if it can be found.

Continued Learning


The internet may be a fascinating and complex web of networks that span the globe, but almost every bit of it operates on a few core technologies. TCP/IP is how connections exchange data over a not-so-reliable network and HTTP is how two computers know how to interpret what's sent over the networks.

There is a glaring issue with all of this - if all this data is flying over the network and it really is possible to tap into those messages, how do we keep others from prying in on our messages? The answer is encryption! TSL to be specific - that's https to you. Well see how that works and how your computer knows the IP addresses of websites in upcoming posts in this series.

More information on UDP, TCP and IP is available in Wikipedia:

https://en.m.wikipedia.org/wiki/User_Datagram_Protocol
https://en.m.wikipedia.org/wiki/Transmission_Control_Protocol

Wednesday, January 31, 2018

Web Basics - HTTP

I occasionally get asked to mentor by those interested in learning to code. My response and approach will vary depending on the situation. I've done this in a pair programming context, over the wire, and by simply offering advice from time to time. I enjoy mentoring, especially when the mentee shows commitment and drive. This blog was originally started on the basis of providing mentorship to my readers. With that, let me get back to my roots for this post.

If you want to learn web programming, you should probably start with the basics - HTTP. If you are mildly in IT (or have a pulse) you should understand the HTTP protocol to some extent. I'll start you on your journey to understanding this lifeblood of the internet.

http logo

What is it?

HTTP is a protocol for communication between two computers. In fact, it is an acronym for Hypertext Transfer Protocol. In HTTP, there is a request sent by a sender (client) to a receiver (server). And the server should send back a response.

The request has two parts, the headers and the body. The headers has information about the request and the body is where you put data you want to send to the server.

Verbs


There are different types of requests. Each type uses a different HTTP Verb (also called Method). Verbs are an important part of HTTP. They tell the server what you (the client) would like to do.

Verbs can be split into two categories - queries and commands. The most common are GET and POST.

GET is used to get a web page or some data. It is a query.

POST is widely used to do something, such as send an email or submit an order form. It is a command.

PUT is another command. PUT is meant for updating something. You might POST a file and later need to update the file's contents so you would PUT an update. There's also a DELETE command.

Another common query is OPTIONS. This is a request foe the server to tell you what you can do. Often servers will only accept certain verbs. OPTIONS can tell you which ones.

There are many others verbs. You can find more at the official w3 site at the end of this post.

Responses

When a request is made, the server can respond. One thing included in a response is a response code. A response code has 3 digits and ranges from 100-599. Each hundred has a different meaning. 100's are reserved, 200 level is success, 300 is content changed, 400 is a client error, 500 is server error.

Some common codes are 200 - OK, 204 - OK with no data returned, 302 - temporary redirect, 404 - not found, 500 - server error.

In addition to the response code, there may be other headers and content returned from the server. Some of the headers describe the content. For example, Content-Type and Content-Length. These tell your client (browser for instance) how to handle the content in the response. Should it display the content in a browser? Save it to a file? Open it in a plugin?

The Network

When we use the web, we are primarily using HTTP over TCP/IP (your next topic :) ). TCP is the transport mechanism/protocol (how) and IP is the addressing schema (where).

When you send an HTTP request, the server will need to know where and how to respond. Some of that information is in the request headers. And some of it is handled by the networking systems (everything that moves your request over the internet, from your client to the server).

Continued Learning

Most programming languages have HTTP clients that you can use is your programs. This is so common nowadays, that you cannot do programming for long without using an HTTP client. And when you do web programming, its at the core!

For example, web apps you use all the time use AJAX (Asynchronous JavaScript and XML) to make HTTP requests using the JavaScript code that the app runs in your browser! These requests do all sorts of things like fetch data for part of the page and send data to the server. When you see part of page spinning while it loads, that's usually the app waiting for a server to respond to an AJAX request!

You can find version 1.1 of the official protocol at rfc2616 

Learning HTTP is the start of your journey to understanding how the internet works. We'll explore TCP/IP in the future...its another core technology behind what the world runs on today.

Thursday, December 28, 2017

Customer Redirection

When your company no longer offers a product and you run out of stock you have to inform your customers. This communication could lead to two different reactions from your customer. On the one hand they could buy an alternative from you, on the other hand they could look elsewhere for the same product.


You could be proactive in keeping your customer happy by offering an alternative that you sell. Make it as easy as possible for them - and while you're at it, offer it at a discount or send a sample size. They're already disappointed that a brand or product they have come to love and trust is no longer available. That broken trust could easily transfer to you...this is an opportunity to make it right.


Here's an anecdote to illustrate the point. Today my wife received a disappointing email that a key product in her melody of beauty products was no longer available, and this coming after she ordered a refill. Needless to say she is going to have to find another product - but which one and from where?


First, she'll scan the web for the same product. Then she'll see what others are recommending as an alternative. No one likes to go on a whim with this kind of thing, it's all about word of mouth or endorsement by someone she trusts.


She does trust the company she placed the order with, as do many women and men. This email is an opportunity to redirect her toward a viable alternative - if it costs more, offer a discount for the first purchase just to make things even out.


But remember, you've also got to make up for the negative impression so offer another discount on top of that or offer to throw in a selection from some items based on her shopping history (another opportunity to make a second product introduction). Use your customers' product ratings or in-house expertise to drive product selection. Or for a more advanced solution use machine learning.


With machine learning, we can build smarter solutions to match your customers with the right redirection. When you incorporate our solutions in your communications chain, you turn missed opportunities into sales opportunities.

Thursday, November 16, 2017

Culture of Interactions

The Agile Manifesto puts the value of "individuals and interactions over processes and tools". Meaning that a conversation is where the work starts. Meaning that if your processes/tools are getting in the way of human interaction then you aren't practicing this value and not getting the benefits of Agile. For example, work requests delivered long-form via email is not an interactive process.


Agile processes generally use User Stories as a starting point for any development work. Just what is a User Story? It's really a placeholder for a conversation, no details except a concise introduction of the user's goal - "As an end-user, I want to print my essay so that I can mark it up with red-ink and make revisions based on the markup." This is the user's perspective. It isn't even about the user story, it's really just there to keep track of something.


As a technologist, I might want to poo-poo the idea of printing and using ink...or even wasting paper as an environmentalist...but this user story is her story, not mine. Through conversation, there may be a change of heart, but that comes into the fold of negotiations.


Through the initial conversation, we should come up with some Acceptance Criteria. It's documented, but not written in stone - it's just a first pass. It should be fairly basic and relatively quick to implement. May not be complete yet, but could be. The Acceptance Criteria is also known as the Acceptance Test...User Test...Story Test. It should describe how the user would validate the implementation (then we automate it, but that's for another time).


"When I click a print button, the essay should print. I should be able to go to my printer and get the printed essay from the printer. All pages should be there."


What can go wrong there? If have any experience with printers, we know there are too many ways that can fail! Printer out of ink, paper, jammed, just not having it that day, not connected, off. Computer/OS not happy with printer config, dialog comes up before printing, not connected to network. Network failure. Mercury in retrograde.


Perhaps in some future world, printing will just work, but for now we need to revise the Acceptance Test...that's why we need to work on it together...


"I can't promise the essay will print, but I can make it so that there's a print button which will tell your computer to print all pages of the essay to the default printer, or I can have it launch the print dialog."


There's some negotiation. There's supposed to be, it's another Value. But first, we should try the simplest thing that works...do whatever the OS does when you want to print. And that just might get the job done.


Without the conversation, we'd have wasted half the project schedule on documenting something that was nearly impossible and then attempting to implement something that was doomed to fail anyways...at least when Mercury isn't playing nice!

Monday, October 9, 2017

Cause of Death: Optimization

So you've got this team - developers, business reps, product managers, QA engineers and they all need to work toward building out products that add value to the business (either directly, by reducing risk, or increasing efficiency). The business has some goals. You have some goals. Bonuses and career advancement and frankly job security are tied to those goals.

At the same time its difficult to find the right people. There are plenty of applicants but who can be trusted and who has the right level of expertise to jump right in and get things done because the backlog is growing and the work is piling up and everyone wants to know if you can deliver and of course when.

The New York Times wrote up a great article on how to find the right candidate. The thing is, once you've found the right candidate (well-qualified, energetic, enthusiastic, etc..) what happens next depends. Lets walk through the process and see why.

Ramping up


In order to make good on your deliverables, you need more headcount so you can get more done. You're at capacity and just can't ask your people to put in anymore. Once that's approved you now have the task of staffing the right person. You need to throw someone in straight away and start knocking things out because your way behind already and in order to justify the added cost, you want to show results.

So now you ask around internally - this is the standard pattern - if anyone knows someone who is ninjas at xyz. I'll tell you right now everyone who is ninjas is currently working. You'll need to do some sales - or typically hire a salesperson (recruiter).

So now you're in it with the recruiters for about 20% give or take. For a 100k salary, that's 20k...say I bet that 20k would make a great training budget! The person you really think you need actually costs more than $100k. First off in a supply and demand economy where supply is limited and demand is high you're on the downside of that. So $130 plus all the other costs and recruiter fees.

After the Onboarding

Once you've shelled out the upfront costs and now this maverick is on your team they'll need to be brought up to speed. For a while this will slow down overall production. One or more of your resources will need to be involved regularly in working with the newbie to get them up on the tools, practices, procedures.

It'll take some time before the dust settles. Lets estimate 4-6 months depending on the learning curve for a well-designed, documented, unit tested, and maintained system...1-2 years for most systems...longer for train wrecks. You are likely to be in the 1-2 year range and please don't lie to yourself because that's counterproductive and were focusing on being more productive. If you know I'm wrong in your case, then you probably don't need me to tell you all of this anyways, you can close the tab and carry on as you were.

Once the Dust Settles


If you're still with me and haven't gone back to balancing your budget or sipping your morning coffee while touring your friends' feeds, I'm glad you decided to stay because it's about to get real.

Taken on the surface and without considering long-term stability, hiring the right person in that moment is a greedy optimization. Greedy optimizations seek to optimize the whole by optimizing the parts of the whole with the hopes and assumptions...well...that it will improve overall performance. Often active stock traders attempt to do the same. Rarely does it work out; one reason is that active traders are likely to sell when things go south. In stock trading, emotions sabotage long-term results. In the same way, needs-of-the-moment hiring practices sabotages long-term health of the team when not considered along with overall team dynamics, culture, and performance implications.

A rockstar-unicorn-ninja-whatever is not going to solve your productivity issues if those issues arise from broader optimization issues - morale, communications and team cohesion especially. What WILL happen if there are deeper rooted issues is that the ninja will come on for a bit, do some things and ninja-smoke off when they figure out how deeply rooted the issues are which causes a degradation of the three most important factors to said ninja - autonomy, mastery and purpose. Not only do we strive for those three, but we are inherently social beings - nature calls for having social well-being. When our social well-being is not upheld we will fall behind our potential.

And you're still on the downside of the market, so there's that problem. I will guarantee you that salespeople are at it regularly pitching the great folks on your team. Any give one of them has inboxes full of emails and messages from recruiters. The other side of most of those openings are mirror images of yours in that their either adding or replacing staff because they've got a pile of work to get done yesterday. Some aren't but most are.

Solutions

You've got to offer something they don't. Compensation, bonuses, and benefits are a given in any decent market. Anyone worthwhile can get those anywhere. Your business may operate in non-IT markets and perhaps are on the upside of the supply chain (unlike in IT). But in your technology departments you're on the demand side - you need good people and the supply is limited.

In order to combat such a position and come out ahead you have to optimize for the long-haul. There may be some short-term goals that occupy much space in the present, but even if those are met you may end up in a position that doesn't set you up well for the next steps, the next goals...it's all about being strategic! Look for more global optimizations rather than local ones - unless the local optimizations truly do lead to a global optimum.

What you've got to do if focus on 2 things - optimizing your workflow and retaining talent. Both of those involve investing in the talent you've got on-hand. If you are willing to take a $20k hit for hiring new personnel or a $120k+ hit annually you certainly have the ability to work on improving the personnel you already have. Additionally, you can certainly afford to improve your system or process. Take a serious look at any bottlenecks - technical or otherwise - and address knowledge gaps, personnel conflicts and process issues BEFORE adding more calamity to the mix by hiring.

Get out there and spread some buzz in the developer community about your organization. When you're ready to hire - when your house is in order and you are prepared to properly on-board - you'll have developers coming to you rather than having to hard-sell. Basically you'd be marketing which makes it easier to sell. Having a great team, process, and organization to brag about is a great way to attract the top talent that you will now be able to retain.

Non-Solutions


Consider hiring all rockstar devs. They may all be fantastic in their own right. And at the same time if they can't work well together or communicate or the process hinders productivity anyways or they end up building the wrong things - it doesn't matter how good each one is.

And how about the case of pushing to get a release done? By pushing I mean everyone working late and just churning away...dropping principles and mounting up debt to get the thing done. That's a way to burn folks out and in this heyday of IT, that's a recipe for exodus. As far as turnover goes, going back to the active traders, fees are costly and chew away at the gains. You've got to have retention for the long-haul.

But that's not the only danger with churn and burn. What's left behind may end up slowing things down significantly in the future. Move too fast, slap things together, drop important practices and time for reflection, team building, etc and it'll only lead to further entrenchment of negative reinforcement - a negative feedback cycle. This has been well-known for a long time now.

Adding people will always slow things down for awhile. Adding them to a late project is classic mistake that no-one should be making anymore.

Outcomes of Successful Solutions

Once you've applied a successful solution, you should be able to compare the outcome to the baseline. That's right, make sure to capture your current state so that you can KNOW that you've solved solved the problem.

You should be seeing better employee retention, improvement of skills, lower stress, higher morale, engagement, proactive behavior, lower bug count if you'd like to address that metric, and more valuable production if your process is really in order. The latter may be beyond your own sphere of control, but perhaps it is not beyond your sphere of influence, but since this post is about the productivity of your own group it may well be beyond scope for now. However, that's arguable since your productivity should actually be measured by the actual value delivered. I'm sure to do a post in the future about that.

Friday, October 6, 2017

Cultural Evolution Part Three

First off, I must apologize for skipping over the topic I promised in the last post in this series. I'm working very hard on that promised post about the algorithm that looks good on paper. Something has come up during the course of a book I'm writing that I feel an urgency to blog about. It fits nicely with the theme of this series, so here it is.


Think about how we are biased from a very young age to play against other teams. To compete and even to think of the others as lesser beings or the enemy. We play soccer, softball, football and chess with the intent to beat the other side and celebrate victory. In our passive involvement in professional spectator sports, we get fired up about our team and even hurl insults at fans of the opposing team in and out of the stadiums and parks. Some of these are our colleagues we work with, others just fellow humans - probably good people too. But that's how we're biased from such a young age. It crosses political boundaries as well - cities, states, nations. As well as cultural boundaries. Always there is this notion of "the others".


In business, and at your company, you've got many teams working in different areas. This "team bias" that's built-in has to be overcome if your company is to act as a whole. While there will always be some level of internal competition, for better or worse, you can tilt those scales to better by setting and socializing the overarching goals of the company. In this way all if your players will be running in the right direction.


How does this help? Setting, socializing, and tracking business goals at the organizational level puts everyone together into the same team. Who or what do you compete for in that scenario? Rather than competing between departments, units, or teams the competition is aligned toward the achieving those goals. As always the organization's goals need to be in alignment with its mission, vision and values!