This is the second series describing the ADD: a radically more productive development and delivery environment. The first article is here: Intro and described the truth and lies about developing software. The second article dealt with ‘Testing’.

Grails, Groovy, and Java

The primary stack I believe is a “Best Practice” is Grails. It is in it’s third full generation, and with each generation it gets better, easier, more powerful, and more ‘aligned’. This last part is not that common with frameworks. A lot of people start writing a framework and it does more and more. With more and more code. The Grails team has been great at ‘pruning’ and ‘aligning’ with Spring and other frameworks.

Using Spring alone is certainly a reasonable practice. The problem is people tend to use Spring wrong. I don’t know why. Either they don’t read the tutorials… or they get confused and a deadline is approaching… or they are cowboys (or cowgirls) that wander off into new territory of abuse to the tools that are in front of them. I used Grails at a company that committed to Spring. So I simply switched to Spring and my code was simple, functional, well-tested, and clean. But the rest of the code base was a complete mess. So there is nothing wrong with Spring but it is harder to use properly than Grails. Spring Boot is trying to help with that. Grails is just better.

Grails is ‘opinionated’ and ‘functional’ and has plenty of examples to show you these opinions and capabilities. We can see a lot of it in the ‘petclinic’ example.

Build with Gradle

Grails uses Gradle to build the project. It used to have it’s own system, but it pruned that away when Gradle became stable and capable. Gradle won a war against other build systems so Grails honored the winner.

Align with Spring Boot

This is the second series describing the ADD: a radically more productive development and delivery environment. The first article is here: Intro and described the truth and lies about developing software. Basically, assuming you are doing everything in the first series, and are using relatively modern technologies and techniques, the ‘language’, ‘framework’, ‘stack’, and ‘methodology’ does not matter very much. And how it matters has a lot to do with the development team and the customer. Customer happiness is the ultimate goal, so if you are not making your customer happy, you should make incremental adjustments (Educated Guesses) to do so. And if you can not “continuously” deliver valuable products to the customer, you need to figure out what is the bottleneck and get rid of it (e.g. google Throughput Accounting).

Testing

“Hey… you said testing wasn’t important”

No, I said automated testing was not necessary. At times it is not even valuable, but at other times it is.

There is no way to avoid ‘Testing’ unless no one is going to use your product. ‘Using’ is ‘Testing’, so everything useful is continuously tested. You have one user, you have one tester. A million users, a million testers. So you can’t avoid testing you software. You can only augment/change who or what tests your software. There are a number of options:

This is the second series describing the ADD: a radically more productive development and delivery environment.

Review of the first series

The first series starts here: Intro and ends here: Voilà!. The bulk of the series was describing how to combine four ingredients:

• Amazon EC2 http://aws.amazon.com/
• Amazon S3 http://aws.amazon.com/
• GitHub.com http://github.com/
• HipChat http://HipChat.com/

in a way that gives individuals and teams 4x or more productivity. These resulting environment solves problems for all three roles in a development team:

• Changers
• Watchers
• Machines

And can be envisioned as so:

Along the way, the ingredients combined to make:

• Machines automatically launch and configure themselves, including hooking them up to each other (auto-wire)
• Changes to IT roll out automatically to all the running machines.
• Changes to the application roll out automatically to all the running machines.
• Creating new environments to be as easy providing a simple parameter (e.g. “fed2”) to a CloudFormation
• The whole thing working on EC2 or inside a Vagrant container… or even on bare metal (but that was not shown)
• Incredible visibility into everything that happens within HipChat… for the whole team.

By the end, we had machines tell us when they launched, when they were operational, when they were deploying a new version, and whether that new version deployed successfully.

and finally, we had a running application at a human-usable URL:

That could be scaled to any number of servers as load demands.

Second series

The above is the core of the ADD and enabling team productivity and customer visibility (even happiness) to shoot through the roof. It by itself is basically ‘complete’. It is missing some details, nuances, augmentation, and such. But it alone is the core of a 4x productivity.

“Complete! What nonsense! Where are your automated tests! How do you check for code quality! What kind of programming language do you use! And where is your description of Scrum, XP, Waterfall, etc. methodology to use with this ADD?!!”

Ah… yes. Over my thirty five years and estimated thirty five thousand hours of developing software, I have encountered all these things. I do have opinions on a lot of these things. Preferences. Best practices. You may be interested. They matter. A little.

How little? Maybe… maybe… 2x.

This is the elevent installment of describing a radically more productive development and delivery environment.

The first part is here: Intro. In the previous parts I described the big picture, Vagrant and EC2, node initialization, the one minute configuration HeartBeat, HipChat integration, presence, EC2, configuring nodes, inter-machine presence, and configuring an application server and its’ stack.

It’s Alive!

We are close to having a viable semi-production environment running the latest version of our code, and with an incredible capability of configuring, controlling, and being informed of that environment. With the simple line:

1
2
3
4
5
6

mkdir -p /root/app/
cd /root/app

git clone git@github.com:grails-samples/grails-petclinic.git
cd grails-petclinic
./gradlew run

We got:

This is the tenth installment of describing a radically more productive development and delivery environment.

The first part is here: Intro. In the previous parts I described the big picture, Vagrant and EC2, node initialization, the one minute configuration HeartBeat, HipChat integration, presence, EC2, configuring nodes, and inter-machine presence.

A good, opinionated, framework

Back in 1972, Smalltalk became the first Object-Oriented Programming Language (Simula was Object-Based but there is a difference). For decades this kind of language was ‘esoteric’. It was like LISP or Prolog or APL: somehow exotic and inaccessible. I was lucky: I had access to Smalltalk at Caltech. I had access to lots of crazy expensive things at Caltech and that made the exotic (e.g. making your own chips) into the mundane (e.g. made lots of chips, they were commonly broken, had to be re-fabricated, and I eventually got bored with all that and moved a level up).

This is the ninth installment of describing a radically more productive development and delivery environment.

The first part is here: Intro. In the previous parts I described the big picture, Vagrant and EC2, node initialization, the one minute configuration HeartBeat, HipChat integration, presence, EC2, and configuring nodes.

Using Presence for configuration

So far, presence has been just information that ‘humans’ consume. It shows up on dashboards, in chat rooms, and so on, but nothing has acted upon it. Until now!

We have ‘app’ and ‘db’ nodes. Clearly the ‘app’ nodes need to find the ‘db’ nodes or the app is not going to be able to persist much. The ‘db’ here happens to be ‘Maria’ but it could be anything from a single DB node to a cluster of Riak nodes. At the moment, I just want to get the information that a ‘db’ node knows (“I exist!”, “My IP is this!”) over to the ‘app’ nodes so they can process it.

It is already there?

But wait? All nodes have ‘repo4’ for writing? Don’t they already have everything in ‘repo4’ for reading as well?

And so they do. The presence information is already there waiting patiently for somebody or someboty to read it.

This is the eighth installment of describing a radically more productive development and delivery environment.

The first part is here: Intro. In the previous parts I described the big picture, Vagrant and EC2, node initialization, the one minute configuration HeartBeat, HipChat integration, presence, and EC2 integration.

Configuring Node Parts and even fuller Presence and visibility

So far our node was a generic ‘ControlNode’. A ControlNode is a node that is in your data center (to be near the other nodes), similar to your actual nodes, and configured to be able to do interesting tasks. It is not a critical part of anything, so you can fiddle with it and just throw it away. Say you want to start configuring a database node. You launch and login to a ControlNode and then try to install a database (for example Maria).

This is the seventh installment of describing a radically more productive development and delivery environment.

The first part is here: Intro. In the previous parts I described the big picture, Vagrant and EC2, node initialization, the one minute configuration HeartBeat, HipChat integration, and presence.

Fuller Presence and EC2 Integration

In the previous part, I went through a very simple but powerful model of ‘presence’ using simply GitHub repositories. The content of those presence statements was enough to figure out what nodes exist, but not much more about them. The second level of presence is to update the state of the node as it changes. For example, a node goes through a few bootstrap steps:

• presetup – The node before any updates are possible (no ability to change status)
• setup – The beginning of the ‘setup’ phase where the node is alive enough to change it’s status
• initdone – The time a node is done are initialization and can start doing ‘work’ as the ‘nodepart’ it is

A node getting to ‘setup’ is pretty important: before that it may be a zombie! And we don’t want zombie’s in our federation!

This is the sixth installment of describing a radically more productive development and delivery environment.

The first part is here: Intro. In the previous parts I described the big picture, Vagrant and EC2, node initialization, the one minute configuration HeartBeat, and HipChat integration.

‘Part’ Provisioning

Each node plays a singular ‘Part’. A ‘part’ is a unique combination of roles (in the chef sense) that identifies exactly how the node should be provisioned, usually globally, but at least for each stacktype. A standard array of parts would be the LAMP stack:

• Load Balancer (lb)
• Application Server (app)
• Database Server (db)

This is the fifth installment of describing a radically more productive development and delivery environment.

The first part is here: Intro. In the previous parts I described the big picture, Vagrant and EC2, node initialization, and the one minute configuration HeartBeat.

HipChat

The fourth ingredient to the ADD is HipChat. HipChat is meant to help people communicate and see the state of the world. It is an excellent communication program with a plethora of integrations. But for ADD the critical capability is to mix notifications of machines with communication between people. You can also have it be used to drive the ADD system (kind of like a command line) but that isn’t very important since it would just make changes in GitHub which can be made in lots of different ways.

The demo of this is very simple: