This is the fourth series describing the ADD: a radically more productive software development and delivery environment. This series is on estimation, and the first part is here: ADD: Estimation.

Scaled Agile

As mentioned at the end of the first part, the extreme reaction to waterfall was to toss Analysis and Design away, and go from very-loose requirements into actual implementation. I believe this was a complete mistake and has caused these “extreme” agile projects to be much slower and unstable compared to another approach.

The alternative is “Scaled Agile”, and by this I mean both:

• It is better designed for larger projects
• It is achieved by scaling the activities of software development projects to be appropriate for the scale and needs of the project
  • Analysis & Design efforts, deliverables, and precision can be smaller or larger
  • Increments of delivery can be smaller or larger
  • Allowance for iteration (repeating something to refine it) can be smaller or larger
  • Overlap of activities can be smaller or larger

The second item was tossed out with the “Waterfall”, at least by many in the industry. A Waterfall is an extreme version of Agile (basically no during-build agility), but going directly from verbal or spread-sheet based requirements is a similarly extreme version of Agile and is unlikely to be as effective as properly-scaled versions of Agile.

Retail Aspect / Evant / XP / Scaled Agile

In 2003, while at Evant, we took an XP-based product (an advanced retail planning system) and worked on scaling the velocity and agility of that project:

• To a bigger and non-XP team
• Through multiple time-zones, including to India
• Outside the development team into Product Management, Technical Sales, Marketing, and Support
• Into a bigger portfolio of products that had inter-dependent deliverables
• Across the portfolio to other product lines and teams

This is the fourth series describing the ADD: a radically more productive software development and delivery environment. The first series starts here: Advanced Development and Delivery and described the core four ingredients of the ADD. The second series starts here: ADD Stack and described the application stack and system components associated with the ADD. Some of these were Java / Grails specific, and some were more general components and capabilities. Together these went through a huge amount of functionality, flexibility, reliability, and scalability. Both for the IT infrastructure (whether on EC2 or on bare hardware) and for the applications on top of that IT infrastructure. The third series starts here: Advanced Development and Delivery Summary and gave an executive summary of the ADD.

This series is about project estimation, and is basically orthogonal to the ADD itself. You can have good estimates or bad estimates. The ADD delivers exceptional productivity. How much you need to estimate is quite independent of productivity. But by using Estimation and Feedback, you can figure out (a) whether to do a project and (b) whether the people you have are up-to-snuff. If you know you have to do a project and you have certain people: don’t worry about the estimate. It is irrelevant. Just “wing it”. But if you have choices about either (a) whether to do a project or (b) what people to use or even (c) figuring out how good your team is after you do a project, then estimations are the core benchmark to give you feedback.

Estimation is crazy-hard

The problem with estimating is you can’t be good at it unless you have done something very similar before. But most workers in an industry are relatively new to it: less than ten years. And if each project takes a couple years to complete, you may have only a couple experiences in your first five years of work. Change technologies, change roles, or change organization, and your estimates are likely poorly based.

When something is too hard for someone to do easily, you need to decompose it to make it (a) easier and (b) separable.

This is the third series describing the ADD: a radically more productive development and delivery environment. The first series starts here: Advanced Development and Delivery and described the core four ingredients of the ADD. The second series starts here: ADD Stack and described the application stack and system components associated with the ADD. Some of these were Java / Grails specific, and some were more general components and capabilities. Together these went through a huge amount of functionality, flexibility, reliability, and scalability. Both for the IT infrastructure (whether on EC2 or on bare hardware) and for the applications on top of that IT infrastructure.

This series assumes you have at least skimmed the other two series, but this one is the summary of the other two. It is an incremental growth of the architecture in a fairly macro sense, where the details are in the other series.

The three ingredients

Wait, it was four a moment ago. The ADD requires four, but advanced project collaboration requires only three. And at the heart of the ADD is advanced project collaboration. Collaboration has no ‘machines’, just people, so we put aside one ingredient. The remaining three are the nucleus of ADD:

• GitHub.com http://github.com/
• HipChat http://HipChat.com/
• Amazon S3 http://aws.amazon.com/

These three ingredients can give individuals and teams 4x or more productivity. And the reason is the same as for ADD. There are two critical roles in any project:

• Changers
• Watchers

The Changers are gonna change, change, change, change and the Watchers are gonna watch, watch, watch, watch. And all of the people involve need to see what is going on (in the best way for them) and not be blocked getting things done by others on the project. They need perfect transparency. Which will ultimately build incredible trust, knowledge, and skill.

The three ingredients form the core of a three-way Venn diagram, that I will draw in hexagonal rings.

This is the second series describing the ADD: a radically more productive development and delivery environment. The first article is here: Intro and described the truth and lies about developing software. The second article dealt with ‘Testing’. The third dealt with the application stack (Grails and other technologies). The fourth discussed UI alternatives. The fifth added some major aspects to the stack: Semi-Structured Data, Templates, and Dual or Isomorphic scripting. The sixth discussed UI frameworks in a bit more detail and ended with an Angular vs. Ember as a core choice. The seventh went into logging, analytics, and monitoring of the running applications and nodes. The eighth was an overview of Federation Components like caches, queues, payment services, and the like. The ninth was on a highly reliable, scalable, flexibly, and ultimately very simple worker model.

WAR!

Continuing with the worker model, we currently have a non-production Application Server running Grails:

• http://fed1-app1.aws.gaps2c.com:8080

It is running grails ‘interactively’ and ‘off-source’. This is fast for deployment, but it is missing a few things that people usually want in production:

• Well-defined artifact that has been tested and can’t “just change”
• A container environment that can provide certain resources to the application (e.g. monitoring)
• Removal of having to understand the ‘building’ technology from the running technology (JVM)
• That production has as few technologies as possible, and is normally missing ‘compilers’ so people can’t tweak behavior

This is the second series describing the ADD: a radically more productive development and delivery environment. The first article is here: Intro and described the truth and lies about developing software. The second article dealt with ‘Testing’. The third dealt with the application stack (Grails and other technologies). The fourth discussed UI alternatives. The fifth added some major aspects to the stack: Semi-Structured Data, Templates, and Dual or Isomorphic scripting. The sixth discussed UI frameworks in a bit more detail and ended with an Angular vs. Ember as a core choice. The seventh went into logging, analytics, and monitoring of the running applications and nodes. The eighth was an overview of Federation Components like caches, queues, payment services, and the like.

Workers!

Among the more interesting federation / deployment component are workers. Workers are interesting because they can do almost anything imaginable and they do it very efficiently. How efficiently? Well, the perfect worker is at 100% when doing a task and dead when they have no tasks to do. That is pretty darn efficient because it approaches 100% and is only not 100% when we choose to not accept the ‘spin up’ latency of getting a new worker. As the amount of work increases, our pool of workers increase, and the spin-up-latency decreases on a per-chore basis. That is very cool: as scale goes up, efficiency increases. There are a lot of places that is not true for computers, so it is nice when it is.

Work? Chore?

So we want to have a worker model where we can add a ‘chore’ (a unit of work, using a term not otherwise used in IT and CS… job, task, etc. are now ambivalent) to a queue of work-to-be-done, and have something do that chore. There are a lot of ways to do this:

This is the second series describing the ADD: a radically more productive development and delivery environment. The first article is here: Intro and described the truth and lies about developing software. The second article dealt with ‘Testing’. The third dealt with the application stack (Grails and other technologies). The fourth discussed UI alternatives. The fifth added some major aspects to the stack: Semi-Structured Data, Templates, and Dual or Isomorphic scripting. The sixth discussed UI frameworks in a bit more detail and ended with an Angular vs. Ember as a core choice. The seventh went into logging, analytics, and monitoring of the running applications and nodes.

Federation Application Infrastructure

The application stack we have so far:

• UI (both client and server)
• Application Server (Grails, Java, potentially scripting engine)
• Database (Maria or similar)

Is very capable. Using it combined with the ADD ingredients:

• GitHub – With resource, presence, application, and configuration information repositories
• EC2 Instances – Running continuously or based on load, and running their ‘part’ plus any dynamic configuration
• S3 – For resources
• HipChat – To let everyone know

Makes for a very functional application. The nodes and their applications can talk to each other based on presence. The nodes and their application can keep certain data in-memory (cached). The nodes can launch other nodes to handle load or do certain tasks. An Application Server is a very generic thing and can do pretty much anything.

Standard Federation Components

Doing pretty much anything and everything turns out to be very confusing. For people. Big monoliths of capabilities are basically beyond comprehension. And the bigger the monolith, the harder it falls. The more likely it falls. And even if you have redundant monoliths, the system becomes very painful to maintain and to learn.

This is the second series describing the ADD: a radically more productive development and delivery environment. The first article is here: Intro and described the truth and lies about developing software. The second article dealt with ‘Testing’. The third dealt with the application stack (Grails and other technologies). The fourth discussed UI alternatives. The fifth added some major aspects to the stack: Semi-Structured Data, Templates, and Dual or Isomorphic scripting. The sixth discussed UI frameworks in a bit more detail and ended with an Angular vs. Ember as a core choice.

Logging, Analytics, and Monitoring

Returning to a bit more of an ‘IT’ issue, how do we handle logging, monitoring, and analytics on our collection of machines? There are a number of choices:

1. Run our own infrastructure
2. Go with an inexpensive provider
3. Go with a more capable provider
4. Don’t

Of the above, I believe #3 is usually worth it until your scale gets to the point of needing #1. Downgrading from #3 to #2 is fine after you have learned the capabilities you are giving up.

This is the second series describing the ADD: a radically more productive development and delivery environment. The first article is here: Intro and described the truth and lies about developing software. The second article dealt with ‘Testing’. The third dealt with the application stack (Grails and other technologies). The fourth discussed UI alternatives. The fifth added some major aspects to the stack: Semi-Structured Data, Templates, and Dual or Isomorphic scripting.

Which UI framework?

The fourth article discussed UI alternatives, but it was mostly about ‘architectural’ alternatives and did not recommend a particular framework to use. I believe that is the correct order: how you use the framework is usually more important than the framework itself. People can correctly and improperly use all kinds of different technology. Use a hammer correctly and you might have a slight penalty (rubber vs. metal), but use it wrong (your thumb is targetable) and you can be seriously hurt.

There are a number of popular frameworks:

• Angular
• Ember
• Backbone
• Sencha
• etc.

And deciding between them may seem like it should be done as “which has the best features?” but unless one of them has a killer feature that you care about (Sencha can run the same code with both a desktop and mobile UI), and is worth its penalties (Sencha is proprietary and very ‘different’) then ‘best features’ is basically meaningless. A better question is “Which one do I understand the best?” and “Which one can other people on the team learn and share-code the best?”.

This is the second series describing the ADD: a radically more productive development and delivery environment. The first article is here: Intro and described the truth and lies about developing software. The second article dealt with ‘Testing’. The third dealt with the application stack (Grails and other technologies). The fourth discussed UI alternatives.

This article is about some fairly advanced capabilities, but capabilities that many projects find useful, and so should be considered within the context of the overall context. They are:

• Data Flexibility and Templating
• Server Side Scripting

Data Flexibility and Template Systems

One of the more interesting capability of an application is when users can control the content and presentation of information. Content is relatively easy given pure structured information is trivial to store in most any database. As the content becomes more unstructured you need to shift models and store it in a ‘meta’ structure like JSON. As the content becomes bigger, you need the flexibility of very large objects (BLOBs and CLOBs) stored either in the database or potentially within content repositories like S3. Using the Annex model discussed earlier, you can simply store a hash in the database and the actual content on S3, and then the client can pull down the information without taxing your server network at all.

Having flexible data is not very useful unless you can present it. If the data is in JSON, you need to be able to take JSON and render it into text, HTML, or something the programming language can work with. There are a number of template systems out there:

• Mustache
• Jade
• Dust

This is the second series describing the ADD: a radically more productive development and delivery environment. The first article is here: Intro and described the truth and lies about developing software. The second article dealt with ‘Testing’. The third with the application stack (Grails and other technologies).

User Interface

Of all the technologies that change, the User Interface is by far the most diverse and most evolved. Modern databases are actually pretty old technology. Modern languages are not significantly different from ancestors in the 1970s. Computers have the same main components. Programmers interact with them with keyboards and screens. If you picked well, you are programming better (tens of thousands of hours help) but are sadly on pretty much the same core concepts you learned in your first five thousand hours. Sigh.

Except. Except the User Interface. The implementation of user interfaces has grown since the 70s (the first bit-mapped and mouse-based UIs were built way back then) through Smalltalk, Objective-C, Flash, Flex, Angular, and other paths. But even the devices have changed. An iPhone is not a computer. An Apple Watch is not an iPhone. Times they are a changing… at least for ‘devices’.

It’s your face

The User Interface is also the ‘face’ of a company. It sells the company. Whether to new customers, existing customers, employees, stock holders, or potential investors. A web site like Rumble’s:

• http://rumblegames.com/

And Rumble’s actual games are trying to appeal to a particular audience (middle aged men) and give them something they want (to feel powerful and sophisticated).

A web site like PortaPlanter:

• http://PortaPlanter.com/