Recently I work a lot on Acceptance Tests and how to write them in a clean way. When working with RSpec and Capybara I often see people wanting to go to a page and then do a couple of tests. The intent is to go to the site, login, then describe what they see, which is good practice (at least in theory).

Something like this (which I consider spaghetti coding anyway, but nice for the purpose of illustration):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

describe 'My Page', :type => :acceptance do
  before(:all) do
    visit '/'
    fill_in 'email', :with => 'pzolnierek@gmail.com'
    fill_in 'password', :with => 'passw0rd'
    click_button 'login'
  end

  it 'shows a logout button' do
    page.should have_link 'logout'
  end

  it 'shows a product information box' do
    page.should have_selector '#infoBox'
  end
end

Trouble is that Capybara will by default reset your session after each test. Bummer! If you look at the Capybara source code you will find this:

1
2
3
4
5
6
7
8

...
config.after do
    if self.class.include?(Capybara::DSL)
      Capybara.reset_sessions!
      Capybara.use_default_driver
    end
  end
...

The OO Solution

I am currently working on another more longish article which will be about Beautiful Acceptance Tests where I use an object oriented way to write acceptance tests, but here is a tip from it.

If you want full control do the Capybara wire-up yourself.

First we create a context object, which will hold our session.

1
2
3
4
5

class TestUser
  include RSpec::Matchers
  include Capybara::DSL
  include Capybara::RSpecMatchers
end

Now you can do testing like a Boss!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

describe 'My Page' do
  attr_accessor :user
  before(:all) do
    @user = TestUser.new
    @user.visit '/'
  end

  it 'shows a logout button' do
    user.page.should have_link 'logout'
  end

  it 'shows a product information box' do
    user.page.should have_selector '#infoBox'
  end
end

Notice the missing :type => :acceptance. Here the Capybara::Session will be embedded in the TestUser instance, instead of the Nested class which is created by RSpec and thus will work across tests.

The token authorization solution

Another even faster to implement solution for some people is to have token authorization.

1
2
3
4
5
6
7
8
9
10
11
12
13

describe 'My Page' do
  before(:each) do
    visit '/?token=some-fancy-token-or-guid'
  end

  it 'shows a logout button' do
    page.should have_link 'logout'
  end

  it 'shows a product information box' do
    page.should have_selector '#infoBox'
  end
end

When interviewing new people for a job as an agile developer, what qualities should you be looking for? Is it sufficient to check the previous experience and the knowledge in the areas he will be needing for your projects or is the attitude more important?

Attitude

A developer is like an engineer, always looking for improvement, always trying out new things and trying to make his work more productive, at the same time he is proud of his work and makes sure it works properly. What has made him act this way is his attitude.

Then you have programmers, which simply translate instructions into code.

I like the word developer, as it derives from development, according to the dictionary, to develop means to grow or cause to grow and become more mature, advanced, or elaborate. Hence a developer is somebody who causes some application to grow and become more advanced. A programmer just translates ideas into code. This is not enough.

According to wikipedia, attitude is

An attitude is a hypothetical construct that represents an individual’s degree of like or dislike for something.

Another view of what attitude is comes from buddhism, there we have the Eightfold Path, which is the way to end suffering, i.e. achieve Nirvana. In terms of software to reach Nirvana, is to end all suffering coming from maintaining legacy project.

An excerpt says:

Right effort can be seen as a prerequisite for the other principles of the path. Without effort, which is in itself an act of will, nothing can be achieved, whereas misguided effort distracts the mind from its task, and confusion will be the consequence. Mental energy is the force behind right effort; it can occur in either wholesome or unwholesome states.

For me in our context this means, that somebody needs to be determined to do the right thing and that attitude or the right effort is the prerequisite to become a great developer.

In real life a developer needs to behave accordingly in all kind of daily situations. As our team becomes more cross-functional, not only in terms of specialization on programming languages or special areas of expertise within the area of coding, a developer needs to have the right likes of the following areas:

1. Quality - A developer needs to have the right attitude to automate deployment, so that every time her work is put into production it works flawlessly. Quality means predictability, that said, a developer needs to work on a proper test suite, making sure what was working once, doesn’t get broken accidentally.

2. Customer Value - what we build must be valuable for the customer, developer must avoid creating waste and communicate with the customer directly to make sure to have the same common understanding of value.

3. Continuous Learning - a developer must always learn new things, in his technology, programming language and also in other areas like databases, operating systems etc.

Experience

Then on the other hand there is experience, which is the sum of a person has learned, of all mistakes he did, of everything good or bad, that happened in his life.

If you are building a new team or a new project where you have no expertise yet, you might want to lay more emphasis on experience and hire people, which have already done what you are trying to accomplish.

We all learn by trial and error, especially software engineers need to see for themselves what is working and what is not. Then on the other hand, we cannot allow to make mistakes which jeopardize our customers business.

In order to move forward safely, you have to have people which have made a reasonable amount of errors so they won’t do them on your new mission-critical project.

Conclusion

In my opinion the art (as in martial art) of software development requires people to be truly cross-functional - in their abilities to learn, to take on tasks which are not necessarily their favorite and to continuously steal knowledge from other areas in order to keep improving. Thus, the conclusion for me is straightforward, the right effort or in other words the right attitude, is crucial, specific skill can be acquired along the path.

So in the end, the right answer is - it depends. If you are looking for a long-term investment, and already have a basis, you might lay less importance on experience and more on the right attitude.

If you do not have a solid foundation in the form of experienced people, you really need to build that first. You can build good software with unexperienced people, it will take longer and will cost you more, but with people with a bad attitude you will most probably build bad software, which will be hard to maintain.

Your agile team has built great software, only to find out it cannot work in production?

Agile has taken the development community by storm. It has improved our everyday lives. It enables us to build great working software in all kinds of environments. But for many companies, covering the last mile, bringing an application into production is the biggest obstacle to being truly agile. Prescribed processes and skill-sets in operations lag behind a decade. We have created cross-functional teams, excluding one of the most important aspects of of software - it needs to run in production!

The Problem

Devs become quicker in delivering releases and new customer value.
Ops cannot release as often as Devs.

Devs don’t know or understand production infrastructure.
Devs develop software that works on my machine.

Ops don’t do TDD, don’t automate. Our sysadmins, cannot write code (messy bash scripts is about the max).
Ops needs days or even weeks to deploy something to production, because of their own queuing.

Separating Dev and Ops creates a waterfall. If anything breaks, the feedback cycle is long!

Conflict of interests

Devs and Ops have been created in most organizations to fulfill a separate role. Devs are designed for change. Ops are designed for preservation. As the organization matures, each department evolves into it’s own direction and have their own goals, sometimes forgetting they both serve the same purpose.

Product Owners want new value asap - on production - that’s understood.
Developers want to release more often.
Ops want high availability, stability, reliability.

Both Devs and Ops, from their perspective, want the best for the business. Only, optimizing any subsystem without context, leads to the destabilization of the system as a whole, de facto decreasing overall performance.

Deployment is complex, high risk, error prone and thus holds a risk of destabilization.

Anti-Patterns

The reasons why Ops departments were created - at least in our case - because there were some anti-patterns which deteriorated our quality and our reputation with customers.

• AdHoc Release - Devs would deploy as soon as they made quick fixes to the application.
• Production Patching - Sysadmins change configs and other parts directly on production, nobody knows what they changed
• Service Monolith - the apps are not modular enough or not modular the way and nobody understood how they work together.
• Lack of Automation - the whole deploy process was done manually, every time!

What we need to focus on

It is not about bringing agile practices to operations, this is a must anyway. Although Kanban is ideal for that kind of endeavor, we do not want only to copy-paste our practices to another department.

We need to focus on the right effort. We are all connected by one purpose, we all need to take have the right mindset to work together on a product as a whole to provide a tremendous customer service.

Thus we need all to keep in mind what is most important:

1. Business Value
2. Predictability, Repeatable processes
3. Stability, Uptime

About DevOps

The Devops movement has done a good job to bring out, many of the pain-points of the current sysadmin vs developer silo situation. I have been hoping, that admins and ops will someday understand, that they must do more than install software, hardware and search logs.

I kind of like the word Devops.
To me it means that sysadmins will be more like developers, adhere to the high standards of developers which have evolved over the last years with xp and agile. It means to me that automated installation will bring us an repeatable process. No repeating mistakes. Identical production servers. No exceptions. Setting up new machines will be easy fast thing as bringing new functionality to the software.
Current Sysadmins must understand, they are like 32bit hardware, to be retired in the near future, unless they adapt and upgrade themselves!

On the other hand, developers must fully acknowledge, that their software is only if it works in production. The Definition of Done has to be extended to “Running of Production”. Every sprint.

Duties

Let’s have a look at the responsibilities of our two parties. Despite that Devops want to include QA into their new movement, I will argue that QA is a duty of both Devs and Ops anyway and thus not talk further about it (either it works or it doesn’t).

Let’s have a look at the duties of the different roles in an organization like mine - anixe - where we develop software for e-commerce to run our customers businesses.

Duties of Sysadmins:

• Systems & Applications Administration
• Infrastructure Management

Duties of Ops:

• 24×7 Tech Support
• Deployment
• Testing & Quality Assurance (QA)

Duties of Dev:

• Builds & Configuration
• Release Management
• Testing & Quality Assurance (QA)
• Technical Analysis
• Technical Project Management

What’s wrong with this picture?

Sysadmins are responsible for Applications - how could they, if they haven’t built them (just assume changes happen fast)
Ops are responsible for deployment - cool, but if it doesn’t work, they’re helpless without Devs assistance
Ops are responsible for Testing - ouch - again, they need to be taught what to test and testing and building is not happening in the same iteration
Devs don’t know what’s going on on production

Possible Solution for a harmonious together

This is not just one possible solution, there are others as well, but here is a temporary one we might want to try. In this context, Ops are sysadmins, the rest of Ops should be part of the agile development teams anyway. I just don’t see how we could integrate admins, which are specialist and only required “rarely” once a platform is up and running.

Ops provide a cloud-like environment. Devs deploy to a transparent cloud-like environment. Devs are responsible for deployment to production!

Cloud-like means, every machine deployed to must be same-same, no matter whether it’s test, staging or production.

Ops use an automated, repeatable process for system installation. This means coding skills are required by every team member. The same clean code, versioning and other practices must be by every team member.

Continuous integration, build, deployment.

Requirements:

• all environments are guaranteed to be identical
• Devs have reasonable access to the infrastructure
• Personal User data must be protected and inaccessible

How far have we come?

Here at anixe, our current workflow looks as follows.

• Devs develop software, it is unit and integration tested
• One some projects: Admins deploy prebuilt packages including the configuration for the target environment. NO manual editing of config files is allowed, every config for every environment is under source control. All applications config files are the same. Differences per platform are in a separate configuration file. Database schema changes are bundled in the deploy and run automatically. We haven’t figured out how to do automatic data migrations, yet.
• On other projects: Ops run an automated deploy script (home-made for .net applications), which takes care of everything, handles backups, rollbacks, etc.
• On ruby projects: We use currently capistrano and Devs deploy themselves. We use here mongo as database, so deployment is easy.
• Ops verify if everything went smoothly and notify customers about the outcome
• For linux deployments are preparing an installer using babushka, which does test-driven-sysadmin installation.

Generally linux deployment is so much easier, a fully automated and test driven platform installation within a couple of days. The same on Windows took us over a month.

After the separation of Devs and Ops we have developed a new anti-pattern which is over the wall deployment. Devs deliver packages and let Ops do the rest.

Now as we are approaching a new age of automation of deployment, we currently face another anti-pattern, and that is that everybody in the organization wants to use their own set of tools. Some use dos-batch programs, some use VB scripts, some bash scripts and so on.

References

Agile IT: A Better Approach to Application Development, Deployment, and Management
I Don’t Want DevOps. I Want NoOps.
What is DevOps? The Rise of DevOps
Organizing a Web Technology Department What DevOps means to me…
What Is This Devops Thing, Anyway?
Agile isn’t just an application development method
AGILE MANIFESTO CO-AUTHOR ENCOURAGES AGILE OPERATIONS
Deployment management design patterns for DevOps Wikipedia on Devops