Thursday, November 13, 2008

Construx SW Development Executive Summit

I have started working again with Construx Software, helping us improve our software engineering practices. I was of course familiar with Steve McConnell, Construx's CEO and his books on software engineering, and I had worked with Construx at a previous company on training and mentoring our development team and development managers.

Earlier this year I attended their 10x Software Engineering course, an excellent software development management course which focuses on understanding and implementing good practices in software development, how to improve quality and team productivity. We enlisted their help on a project assessment with a partner, and Construx is scheduled to come in later this year to teach a Developer Testing Bootcamp course to the development team.
In October I also attended Construx's Software Development Executive Summit: an intimate, intense and highly-focused series of roundtable sessions and high-quality keynotes with the heads of software development and software engineering at large and small companies across North America, Europe and Asia. Like other Construx offerings, the summit was pragmatic, selective, carefully organized and very professionally run. The keynote speakers included Martin Fowler of ThoughtWorks and Ken Schwaber, the creator of Scrum, as well as Construx's CEO Steve McConnell, author of Code Complete and Rapid Development, and Matt Peloquin of Construx; and interesting case studies presented by IT executives at MSNBC and RIM.

It was a unique forum: a chance to meet and share ideas in an open, constructive and respectful environment with serious people who all had extensive experience leading software development and who were all looking for ways to improve. There were so many different stories: companies who had great success (or disappointing failures) at outsourcing (onshore and offshore); companies who were successful delivering using small, agile, results-oriented collocated teams; other companies who followed highly structured, metrics-driven SDLCs and were equally successful. The development organizations ranged in size from a handful of developers to hundreds or thousands of engineers in multiple countries. The roundtable sessions provided me the opportunity to explore problems and share ideas with experienced development managers, product managers and project managers, and thinkers like Martin Fowler. The social engagements provided excellent networking opportunities and were generally good fun, and there was no pressure from vendors or sponsors.

What key ideas did I take back from the summit?

  1. The first key to success is talent. Get the best people you can. Treat them well and support them, give them what they need. Be careful when hiring, and spend time with new hires, help them to be successful. Keep teams together as long as you can: continuity and cohesion of a team pays dividends.
  2. There is no “one way to skin a cat”: software development poses different problems to different organizations, and there are different answers to these problems. What is important is to execute, and to constantly review and improve.
  3. If you want to show value to your customers, deliver value often, deliver what is important first. It’s all about incremental delivery.
  4. In globally distributed teams, follow-the-sun works for operations, but doesn’t for development. Co-locate teams whenever possible.
  5. Develop technical leadership within your organization. Create a path for talented and ambitious technical people who do not fit or do not want to pursue the management track. Follow the lead of IBM, Microsoft and Google and offer a “distinguished engineer” career path where senior technical people are given respect, latitude and a voice in product direction.
  6. Don’t expect to save costs through outsourcing. Outsource for flexibility, to flex your organization’s delivery capability; and to gain access to talent. To outsource successfully takes a lot of discipline and management attention and supporting costs.
  7. Constantly be aware of, and beware of, technical debt. Don’t bet on “throwing one away” when you build a system. Agile methods without discipline (comprehensive reviews or pair programming, developer testing, …) gets fast results at first, but builds up a lot of technical debt that has to be addressed eventually. If you start with disciplined, good practices from the beginning you won't dig yourself as deep a hole.

This is an event I look forward to attending again in the future and will defintely recommend to my colleagues.

Sunday, November 2, 2008

Software Quality and Software Security

Recently I came across a series of posts by Fortify Software arguing that that “software quality is not software security”. This position doesn’t make sense to me: security, like reliability and performance and usability and maintainability and so on must be taken into account when building a system; and it is the software quality program that makes sure that this will all be done correctly. Most software is not secure because it is not built in a high-quality way: good software is secure, as well as reliable and scalable and fast and maintainable.

Gary McGraw of Cigital, one of the thought leaders in software security, describes how to integrate software security into SDLC. In the book Software Security: Building Security In he introduces a set of touchpoints: key areas in the SDLC where security must be considered in designing and building software. The touchpoints, listed in order of effectiveness, are:

  1. code reviews, including both manual code reviews and automatic checking with static analysis tools, looking for errors in API use, inadequate validation and error handling, proper logging and specific security-related bug-patterns
  2. risk analysis in architecture and design
  3. penetration testing
  4. risk-based security testing
  5. abuse case scenario planning
  6. security included in requirements
  7. secure operations
  8. and external, expert reviews including third party security walkthroughs of architecture, design and code.

In a Computerworld interview Gary McGraw goes on to say that

“Software security relates entirely and completely to quality. You must think about security, reliability, availability, dependability — at the beginning, in the design, architecture, test and coding phases, all through the software life cycle.”

Code reviews, static analysis, risk analysis in architecture and design, risk-based testing, … all of these are necessary in building high-quality software. What is required in addition is building the team’s knowledge of software security: most programmers don’t learn much if anything about software security in school, and need to be taught about the problems and best practices and tools needed to build secure software systems, just as they need to be taught about developer testing, or high availability systems design, handling concurrency issues in massively parallel systems, and the other complex problems that need to be solved in real systems. Secure software requires:

1. Security awareness training and technical training from companies like the SANS Institute, Foundstone and Cigital. First to understand that IT security is not just about hardening infrastructure and secure operations, that a secure system requires building software in a secure way in the first place. Then training in threat modeling and defensive programming to understand the technical problems, the risks, exploits, attack patterns and common vulnerabilities such as the OWASP Top 10, and best practices, tools and other resources available to build secure software.

2. Developing and sustaining “an attack-based” approach to designing and building systems - making sure not only to check that software meets the specifications, but also checking to ensure that “bad things don’t happen”. Architectural risk analysis, abuse cases, exploratory testing, stress testing, war games, fuzzing, gorilla testing and other types of destructive testing all need to be done: not just to ensure security but also the reliability and overall quality of the system. In other words, a negative, attack-based posture should already be followed in testing and reviews, not just for security.

From Gary McGraw again in a recent podcast "How to Start a Secure Software Development Program":

"I guess the biggest difference is thinking about an attacker and pondering the attacker’s perspective while you’re building something”

To build this perspective takes management focus, time, training, mentoring on what to look for in design and code reviews; and continual reinforcement.

Evangelism on the part of software security experts and highly public IT security failures have been effective in raising awareness of the important of software security: that software needs to be designed and coded and tested with security in mind. Software security, making sure that security is considered in every step, is another important part of building good software - it's another part of the job, handled the same as reliability and performance and other hard problems in building big, real systems.

Sunday, August 3, 2008

Safety Net

When I was younger, lighter and more of an adrenaline junkie, I took up rock climbing. It was an excellent way to push myself both mentally and physically, and it forced me to focus absolutely, completely on the moment. A successful day of climbing required thorough preparation, training and conditioning, the right equipment, a good sense of balance and timing (which often eluded me), and smooth team work with my climbing partners. One of the first things I learned was that unless you were a maniac or a supremely gifted super hero, you had to put in protection as you climbed, to ensure your safety and the safety of your climbing partners.

Building real software, software that is good and useful and meant to last, is equally challenging. I am not talking about one-off custom development work or small web sites, but real software products that companies run their businesses on. There is no one best practice or methodology, no perfect language or cool tool that will ensure that you will write good code. Instead, real software development demands discipline and focus and balance, and an intelligent defense-in-depth approach: consistently following good practices and avoiding stupid ones, hiring the best possible people and making sure they are well trained and have good tools, and carefully and conscientiously managing the team and your projects.

Having said this, if you need one place to start, if you had to choose one practice that could make the difference between a chance at success and the almost complete certainty of failure, start with building yourself a safety net: a strong regression test capability that you can run quickly and automatically before you release any new code. Without this safety net, every change, every fix that you make is dangerous.

I am surprised to find software product development shops with large code bases that do not have automated regression testing in place, and rely on black box testers (or their customers!) to find problems. Relying on your test team to manually catch regressions is error-prone - sooner or later a tester will run out of time, make a mistake or miss (or misunderstand) a test - and awfully awfully expensive - it takes too many people too long to test even a moderately complex system. Regression testing is important, but you will get a lot more value from the test team if you free them up to do things like deep exploratory testing, destructive testing, stress testing and system testing, simulations and war games, and reviews and pair testing with the developers.

If you don’t have an automated test safety net today, then start building one tomorrow. Find someone in your team who understands automated unit testing, or bring a consultant in, start writing unit tests for new code and add unit tests as you change code. Run the test suite for each build as part of your continuous integration environment (ok: if you don’t have CIM set up already, you will have to do this too).

Starting from nothing, there’s no point in trying to measure test coverage. So begin by counting the number of tests the team adds each week and track the team’s progress. As bugs are found in production or by the test team, make sure to write tests for the code that needs to be corrected, and spend some time to review whether other tests should be added in the rest of the code base to catch this type of failure. Monitor the trend to ensure that new tests continue to be added, that the team isn't taking a step backwards. Just like building software, take an incremental approach in adopting unit testing.

Once you have a good base of tests in place, use a code coverage tool to identify important areas of code that are not tested. Take a risk-based approach: if a piece of code is not important, don’t bother with writing tests to attain a mandated code coverage % goal. If a critical piece of code or a core business rule is not covered by a test, write some more tests as soon as you can. Then use a mutation tool like jumble or jester to validate the effectiveness of your test suite.

Writing tests that do not materially reduce risk simply adds to the cost of building and maintaining the system. Some fundamentalists will disagree and demand 100% coverage (although with the failure of Agitar the hype on this subject has subsided somewhat) . A recent post on achieving good ROI on Unit Testing explores how much unit testing is really necessary and valuable. Rather than writing unit tests on unimportant code, you could spend that time reviewing design and code, implementing static analysis tools to catch programming errors, helping with functional and integration testing, or, hey why not, designing and writing more code which is the point after all. For most projects, achieving 100% code coverage is not practical or even all that useful. But using good judgment to test high risk areas of the system is.

Test-first or not, but work with developers to write useful unit tests, and try to review the tests to make sure you don’t end up with a dozen tests that check variations on the same boundary condition or something else equally silly. Writing and maintaining tests is expensive, so make each test count.

While a good set of unit tests is extremely valuable, I don’t agree with some XP extremists who believe that once you have a comprehensive set of unit tests in place you are done with testing. Remember defense-in-depth: unit tests are an important part, but only a part, of what is needed to achieve quality. No one type of test or review is thorough enough to catch every error in a big system.

A good unit testing program requires investment in both the short-term and the long-term. Initial investments are needed to create the necessary infrastructure, train developers on the practice and tools, and of course there’s the actual time for the team to write and review the tests, and to integrate the tests into your build environment. In the longer-term, you will need to work with developers to continually reinforce the value of developer testing, especially as new people join the team, and ensure that the discipline of writing good tests is kept up; and the test suite will need to be constantly updated as code is changed or as new bugs are found.

Designing and writing good unit tests isn’t easy, especially if you are following a test-driven development approach. And not only do you have to sell the development team on the value of unit testing; you also need to convince management and your customers to give you the time and resources necessary to do a proper job. But without a solid testing safety net, changing code is like climbing without protection: sooner or later you or your buddy is going to make a mistake, and somebody’s going to get hurt.

Site Meter