“In theory there’s no difference between theory and practice; in practice, there is” – Yogi Berra
Agile and its manifesto – In theory
If Berra’s maxim applies to programming it presents a problem for many modern development operations. In my experience, most companies label their development method as ‘Agile’. Adherence to this method usually involves daily short meetings and some software to make sure that small, discrete tasks and problems are tracked. Here is the Agile Manifesto (link):
- Individuals and Interactions more than processes and tools
- Working Software more than comprehensive documentation
- Customer Collaboration more than contract negotiation
- Responding to Change more than following a plan
The original creators of Agile expanded on their manifesto, saying (link):
We follow these principles:
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
- Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
- Business people and developers must work together daily throughout the project.
- Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
- Working software is the primary measure of progress.
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity – the art of maximizing the amount of work not done – is essential.
- The best architectures, requirements, and designs emerge from self-organizing teams. [emphasis added]
- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
So, your project is Agile? Then it must “welcome changing requirements”, right? Well, this is your lucky day! Let me tell you about the new GDPR privacy requirements. (I’d like to be at that meeting.)
Sounds great, doesn’t it? Working together, customer satisfaction, trust, business and technical working together, face-to-face conversations all around. It can work if the underlying assumptions are met: highly motivated and qualified staff, a long-term team at the same location, the best tools, technical excellence, good design that just happens naturally. I’ve seen such projects work well, but I’m not sure how suited they would have been with GDPR compliance added to the mix.
‘Agile’ in practice
More to the point, projects that will meet the underlying assumptions of Agile in the age of hyper-globalized cost-cutting and outsourcing are likely to be rare. Your project cannot be good, fast, and cheap at the same time, which is the implicit promise of Agile-lite. If your organization insists on fast and cheap (as most do), then it shouldn’t be surprised when good suffers (most are surprised anyway).
From what I’ve seen, the Agile label is more a way to start programming without thought-out (let alone signed off) requirements, designs, models, test plans, or anything else that does not appear on the screen. New arrivals on the development team (or maintainers who come after the developers have finished) usually go through an apprenticeship to orient themselves. After all, with no written material, there’s no other way.
I can state from experience that, on many projects calling themselves Agile
- developers are forbidden to contact business stakeholders directly; communication is done through an intermediary (usually called a business analyst) who has neither business nor technical expertise
- working software is delivered in the early phases, but delays become longer as upstream shortcuts (collectively known as ‘technical debt’) multiply and the web of cross-dependencies gets restricts one’s options
- environments and tools are chosen with an eye to minimizing costs
- new hires are expected to find and install their own tools and arrange their own accesses to systems, a process usually taking weeks
- face-to-face conversations become fewer as everything that can be outsourced is sent off-site
- anything that can be stuck into the cloud goes there
- ‘design’ never moves beyond vague pictures of boxes and arrows. (We don’t have time for that; this is Agile!)
- the unified, long-term team that the Agile Manifesto implicitly assumes never exists, as a revolving door of employees, consultancy hires, and freelancers constantly joins and leaves the project throughout its lifecycle
One point of this long list is to underline the disconnect between in-house software development as it is practiced and the kind of development that will be needed to comply with the GDPR.
Impact-assessing a project
If you’re an analyst and want to assess GDPR’s impact on a software project, ask to see their requirements, design, and modeling documents. If such documents exist and are not embarrassing (say, requirements consisting entirely of slideshows, or hundreds of documents scattered on unsearchable repositories), then you’re already above average. If the documents are fairly up-to-date and complete, then you’ve got a winner. (This still leaves the question of how much the project adheres to its plans, but if the project documentation is in good order, it is likely that the rest of the project is in good order as well.)
If you’re a developer and want to assess a software project, ask to see some code. Does it have comments, appear to conform to a standard (e.g., naming conventions, error handling)? Ask the staff how they do code review, version control, or regression testing. There should be good records of all of these quality measures. Their absence lets you know that, however well the operates on the screen, the risks of major revisions (such as retro-fitting GDPR compliance) will be high.
More to the point here: How will you demonstrate privacy-by-design in the absence of such documentation every step of the way? How does GDPR compliance happen with Agile, even if the method is conscientiously applied? After all, privacy-by-design implies that you have a design and that you follow it according to a plan. If your approach is concerned with “responding to Change more than following a plan”, how does that work? Is it possible to plan out the GDPR part, but stay Agile-unstructured for the rest?
I doubt that Agile, as presently practiced, will produce good compliance. After all, in order to plan your privacy measures you need to know what personal data you’re processing, what you’re doing with it, why you’re doing it, and how you intend to protect it. (Can “privacy by design” mean anything other than this?)
But how can you know what data you will handle without requirements, design, functional and data models, and all the trappings of the pre-Agile era? If you don’t have test plans to ensure that you’re really protecting the data in the way that you specified in the upstream planning, then how can you be sure that something has not slipped through? What will you show to the auditor from the Privacy Authority?
If you do know what data you’ll be processing, plus the how and why, you’re already half-way to having old-fashioned, formal requirements. By the same logic, you need to specify how you’re respecting GDPR in all the downstream phases of development as well.
Back to the future again?
I hate to break it to the Agile crowd, but it’s hard to see how let’s-just-trust-each-other-to-know-what-to-do-and-do-it-right is not likely to pass muster as “privacy by design”. GDPR is, in effect, a new set of requirements not subject to on-the-fly modification
During the 1990s, when I was a student in computer science, the development-methods course was called ‘software engineering’, reflecting the idea that building software is like building houses, airplanes, electric devices and so forth. (It isn’t, but that’s another post.) The method I was taught was CMM, the Capability Maturity Model, which was oriented toward producing high-quality software (which meant: has the fewest defects), by cultivating and improving the skills of a stable development team.
Stable teams went away after the 1990s, but the belief that looking before you leap is a good idea still informs high-stakes software practice. (For better or worse, GDPR raises the stakes on development.) My argument here is that “privacy by design”, purge-on-demand, and user-data-portability (among others) are more likely to be achievable with an engineering approach than an improvisational approach.
In one of my favorite development books, Steve McConnell (author of the many books, including the famous Code Complete, and also project manager for the first version of Microsoft C++) sums up his own manifesto as meaning:committing all requirements in writing
- using a systematic procedure to control additions and changes to the software’s requirements
- conducting systematic technical reviews of all requirements, designs, and source code
- developing a systematic Quality Assurance Plan in the very early stages of a project that includes a test plan, review plan, and defect tracking plan
- creating an implementation plan that defines the order in which the software’s functional requirements will be developed and integrated
- using automated source code control
- revising cost and schedule estimates as each major milestone is achieved. Milestones include the completion of requirements analysis, architecture, and detailed design as well as the completion of each implementation stage
Steve McConnell, Software Project Survival Guide (1997), p. 20
Which approach do you think is more likely to comply successfully with GDPR?
The GDPR is not Agile-friendly
There are many assumptions in the expanded Agile Manifesto, but number 11, which assumes that requirements emerge from self-organizing teams, implies that requirements can be tailored along with other project variables. The problem here is that GDPR cannot emerge; it already exists as a non-negotiable set of requirements.
John Adams, the second president of the United States, famously defined a constitutional republic as “a government of laws and not of men”. The Agile Method, even when practiced as intended, envisions a project of people, not processes (rules). The GDPR, unfortunately for Agile partisans, consists of rules.
What do we, as GDPR practitioners, do about it?
It is one thing to diagnose a problem, quite another to prescribe an effective remedy. I have no magic bullet, but will in future posts propose specific methods that should be more effective than simply carrying on as we always have, while allowing a measure of Agile-like flexibility within an
A GDPR-compliance oriented shadow team
The best analogy I can think of is the person on an Agile team who is called the toolsmith, or something similar. This person doesn’t design or develop, but rather develops and maintains essential background functions like version control, backup, build and deployment scripts, that is, the technical framework. In other words, he or she creates and maintains a framework for success, such that the developers can do what they do best without having to worry about losing time or code to an error or omission.
The toolsmith can handle the technical side, but his or her counterparts are needed on the business and functional sides as well. The business privacy person would be a part of requirements drafting and could challenge the business stakeholders on the business case for storing individual items of personal data. After all, you may eventually be challenged by the DPA, plus not keeping data is the easiest way to avoid exposing it.
After the requirements establish what data is needed, the functional analyst can perform a similar role, but instead of deciding whether the data needs to be processed by the envisioned application, considering how best to do it. For one thing, the functional privacy person will review and sign off on design-level artifacts, making sure that all standard privacy measures have been implemented (for example, by using checklists, which themselves will be stored as part of the privacy documentation).
When given the inventory of data to store, the functional analyst is charged with finding out whether it is already stored elsewhere in the organization and if so re-using it (the data inventory will be useful here). This avoids the unnecessary maintenance of multiple independent copies of the same data, which is a nightmare not only for GDPR but also for traditional goals like data quality and consistency.
What I imagine is a GDPR team that shadows active projects and creates a compliance framework, maintaining tools, monitoring practices, auditing artefacts, and generally making sure that compliance is built into the project. After all, it seems both distracting and unproductive simply to dump GDPR responsibility onto developers. A single compliance team can not only be specialized and effective; it can also ensure a consistent, organization-wide approach to compliance.
GDPR-compliant Agile development could be analogous to a student progressing through a course of study. The student is free to study and complete assignments in his or her own way, but must pass an exam at the end of the course. In the same way, individual project stages would have to pass a GDPR exam before progressing to the next stage. Integration testing and project promotion (moving code from development to test to production) would have their own exams. Of course, these exams would be documented as retained as part of the organization’s records of methodical compliance.
In future posts I plan to set out some specific ideas for elements of a compliance framework.