Note: This is an old post and may or may not accurately represent my current views, current technology, etc.
It seems like most startups and even many larger companies have adopted some form of agile software development with scrum being very prominent. Although the degree to which “true” scrum is properly followed varies, many younger software engineers haven’t ever had to experience heavyweight methodologies such as waterfall.
The waterfall methodology puts heavy emphasis on process and rigidity. This requires significant up-front planning, which requires an extensive specification. Since there is so much effort to plan out each step of the process in the beginning, waterfall causes a natural pushback to changes. This also causes difficulty with testing because you are typically aiming for developing the final product that is delivered, which is months or years down the road, so the product stays in an unshippable state for most of that time.
Agile methodologies such as scrum are focused on much shorter feedback cycles. With scrum, you plan sprints that are relatively short (I’ve seen one-week, two-week, and four-week sprints, but there’s no set requirement for length). Each sprint has a planning session, the days of work, and a retrospective with the idea being at the end of each sprint you have something that you can ship (you don’t have to actually ship it, but the product builds, it’s usable and tested). Since you’re not planning out details months in advance, a significant change can come in and be embraced in the next sprint.
People who have always used the waterfall process have a hard time adapting to scrum. Without planning out for months or years, they feel like things are chaotic. How can you promise what you’ll deliver if it’s not heavily planned ahead of time? The funny thing is that scrum actually creates a measured velocity so that you become better at predicting what you’ll be able to do in a given sprint. Although you aren’t planning months in advance, the planning you are doing is actually directly informed from what you’ve done before and gets better every time.
One of the big benefits of scrum is that it works so well with user centered design. You might be developing a camera app and user feedback seems to suggest that people really like the concepts of a few filters you’ve described. You develop some prototypes that demonstrate the filters without worrying about making them perfect or efficient, so you can start having some users try them out. What you realize is that they didn’t care as much about the filter as they did about how it affected their appearance in the photo. What they were really looking for was something that removed blemishes (wrinkles, acne) and softened skin. Instead of wasting time making the filters faster, you can shift and focus on automatically removing blemishes in that next sprint. That lets you both satisfy users (which increases the chance of your product succeeding) and satisfy the team because they’ll actually know that they developed the right thing (and additional analytics can further reinforce that).
Scrum allows you to start work even if the product definition isn’t stable. It prevents unclear requirements from preventing the start of a project. It lets you adapt to challenges with many unknowns (e.g., machine learning). And it even lets you easily handle changes to the number of people working on a project. All of these are issues that waterfall struggles with.