Agile Testing Days inspiration – Holistic Testing with Lisa Crispin

Agile Testing Days warmed my heart and soul, and energized my brain. I could write a small book about it, so much learning and inspiration was packed in. (I wanted to write a short post, but again, I have failed at that goal!)

Janet Gregory and I already shared some of our takeaways from the conference in our latest Donkeys & Dragons video chat. Please check that out, I’ll try to not repeat myself here. I’ve been trying to focus on – “what actions have I taken myself thanks to an idea or nugget of wisdom acquired at ATD?” Only a few so far, but lots of plans for next year!

I arrived at the conference hotel a few days before the conference started. The welcome banner tugged on my heartstrings. Potsdam and ATD really are a home from home for me! It was weird being there without Janet, as we had teamed up at ATD for 15 years. We all missed having her there. Doing a tutorial on holistic testing together with Ashley Hunsberger was the perfect way to start the week. Ashley and I also hosted Lean Coffees together each morning.

And now – just a few things that have fueled my engines since ATD.

GenAI and LLMs are here to stay

The hype train keeps speeding along, and the capabilities of LLMs and GenAI tools seem to increase exponentially day by day. I spent a lot of time in 2023 and early 2024 learning about them. Since I’m not working as a hands-on tester in a team now, I don’t have an opportunity to use them in earnest. It’s clear that I need to stay informed, at least.

I had wanted to attend the workshop “Guiding Hands-off AI using Hands-on TDD”  that James Lyndsay and Bart Knaack facilitated at ATD. Unfortunately, I had a scheduling conflict. I had volunteered to coach a few teams who were attending ATD together, and I had a session at the same time as the workshop. (I enjoyed working with the team and hope they found value in our session too!)

Fortunately, James offered a similar online workshop, “AI vs TDD in 90 minutes: A workshop to explore LLM code generation based on tests”, a few weeks later. I was able to join. It was fascinating to send a test to an LLM and have it write the code to make the test pass. Sounds magical, right? Well, like many things one explores as a tester, there were good things and bad things. Running the scripts individually on our own machines, we often got different results form each other. As we added more tests, results often got more strange.

The workshop reinforced my limited experience with using “AI assistants” and tools like Gemeni and ChatGPT. If you already have the skill set for what you are doing, the GenAI tools and LLMs can help you do a better job and possibly save you time. If you don’t have the knowledge already, you could get into trouble.

My takeaway: Everyone working in software needs GenAI / LLM tools in their toolbox, and we need to know how to use them safely, ethically and effectively.  I will continue to take advantage of opportunities to learn how to use these tools together with others in our awesome software community. And – do not try to do test-driven development by using an LLM to generate the code to make the tests pass!

Who am I, professionally?

The keynotes from Jenna Charlton and Ashley Hunsberger gave me so much food for thought, I’m still processing it all. I’ve tried to separate my personal identity out from my professional identity over the past few years. I transitioned from being a full-time contributor on a team (for 40 years) to becoming a freelance consultant right before the tech economy imploded. I’m someone who loves to feel appreciated by teammates and customers. Working solo, and not getting many consulting gigs, was a blow to my self-image.

I felt pretty low after the conference, to be honest. Not working full time has made me feel less relevant. Most teams I was on the past few decades worked on the leading edge of technology. We learned the newest good practices and chose the frameworks and tools that best fit our context. Having enjoyed working in pairs and ensembles so much for the past 25 years, I find I’m not too good at learning on my own. Which is why I value opportunities like James’ workshops. I feel like I’m not relevant anymore, people aren’t looking to me for the latest ideas.

At the same time, I’ve gotten a lot of positive feedback lately on the books Janet and I have written together – which include contributions from many others in our community. Even our first book remains relevant and useful. That makes me feel great. And I wonder, how can I continue to contribute? I have a couple of topics in mind that I want to write about and get people talking more about, so that is good motivation.

This small existential crisis has led me to value our community more. We learn better when we learn together. I’m lucky to have time to listen to others sharing their experiences. I can share what I learn from them. I accept that I can’t keep up with the leading edge of every aspect of technology. And, I can learn more about the topics that interest me most, and that let me help others.

Continuous improvement – step by step

One of my ATD highlights was pairing with Rafael Amaral to do a workshop on using metrics to guide continuous improvement. Rafael had the idea for this topic, and we worked all year to put together the materials and exercises. We decided to use the Google DORA key metrics as the basis for the session. These are based on 10+ years of research by DORA, and in my experience, they are helpful across a wide range of contexts. They’re a good place for people to start understanding how to use metrics to track improvement efforts.

As part of the workshop, we introduced the DORA Core model,. We emphasized that the capabilities, such as climate for learning and fast feedback, and the outcomes of organizational performance and well-being are more important than the metrics themselves. Quality attributes were another point of discussion. Teams need to prioritize. For example, do customers value fast response time, security, or accessibility most? The DORA key metrics are a good way to measure process quality, as Janet Gregory and I discussed in a recent blog post. You need good process quality to achieve product quality goals.

Participants worked on exercises in groups of four, discussing the problems faced by the fictitious (but based on real teams) team in our case study. In the final exercise, each group designed an experiment to address one of those problems. They biggest challenges for the participants was to find a good way to measure progress of the experiment, and to figure out an appropriate timeframe to get feedback quickly enough. This is a big challenge for teams in real life as well! Given the amount of time they had to work on the experiments, they did a great job.

At the end of the two-hour workshop, we got positive feedback from participants. Several suggested expanding it into a full-day workshop. That would definitely work better. It was hard to condense a useful amount of practice and learning with metrics into a short workshop. Hopefully we’ll get a chance to do that!

The warm feeling lingers on

As I write this towards the end of 2024 (it’s been hard to find time to work on this post!), I still have joy in my heart from visiting my home away from home,

hanging out with old friends, and making new ones. This helps me subdue my inner critic. I feel energized for 2025! I plan to write more, and I already have been gathering thoughts for a bit of a rant in my next post!