Agree overall. However, 'data and AI' seems to be generalising a lot here. Take the case of a BI team developing dashboards over a DWH. If the DWH is mature, is well modelled and has good data pipelines, and the BI team have worked with the DWH for some time and have developed other dashboards before, then there should be few unknowns in the dashboard development process. It should be able to be handled in an agile type fashion. In fact dashboards can benefit from some iteration because it is hard for users to know just what they want until they start to see something.
Thank you so much for your comment James. I completely agree with you. What you are describing is an environment with a well established collaboration and communication in a mature enough team. Those are key elements to crafting impactful Data & AI Products.
I actually think you could re=write your article substituting software development for data and AI development and get a similar reaction from many in the agile software community. I think you even say at one point “the problem isn’t with agile … it’s how it’s applied”.
Forgive me if I tell a story. I was working with a customer’s data team on getting some new complex dashboards built and discussed with the CTO/CDO the need to slice the work finer in order to generate faster feedback. Their software teams were familiar with Alistair Cockburn’s elephant carpaccio workshop and I had been thinking of a version of this for data teams. I was convinced you could not do the “pure” software dev version. Long story short we ended up doing the “pure” software dev version and the data team agreed this helped them and they could adapt the practices for their circumstances. I was convinced that agile software development techniques needed to be adapted for data engineering but was proved wrong.
Yes, they need to be adapted for data engineering and data and AI products but the underlying principles of agile actually work well.
Thank you so much Martin for this very insightful comment! It is very true that a big part of what we wrote can be transposed into Software environment. In fact, I believe Data & AI is in some ways following the path that Software Engineering took a few years ago, so we should learn from that.
I love your example that illustrates that yes, with the right level of communication and agreement on the problem to solve, agile software development techniques can be a very good fit for Data initiatives and products.
Agree overall. However, 'data and AI' seems to be generalising a lot here. Take the case of a BI team developing dashboards over a DWH. If the DWH is mature, is well modelled and has good data pipelines, and the BI team have worked with the DWH for some time and have developed other dashboards before, then there should be few unknowns in the dashboard development process. It should be able to be handled in an agile type fashion. In fact dashboards can benefit from some iteration because it is hard for users to know just what they want until they start to see something.
Thank you so much for your comment James. I completely agree with you. What you are describing is an environment with a well established collaboration and communication in a mature enough team. Those are key elements to crafting impactful Data & AI Products.
I actually think you could re=write your article substituting software development for data and AI development and get a similar reaction from many in the agile software community. I think you even say at one point “the problem isn’t with agile … it’s how it’s applied”.
Forgive me if I tell a story. I was working with a customer’s data team on getting some new complex dashboards built and discussed with the CTO/CDO the need to slice the work finer in order to generate faster feedback. Their software teams were familiar with Alistair Cockburn’s elephant carpaccio workshop and I had been thinking of a version of this for data teams. I was convinced you could not do the “pure” software dev version. Long story short we ended up doing the “pure” software dev version and the data team agreed this helped them and they could adapt the practices for their circumstances. I was convinced that agile software development techniques needed to be adapted for data engineering but was proved wrong.
Yes, they need to be adapted for data engineering and data and AI products but the underlying principles of agile actually work well.
Thank you so much Martin for this very insightful comment! It is very true that a big part of what we wrote can be transposed into Software environment. In fact, I believe Data & AI is in some ways following the path that Software Engineering took a few years ago, so we should learn from that.
I love your example that illustrates that yes, with the right level of communication and agreement on the problem to solve, agile software development techniques can be a very good fit for Data initiatives and products.
Does this help? https://www.linkedin.com/posts/jhighsmith_the-agile-manifesto-has-often-been-misinterpreted-ugcPost-7255291429898366979-kauI