Yesterday was an interesting day to say the least. I was in one of our weekly data team meetings when the topic of our release notes came up. Needless to say a very heated discussion arose from it.
Apparently after Monday afternoon’s database change roll out occurred there were 2 stored procedures that broke due to a column being renamed to match the proper naming convention of the project. The developers got upset at one of the DBAs who in turn got upset at us. He came to me to discuss his concerns and figure out what happened.
Here’s the timeline:
- Last Friday AM: Review of all code to be deployed against the current developer database for compliance with established guidelines.
- Last Friday PM: Send email to everyone that touches the database with a list of the changes with notice to read the changes, check and fix your code and be ready for the changes to be applied Monday afternoon
- Weekend: Time to relax and not worry about work, amazingly no contact from anyone concerning upcoming changes
- Monday AM: Send reminder email of changes to be made that afternoon. Remind everyone to double-check code against the full detailed list of changes to be made to the database.
- Monday PM:After hearing nothing from anyone, changes applied and notice sent to everyone.
- Wednesday morning: DBA comes over all upset since some of his procedures broke and the application in development wasn’t working anymore.
I sat there,looked at him and asked one very simple question:
Me: Did you read through the high priority email notice send out both on Friday and Monday?
DBA: No, I never read it
Me: Then you have to deal with it. We send notice and everyone has a responsibility to read it. There is a reason the very first part says: “These changes may affect code. Please review the changes and verify your code is ready for the release”
DBA: “Isn’t it our responsibility to check the procedures”
DBA: Why didn’t we?
Me: Code review committee reviews the syntax and compliance, we don’t check the functionality of the SQL and Java developers. We made that clear from day 1. You are part of the development DBA team, why did you not check the procedures you created for them that reference the tables being changes?
Thursday morning was a scheduled data team meeting and this topic came up. Needless to say there were many opinions on what level of service to provide to the application developers, but it was decided that we will continue provide advanced notice and will include a list of all procedures that may be affected. Beyond that the responsibility will be on those being notified to maintain their code. The data team is not going to do everyone else’s work. The scariest part for me as a team lead was learning that only 25% of the data team actually read the weekly change notice. Of course, those making the most noise were ones that did not read the emails. I was very good and held my tongue quite a bit in the meeting, but I’m pretty sure the emotion behind my thoughts was relayed appropriately. I am hoping this doesn’t continue to be an issue as this project progresses and ramps up even more.
I swear there are times that I feel I am the only one that has experienced things like this.
If you have dealt with a similar issue, how did you resolve it?