Gamifying the DBA Experience

All work and no play makes us dull DBAs.

Being a DBA can be a really thankless job. From Jr. Shutdown Immediaters to Sr. Rockstar Oracle DBA ACE OCM PMP LOL A+ BBQs, we’ve all trudged into work at some point in our careers only to find that we have three impossible tasks, four humiliating tasks, and no less than seven “why is the database causing this query to be slow we haven’t changed it in two years” tasks. Yet we keep on doing it, day after day, for love of the data. Or something like that.

I wrote an article a while back about galvanizing the DBA team into action and making them stand out a bit more in the corporate world. In the end, a lot of that boils down to making sure the DBA team looks less like a cranky support group that constantly needs disk space and expensive Oracle licenses and more like a team that wants to work as a key part of bringing new products to market (and thus earning money). But even then, it can be a tedious, mind numbing task at times.

So how can we spice up the DBA experience and the IT life in general?

Gamification, of course

Everybody loves games. More to the point, people like being competitive and reveling in a sense of accomplishment and achievement. The idea behind gamification is that many things in the day-to-day non-gamified world can have principles of game theory attached to them. For the most part, the current trend of gamification is a corporate strategy that can be used to further engage a customer base (and is predicted to be a $1.6bn market by 2016 as a result). But even from a departmental or team point of view, it can have great benefits to increase morale, productivity, and accomplishment. And with a little luck and project management, it can be used to solve critical issues for your business.

What gamification is:

• Making tasks more like missions. Everything from tuning a query to backing up a database can be a mission. Instead of becoming yet another ‘requirement’, tasks can become goals.
• Providing a system of Achievements, Badges, Rewards, etc. Modern games are nothing if they don’t have some sort of achievement system that allows players to share their accomplishments across their social network or game community. Within a business, that community can be the IT group (like the DBA team), the department as a whole, or even business-wide.
• Competition! Pitting people against each other in a fun, non-threatening way is a great way to improve morale and encourage growth of skills outside of the standard “do this and do it well or you won’t get a raise” paradigm.
• Leaderboards, which are good for tracking the competition and achievements but are also useful for status and review purposes. What’s the point of earning achievements if they aren’t tracked, or competing without scores?

What gamification is not:

• Strapping your DBA team into slingshots and giving them an achievement for knocking over the most developers.
• An IT version of Thunderdome where two DBAs enter, and one DBA leaves.

Let the Games Begin

So it sounds great in theory, but what can we possibly do to bring it into our DBA practices? As with all answers regarding database management, it depends. If you’re a consulting organization, gamifying is simple because you generally are performing work and solving problems for a wide array of clients. You track team members’ status and customer satisfaction anyways because it is part of the business model. For a DBA team inside an IT department at a major corporation, it can become a little more difficult. So let’s look into some of the ways you can make games part of your actual work.

Backup and Restore Challenges

Setting up and monitoring a good backup and disaster recovery solution can be a serious pain in the neck. But everyone would agree it is absolutely vital to your company and your job security. But even worse than setting it up and monitoring it is having to actually put those backups to use in the case of a critical failure. Whether the problem is a single corrupt data block or an entire database ground to dust, having to actually perform a restore when the heat is on and the managers are circling and all your plans for the weekend are being piled up in the “it’s the thought that counts” bin like so much garbage… well, it sucks. There, I said it.

Maybe that (with a hint of staffing problems and financial woes) is why companies are failing to adequately test their DR plans. That’s a highly dangerous practice, right there with giving developers DBA access to production and typing drop database commands and saying just kidding while you fumble for the backspace key. What is the point of organizing and configuring a complex backup solution if you only plan on testing it when disaster strikes? I can’t count the number of times I’ve worked with clients who said “oh, we needed that file?” or “well, the logs all looked okay” as their hopes for database recovery dropped like their company’s stock price after a three day outage.

So test your backup and recovery plan. But you can make it more fun by running simultaneous tests or presenting interesting and sometimes horrifying cases to the team when they are asked to perform restores. In this article I wrote about the Backup/Restore Workshop day when I worked for Oracle University, and how one student took his preparations to a whole new level, ultimately giving me the old what for, so to speak. Make a competition out of it. Challenge your DBAs to recover from sticky situations and to do it quickly and efficiently. Engage other teams too (such as application server administrators), who ultimately should be part of backup and recovery testing. Doing so invites the teams to improve their speed while also giving the business a good idea of where there may be bottlenecks.

Questionable Query Quality

Not only is this topic worth 192 points in a triple word score in Words with Friends, it’s also a serious issue that almost every DBA/Development group has to face. Writing queries can be a very tedious task to the point where many groups skip that whole tuning thing and go straight for production, guns blazing. This is, of course, a bad idea.

At one company I worked for we came up with a really interesting, fun, and productive way of tackling this issue: we pitted the query developers against each other in battle. Developers would requested to put a comment in their SQL code with their name so we could 1) track who wrote it, and 2) hold contests periodically. So when someone did “SELECT /* Steve Karam */ count(*) from RIDUCLOUSLYHUGETABLE;” we knew who it was and where to go to get it fixed.

That can make for some seriously awesome competition. Each month, quarter, release, etc. you can host ‘award ceremonies’ honoring the best and worst of the time window. Grab explain plans for all the queries and come up with valuable metrics to track like:

• Most readable
• Best use of indexes
• Least blocks touched
• Most convoluted query
• Least likely to make it to production
• Most likely to result in concurrency waits

It’s all in good fun. And it makes query reviews almost bearable. In fact, I’ve noticed in cases where I used this (whether in real environments or in training sessions) the participants were not embarrassed or angry about writing a bad query. They wanted to figure out how to make it better so they could be in a different acknowledgement tier the next time around.

Professional Growth Challenge

While there are a ton of competition ideas within the DBA team (best use of indexes, most SQL profiles used, fastest mouseslinger in the Grid) there is also that of professional development. How about a competition on who can get their OCP certification first? Or one of the many Oracle Expert certifications out there? Heck, even a contest to see who can rack up the most obscure or highest count of certifications from anywhere in the industry would be fun and potentially enlightening. Not only will the team benefit from the certifications and the process to get them, but it makes the team look better as a whole when everyone is clamoring to get professionally acknowledged the fastest.

If you have a really top notch team, then challenge them to get their OCM. While this will require some company backing to keep it fair since the OCM is an expensive and time consuming test to take, it definitely has its rewards. It is a great way to encourage professional growth within your team using pre-built metrics.

D&D Tabletop Drills

My current company had a pretty cool idea as the result of a stability and process review initiative. We started doing tabletop drills once per quarter. For these drills we assembled a team of moderators (usually one manager from each IT group) that came up with a couple scenarios of problems that can occur in our production systems. A ‘player’ is chosen from each team to act as representative: one DBA, one applications administrator, one system administrator, etc. And we would get everyone in a room, surrounded by observers, and act it out.

The idea is to act out the response to the crisis exactly as we would if it were really happening. Sometimes input is needed; for instance, if they discover that this scenario includes application servers being down, they can ask how many are down. Moderators answer questions and keep track of the progress to gauge how the activity goes as a whole, whether SLAs were acknowledged and met, and what can be improved. Following the activity, observers, moderators, and participants have a discussion about the scenarios as a whole and what action items should occur as a result of the findings.

I like it because it’s like the IT version of Dungeons and Dragons. We’ve got game masters and a group going through a perilous journey through a broken and charred land that was once a shining beacon of production goodness. Heck, we could even include dice next time. “How many DBs are down?” “Roll 2d6”. Critical hit, go to DR!

What’s In It For Me?

Even though there are built-in rewards to many challenges — better queries, faster backups, certifications, what have you — people still want to have something a little more, particularly in the gamified experience. So what can you offer up your team or colleagues for a job well done?

Achievements: One of the cornerstones of game theory, achievements are fun and fulfilling. Use stickers, logos, Portal shoutouts, even badges inside the company intranet to show accomplishments. I personally think it would be neat to see “Achievements” next to Job Description, Contact Information, and Org Chart in my company portal filled in with things like “Onward and Upward – Upgraded three databases in a two week period” or “Did I Do That? – Fixed a fat-finger mistake with flashback or restores”. In fact, I would go out of my way to do amazing things just to earn more achievements.

Seminars and Conferences: Maybe the DBA with the most accomplishments gets to go to Oracle Open World this year. Or perhaps the developers with the best (or worst, to be more logical) queries get tickets to the next big Java conference. Whatever the case may be, I’ve noticed a growing number of companies unwilling to send their teams to conferences. In some cases they believe them to be merely a chance for the DBA (or other professional) to find another job. In others, they see it as a waste of money compared to other training. While ultimately these are horrible reasons to deny technical conferences to your team, perhaps using achievements and fulfillment of goals as a way to send people will improve performance as well.

A break: You could always give the winner a day to work from home (or off, you taskmaster you). Or for the thrill seeking types, the ‘losers’ have to give the winner a week off on call by taking it for them.

Will Turner: What are they wagering?

‘Bootstrap’ Bill Turner: Oh, the only thing we have. Years of service.

Company Certifications: Why not create a set of certifications within your own organization? As someone who has been in education and publishing for quite a while and as one of the lead writers of the Oracle 10g Certified Master exam, you can trust me when I say that certifications are an outstanding way to bring job satisfaction and a feeling of recognition to an IT team. Come up with certifications within your company that can be awarded for exemplary work or completion of certain achievements. Use those certifications during review time as a concrete measurement of success within the company. Make it possible for anyone to get certified in-house (e.g. developers can get in-house DBA certifications) to keep the career paths open at your organization. There are many ways you can acknowledge people for hard work; a certification they can put on their company profile, their emails, or even their resume can go a long way toward that goal.

Let them destroy something: This one sounds funny, and it is… but next time you migrate from one server to another or decommission an environment, let the winners of various competitions be the ones to type and press enter on that DROP USER command or DROP DATABASE. A company I worked for a long time ago did this. We finally moved off of an old and painful architecture. Once the database was moved over and the old one set to be deleted, we allows developers to drop the tables they hated the most and gave the honor of dropping the main schema to our lead (and most overworked) customer service representative. It was a great way to celebrate the occasion.

Conclusion

Some of these ideas may seem far fetched or downright silly. But you have to track employee progress, tasks, and status anyways. Why not do it in a way that gives everyone a sense of competition and accomplishment?

Gamification doesn’t have to be strictly for companies to rope in consumers. By using it within teams it can also improve morale, grow employees at a professional level, and give everyone something to strive for.