Just Enough Governance for Notes Blog

1 - Define "Change". :)

Do you only mean code changes? Or are data changes also part of your question?

I can see many reasons that data might need to be updated in production as part of operational support. And I can see where developers would be given that access.
Of course, if you are writing a private agent to do complex data updates, it should still go through a governance process.

But the question isn't why developers have this access -- it is more a question of where you draw the line between a change that requires governance vs. not.

We've talked about drawing that line with the Designer client -- if the change could be done by a power user without the designer client, it is probably OK to let a developer do it, too, as part of production support.

2 - To be clear, I don't think developers should make design changes in production either. As you can probably guess by my earlier posts on this overall issue.

That being said, I can imagine situations where it could happen relatively easily. Given: Notes being used in a prototyping, RAD environment with daily or hourly application changes; A lack of understanding that Notes applications are, in fact, real applications, fostered by the creation of many, many document libraries; and The very fuzzy line between when a Notes database is small and simple and used by 3 people in one department to when it becomes large and complex and used by hundreds of people across the company.

As I said, I don't condone it, I don't agree with it, and in my shops I don't allow it. But I can see how it happens sometimes.

3 - @Dave - By change, I meant design change, but I would also throw in any data change that affect the functionality of the application. So profile documents and configuration documents would fall into there also. I guess I can accept the gray areas the more you get into documents, but it just seems like a very slippery slope when you allow people to tinker with live applications.

4 - I do not approve of design changes on production, but Notes does have the backend and frontend in the same place.

So let's say you need some backend data from an application, to show a colleague. On a .NET/SQL application, you just query your SQL backend. On Notes, you create a shared view - thus changing the design.

5 - I need to start just the way everyone else has by saying I don't encourage or condone this.

That being said I think there are two reasons that this goes on. One is historical and the other is practical.

If you look at Notes' history going back to R3 days it was an egalitarian platform where many applications were developed by power users including sales people and administrative assistants. This was one of the great things about Notes at a time before the Internet. Unfortunately as Notes grew out of being a workgroup tool for ad-hoc collaboration to being an enterprise application server many of the habits of the old world died hard in some organizations. Not to mention some of the horrible applications that were developed and in some cases that live on.

The other reason this goes on is because it can and most of the time it seems like a great thing...there is a limitation or problem in the application and it is fixed or improved immediately. We all know that for every 10 changes that are innocuous there is one that is tragic which is remembered long after the ones that went by unnoticed have been forgotten.

The key is to structure the security on the applications in production so that there is no temptation to tamper with the production app because it isn't possible.

Regards,

Rob

6 - Any legitimate Lotus Notes shop should have a release or "move to production" process. Some shops are now locking out Notes Developers from seeing production design or data.

Changing production design/code on the fly is what gives us a bad name.

7 - @Tom - I think you hit on the point I was trying to make.... it's the reputation of Notes developers that is at stake here. On one hand there probably isn't a developer out there that hasn't felt the pressure of from a manager or business executive to make a change immediately. Then those same people start complaining that the Notes servers are an uncontrollable mess. When developers, or maybe even development managers, start allowing this sort of thing to happen, it will only hurt the impression of Domino in the long run. This is one of the big hidden benefits that developers overlook when you talk about governance. It's not all just red tape and procedures, these things are designed to help everyone, and can make your job and reputation safer in the process!

8 - Craig,

Because some of us perform tests on the development side rather than on the admin side.

I'm all for staging systems, but how often do you USE them? (Okay, you might, but I know plenty of people who have them and don't)

The value of well-governed change management systems is to MONITOR THE EFFECT OF PROPOSED CHANGES. If you separate dev from test from production, and you force developers to put stuff into test -- who authorizes the move from test to production? And what standard do they use for that decision?

That's not a Notes thing. That's an organizational governance thing.

There's no point to architectural separate unless you also have process independence. And there are a lot of organizations for whom the benefit doesn't outweigh the cost.

If your developers are fast enough at debugging problems, then SOMETIMES it makes more sense to deploy immediately, and endure some percentage of bugs and subsequent fixes, because it means for the X number of changes that DON'T break, your business value of those changes is realized much faster.

9 - @Nathan - So.... do you agree or disagree? I think you *can* have a pretty light weight and fast deployment model ensuring value of changes are realized quickly. I would argue that doing things the other way costs more in the long term, than you get from the short term quick fix mentality. Quick fixes often cause other quick fixes... how many goofs does it take before you loose any of that value? As you say, there may be X number of changes that DON'T break... but how many changes DO break...?
Of course it's not a one size fits all from an organization's perspective (which is why we called this *Just Enough* Governance for Notes), but I think there should be a minimum. Like you mentioned, it's the process that is the key. Maybe dev can move it to test *and* production. I would rather have a rudimentary smoke test in a controlled environment before my users have to experience some oversight I've made.... or worse, some agent that is now running amok because I didn't try it first. I think Bill and Paul have shown us plenty of the consequences of bad practices and untested code.

10 - @9 - Craig, let me offer a short answer.

"Just because it doesn't work for you doesn't mean it can't work for me."

11 - A few points

1) Lotus Notes is not the only technology this is done in. Database and web development in production are also both very common. The truth is the only environments you rarely see dev in prod is when you are dealing with code that has to be compiled before it can go live. While Notes is often the CIO whipping child in this arena its far from the only offender.

2) In small growing companies the Lotus Notes developer is the administrator in many cases. While they should use a dev environment, what can you say, absolute ACL power corrupts absolutely.

3) Often the speed of development (likely in production) is why Notes is chosen for grass roots projects. Everyone enjoys quick fixes till years later the application becomes mission critical and an edit in production finally creates havoc.

4) Even in controlled environments you can see a developer enter production under a "Break the Envelope" scenario.

12 - Brian, Thanks for jumping in!
I don't think you points have changed my mind... in fact, they have only strengthened my argument....

1) This doesn't make it acceptable. If Timmy jumped off the bridge would you do it too? I would be surprised to believe that ebay or Amazon tolerates developers making changes to live code...

2) If you know that there is temptation, why isn't the IT staff being managed better? Might sound harsh to say, but staying with the ebay example, this dev/admin you refer to could cost the company millions of dollars. Plus you knew it would happen.

3) I think your statement says everything. You are basically saying that you know you are going to get burned, but did nothing to protect against it.... very unprofessional!

4) This doesn't have to be the case. Proper testing and a control release process does not have to be a reason that you can't also do things fast. If there is a huge reason to fix a bug, maybe you should be reverting the production app back to the working copy and then fix the bug. Hopefully next time, you'll be more careful.

The whole thing can be summed up with "just because you can doesn't mean you should!" I still think Tom said it all in #6 above - all of these things just give Notes and everyone who loves it a bad name. The CIO is treating Notes as the whipping boy is directly related to ever point you made. If you want more respect from the CIO, it is going to ave to start with your practices.

13 - I'd like to introduce you to the concept known as a "small business". In this entity there is no CIO. There is no CTO. There is usually no IT Director. The IT department often consists of the IT Manager and may include one or possibly two other people.

This is the environment I have been in for the past 10 years. I haven't worked at a company with more than one developer (me) since the mid-90's. I have done consulting for larger companies with dedicated project managers, and I have done staged deployments. My day jobs have always been with smaller companies, though, and I prefer to keep it that way.

I used a test server for a while but it only confused my users. They would get frustrated when the workspace icon for App/Test no longer worked. They didn't care that there were multiple servers, they just want to click the icon and get to the application.

So, I meet my users needs by developing on the production server. They would rather not have to deal with a staged rollout, and I believe my job is to meet their expectations rather than tell them they're wrong. Just because you don't think this is a good idea doesn't mean it's not valid. I understand the risks and my users understand the risks, and I won't apologize for that.

14 - I am quite familiar with small businesses. In fact Teamstudio itself is a small business with only 70 to 80 people world wide. We have test servers and our users are educated about the test process and those who do not get it do not participate. Just my opinion, but I feel quite strongly that the user base should not dictate to the IT department what the best way to develop applications is. The users surely won't listen to you regarding the best way to do their job!
The size of the business shouldn't have anything to do with how IT is managed or the risks that the IT department takes. Developing in production equates to an increased risk to the business the IT department is serving. It's up to you as the defacto IT Director to make sure the business owner truly understands the consequences of you, the developer, making a mistake. So I think it *IS* your job to make sure things are done in as risk free a process as possible. When something does go wrong, it's going to be you that takes the heat, not the users.

15 - What I don't think is fair is to characterize everyone who deploys directly to production as an ignorant moron. To quote Nathan: "Just because it doesn't work for you doesn't mean it can't work for me."

16 - Hold on a minute. I didn't call anyone an ignorant moron. I said nothing of the sort.
I also didn't imply that deploying directly to production was the problem. The post was referring to practice of *developing* in production.

17 - It's not what you said, it's how it comes across. The tone I got from your comments was one of superiority and condescension, like anyone who disagrees must lack the intelligence to comprehend such a heady idea as multi-tier development.

I will concede that I collapsed the concepts of developing and deploying to production as having the same connotation. It seemed to me that others were using the concepts interchangeably and I went along. I probably misunderstood the intent.

18 - @Craig

I'm not saying I advocate any of the points I've given. But what I am saying is I understand how businesses have ended up with wild overgrown notes envinorments. Often the choices were not even made by the developer but the business needs themselves. Sometimes the advantage of speedy deployment and instand updates outweigh any desire for supportablility when it comes to certain business needs. Some small businesses cannot afford separate developers and admins and more than a single server.

What I am saying is you cant just say Lotus Notes as a product is the problem, or developers themselves are always the problem. Each problem is specific to the business and choices the business made. I've known few devleopers unwilling to do what they are told when the business gives a directive.

19 - Brian, thanks for the clarification. I understand how organizations can get into the messes that they are in. There are a lot of pressures that need to be balanced. However, as you say, the overgrown environments and under performing applications are the results of these earlier decisions. The end result is what people look at when considering whether they should stay on Notes. This is my point about being taken advantage of. Who ever it is who makes the choice to move off of Notes is not going to look at the track record of decisions they made to favor speed of development, but what the current situation is. If management isn't taking the long term view of maintaining the infrastructure and making choices that will ultimately undermine Domino, then it is up to us, the developers, to make those choices.
I also don't think this requires separate admins or developers. It makes a lot of things easier, but you can do all this with the same stuff you have today. Move development into templates that are kept in a subdirectory on the development server. This doesn't change anything except *where* the changes are made, which makes all the difference. Your test application can be in a test directory and have the ACL setup so that only you have access to it. You need to have "Just Enough" for your situation.

I'm nt trying to blame anyone, but I want someone to take responsibility. I have seen so many companies where Notes is treated like an outcast, that is dead or on the way out. When you talk with the admins and developers, everyone points the finger at management, but when you actually look around at the state of the apps and the servers, I would probably recommend the same thing. Management is always going to ask for the world, it's up to us to make it happen, and make it maintainable. Management doesn't know any better. We should.

20 - @Craig, Unfortunately I think more than a few admins and developers need to be pulled out of a depressive funk where they believe "Notes is dead." Until they buy into the faith again and do things right, they are a self fulfilling prophesy.