Application development in production anyone?
Bookmark :
While traveling to customer sites, I see many different setups for developing notes applications. Developing in production (yes, more times than you would think), two tier environment (Dev and production servers), and the recommended three tier environment (Dev, Test, and Production servers). Some people have separate domains for dev work, QA etc.
Two real-world instances I've seen in the field come to mind that document the perils of making code changes live in production.
1) Developer adds new columns and makes other changes to a View used for keyword look-ups. His code apparently functions as he expected when he checked against a particular document. However the changes made to the View broke many of the look-ups used from agents that run as background processes. Suddenly the sales team cannot quote, or sell at all. Coincidently, the backups had not been operating. It took 2 days to recover. With estimated lost revenue at approximately $750,000 USD.
2) Developer fixes some minor bugs and typo's (one of them a field name) on a Web-based application. Runs an agent to update all documents. And the data is updated correctly. However, because the Web-based Agents in the database contained the typo as well, they failed each time someone tried to open a document. Because the document was never opened, updates were not applied and critical client information was lost. This went on for 2 weeks before it was noticed and corrected. This cost his company 2 long-term customers at about $5 million in revenue per year.
Had the developers in both these cases run the applications through a proper test cycle, both instances likely would have been caught. The reason not to run the application in test was always because the changes were minor and there was expected to be minimal impact on the operation of the application. So do you still think the idea of developing in production makes sense? I'll bet many of you have had similar experiences. Don't be shy. Let's hear them!
Comments
IT was soooo much fun... seriously
I developed an initial application, demo'd it, the business liked it so we put it in.
They started using it ,emailing me functionality that would be usefull as they used it, based on customer interaction.
When I got an email I would do the dev in the template on the live server and then refresh the design. Then shout across the office that the functionality was in.
This was such a cool way of working the level of interaction between developer and end user was fantastic.True RAD.
That application is still live today, and ohh my god if you look at the underlying design its S**t, but its still working, still functional. (I am in the process of proposing a complete re-architecture of the app!!).
In reality I would never want to work that way again, and only got away with it because the company was so new and so small...
But it was exciting !
Posted by Neil At 05:31:22 AM On 09/27/2007 | - Website - |
I added a new agent to a db in production to alert Marketing of certain information.
A little while later, the Marketing called me to ask why I had just sent her 30,000 emails . . . and counting.
In this case it wasn't a career-limiting mistake. If the email had gone to a larger group of people it would have been a *real* problem.
The moral of the story:
a) Don't develop in Production
b) Test everything
c) If ignore these rules, make your changes using someone elses ID!
Posted by Anonymous At 08:28:51 AM On 09/27/2007 | - Website - |
Posted by Greg At 08:46:36 PM On 10/02/2007 | - Website - |
Apart from that, I believe that it really depends on the size of the company. We have about 250 employees and I'm the only Notes admin and developer and I do develop on the production server. Mostly it's adding functionality without changing what's already there. Or the database is completely new and only acessible for a few test users.
In the rare cases where documents change, I use local replicas to test the impact.
For such a small company, it just makes no sense to use two or three separate environments.
Posted by Thomas At 03:00:44 AM On 10/24/2007 | - Website - |
Posted by Craig Schumann At 07:29:00 AM On 10/24/2007 | - Website - |
Posted by Rob McDonagh At 10:30:32 AM On 10/24/2007 | - Website - |
Posted by Wayne Sobers At 04:42:44 PM On 10/24/2007 | - Website - |