Sunday, October 03, 2010

Remembering a dark day ...

It must have been late June 2000, I had been in my first post university job for only a couple of weeks. The 2 main developers at the company were PhD students and desperately want to get back to their research but the current project had been eating more and more of their time until it was all they did.

In the late June 2000 timeframe we are talking about they finally declared the system good enough to launch, so it did. They immediately left the company to complete their research and I remained to maintain the system. Me, a newbie to "professional" development, new to Java and new to the rather large and complex code base they left behind.

For a couple of days things were fine. I tinkered with a few bugs and prepared the next release. That next fatal release when all of a sudden everything went wrong. Old bugs, from before I even started at the company, resurfaced. Nothing made sense and there was no one around that could help.

By about 3am I had managed to patch enough that the system was at least up if not fully functional. Over the next few days I patched bug after bug. Then one of the original developers came back, having put a decent amount of worked into his PhD. He looked over what had happened, the list of bugs that had seemingly returned from the dead and immediately realised what had happen.

You see there was no source control in place. So, to keep life simple, they had split the development into 2. While they both had a complete source to compile against, they would each only change one side of it.

This had worked through most of the development, until the day they left. When they put their source into a single folder for me, for the first time since starting. One developer put his source code in first, then the other put his in. Both his up to date correct half that he had been working on and his old incorrect copy of the other developer's half. Thus all of the bugs fixed in the previous weeks by developer 1 returned.

----

I recount this story because it has been in my mind of late. I have been putting a lot of effort into planning regression tests for a change I am implementing which first brought this story to mind. Also I was chatting to a developer the other day who was bemoaning bugs returning from the grave. Coincidentally I was also reminiscing with someone about horrendous problems we had on a project that did use source control, the variant of which turned out to be a costly mistake.

From my story and those of others in this area I take away 2 lessons that I try never to forget. Firstly, you must have source control, and an associated plan of how you will use it. Secondly, you cannot assume that simply having and using source control is the answer. While it may prevent the total disaster I presented above, I have seen many projects have source management issues even with source control in place. Be sure to review, on a regular basis, how you are using source control and whether it actually offers you the safety net you might assume it does.

23 comments:

Coach Factory said...

michael kors, true religion jeans, nike air max, air max, louboutin, nike air max, polo ralph lauren outlet, oakley sunglasses, prada outlet, ray ban sunglasses, louboutin, prada handbags, longchamp handbags, chanel handbags, oakley sunglasses, michael kors outlet online, true religion, michael kors handbags clearance, coach purses, toms shoes, coach factory, ray ban, louis vuitton outlet online, gucci outlet, michael kors, ray ban sunglasses, jordan shoes, tory burch outlet, burberry outlet online, coach outlet store, true religion outlet, nike shoes, michael kors outlet, nike free, michael kors outlet online, oakley sunglasses, kate spade handbags, louis vuitton handbags, kate spade outlet, longchamp outlet, air max, coach outlet, polo ralph lauren outlet, tiffany and co, cheap michael kors

mao qiuyun said...

juicy couture outlet
soccer jerseys
ralph lauren outlet
hermes bags, http://www.hermesbags.co.uk/
christian louboutin outlet
louis vuitton bags, http://www.louisvuittonbag.us.com/
louis vuitton handbags
cheap nfl jerseys, http://www.cheap-nfljersey.us.com/
fitflop shoes
ugg boots, http://www.uggbootsclearance.in.net
oakley sunglasses wholesale, http://www.oakleysunglasseswholesale.in.net
michael kors uk
michael kors outlet
the north face clearance
manchester united jersey
iphone 6 cases
pandora
michael kors uk
rolex watches
coach outlet store, http://www.coachoutletstoreonline.in.net/
giuseppe zanotti outlet
converse all star
lululemon
air max 2015
mcm handbags
michael kors factory outlet, http://www.michaelkorsfactoryoutlets.in.net/
michael kors factory outlet, http://www.michaelkorsfactoryoutletonline.com
cheap soccer jerseys
true religion jeans, http://www.truereligionoutletstore.us.com
abercrombie
tory burch shoes
hermes birkin
coach outlet
tory burch outlet online
oakley sunglasses
0910maoqiuyun

风骚达哥 said...

20150930 junda
Abercrombie & Fitch Factory Outlet
Abercrombie T-Shirts
canada goose outlet online
coach factory outlet online
Wholesale Authentic Designer Handbags
Oakley Vault Outlet Store Online
Louis Vuitton Handbags Discount Off
nike trainers
michael kors outlet
coach factory outlet
cheap toms shoes
Michael Kors Outlet Online Mall
Coach Factory Outlet Online Sale
michael kors uk
Abercrombie And Fitch Kids Online
louis vuitton
true religion jeans
cheap jordan shoes
Louis Vuitton Handbags Official Site
Mont Blanc Legend And Mountain Pen Discount
cheap uggs
tory burch outlet
ugg boots
coach outlet
Michael Kors Handbags Huge Off
Hollister uk
michael kors handbags
louis vuitton outlet
uggs australia
Discount Ray Ban Polarized Sunglasses

Wenhao Guo said...

air jordan 11 free shipping
nike air max 90
swarovski outlet
michael kors online outlet
nike running shoes
baltimore ravens
winter jackets outlet
san antonio spurs
cheap uggs
iphone cases
chicago bulls
coach outlet store
ugg outlet
true religion jeans,cheap true religion jeans,true religion jeans wholesale,true religion outlet,true religion sale,true religion outlet online
cheap oakley sunglasses
mizuno shoes
uggs outlet
replica handbags
kobe bryants shoes 2015
ghd hair straighteners
michael kors outlet store
new balance sale
rolex watches
true religion jeans
michael kors handbags clearance
tods shoes,tods shoes sale,tods sale,tods outlet online,tods outlet store,tods factory outlet
coach outlet online
hermes outlet online
chanel handbags
ugg boots
michael kors outlet online
mulberry outlet online
ddddd1110

xumeiqing said...

151204meiqing
gucci shoes
oakley sunglasses
p90x workouts
coach outlet
uggs for sale
coach outlet store online
oakley sunglasses
ugg australia
uggs outlet
louis vuitton handbags
giuseppe zanotti sneakers
montblanc
michael kors outlet
oakley sunglasses
pandora jewelry
louis vuitton
canada goose jackets
canada goose outlet
north face jackets
hollister uk
christian louboutin outlet
ugg boots
ugg boots
uggs outlet
uggs outlet
ugg outlet
louis vuitton handbags
oakley sunglasses wholesale
ugg australia
ugg boots outlet
concords 11
michael kors handbags
jordan retro
michael kors outlet
louis vuitton purses
christian louboutin outlet
christian louboutin outlet
oakley sunglasses
michael kors handbags
louis vuitton backpack

dong dong23 said...

nike roshe run
kate spade
cheap nfl jerseys
air jordan retro
jordans for sale
louis vuitton
kobe 11
toms shoes
ray ban sunglasses
louis vuitton handbags
nike air force
toms shoes
caoch outlet
coach outlet
giuseppe zanotti sneakers
basketball shoes
mont blanc pens
nike sb shoes
asics shoes
pandora jewelry
ray ban outlet
nike store
gucci outlet
hollister uk
designer handbags
louis vuitton outlet stores
coach factory outlet
michael kors outlet
ray ban sunglasses outlet
cheap jordans
tory burch outlet
adidas originals shoes
toms shoes
adidas stan smith
ralph lauren outlet
fitflops sale clearance
longchamp outlet
vans shoes
ray ban sunglasses
coach factory outlet
20164.22wengdongdong

Yuanyuan Lin said...

6.22lllllyuancheap nfl jersey
chrome hearts outlet
true religion outlet
true religion jeans
tory burch outlet
nike air huarache
nike huarache
cartier sunglasses
michael kors outlet
ferragamo shoes
coach outlet online
christian louboutin outlet
beats headphones
louis vuitton handbags
louis vuitton pas cher
true religion outlet
tiffany outlet
ralph lauren pas cher
herve leger outlet
christian louboutin shoes
christian louboutin uk
timberland shoes
michael kors canada
michael kors factory store
nba jerseys
tiffany outlet
basketball shoes
toms shoes
air jordan shoes for sale
adidas wings
mulberry handbags
ralph lauren outlet
ferragamo shoes
michael kors handbags
ralph lauren polo
6.22

Meiqing Xu said...

coach outlet online
coach outlet store
asics shoes
oakley sunglasses
ray ban sunglasses
uggs
adidas nmd
babyliss flat iron
canada goose parka
lacoste shoes
true religion jeans
ghd flat iron
discount oakley sunglasses
michael kors handbags
michael kors handbags
cheap jerseys wholesale
under armour outlet
adidas supercolor
cheap oakley sunglasses
fitflops shoes
nike air max 95
coach outlet online
longchamp
canada goose jackets
fitflops outlet
nike cortez shoes
adidas nmd
michael kors bags
white converse
louboutin uk
hermes outlet
cheap nhl jerseys
vans outlet
ray ban sunglasses
michael kors handbags
air max
zhuo20160714

Gege Dai said...

jordan 4
nike outlet store online
michael kors outlet clearance
instyler ionic styler
true religion jeans
polo ralph lauren outlet
ugg outlet
fitflops sale clearance
coach outlet online
michael kors outlet
ray-ban sunglasses
mulberry handbags
prada sunglasses
nfl jersey wholesale
herve leger outlet
canada goose outlet store
ugg boots on sale
ray ban sunglasses
toms outlet
louis vuitton bags cheap
lebron shoes
oakley sunglasses
ugg outlet online
pandora outlet
cheap mlb jerseys
ugg boots uk
michael kors handbags clearance
fitflops sale
cartier watches for sale
louis vuitton handbags
coach outlet store
gucci outlet
lululemon outlet online
tiffany and co
rolex watches
michael kors outlet online
swarovski crystal
0815jianxiang

Bert Carl said...

Nike Magista II TF
MBT Kisumu 2
Timberland 6 Inch Boots Shoes
Nike ELASTICO Finale III Street TF
Nike Roshe Run BR

Bert Carl said...

Adidas Originals ZX 9000
Adidas Originals NMD R1
Air Jordan Horizon Women
Adidas Ace 16 FG
Adidas Ace 16 Purecontrol
MBT Fora Gtx Damen
Nike Air Max 95
Air Jordan Flight 89

Bert Carl said...

Adidas Originals ZX 8000
Adidas Messi 16 AG
Adidas Originals NMD C1 Chaussures
Nike HypervenomX IC
Adidas Yeezy Boost 350 Women
Air Jordan 3.5 Homme
Nike Air Max 90 Homme
Adidas Climacool Fresh Bounce
Timberland Classic Oxford Herren

Bert Carl said...

Nike Elastico Pro III IC
Air Jordan 4
Air Jordan Westbrook Herren
Nike Free Flyknit NSW,Nike Free Flyknit NSW Wolf Grey
Nike Air Force 1 Homme's Chaussures
MBT Flame Chaussures
Nike Roshe Two Flyknit Shoes
MBT Kimondo Men

chenmeinv0 said...

louboutin outlet
cheap rolex watches
canada goose outlet
burberry scarf
spurs jerseys
ugg boots outlet
michael kors outlet clearance
louis vuitton handbags
uggs sale
gucci outlet online
chenyingying20161130

Jian Zhuo said...

louis vuitton canada
true religion outlet
adidas outlet
louis vuitton outlet
coach outlet
michael kors outlet
louis vuitton outlet
coach outlet
ugg australia boots
christian louboutin
2016128

Bert Carl said...

Adidas Messi 16 FG
Nike Air Max Motion LW
Air Jordan 4
Adidas Ultra Boost Olympic
Supra Bleeker Women
Timberland 6 Inch Boots
Nike Zoom Ascention Men
MBT Kitabu Gtx Shoes
Air Jordan 2 Men
Adidas D Rose 7 Men

Bert Carl said...

Adidas Stan Smith
MBT Fora Gtx Women
Timberland Chukka Boots
Nike Magista Obra IC
MBT Lami Damen
Adidas Messi 16.3 IC
Timberland 6 Inch Boots Herren
Nike ELASTICO Finale III Street TF
Adidas NMD C1
Nike Air Max 2016

Bert Carl said...

MBT Kimondo Gtx Men
MBT Fora Gtx Damen
MBT Kifundo Men
Timberland 6 Inch Boots Shoes
Air Jordan Reveal Men
Nike Air Jordan flight 45
Nike Air Yeezy 2
Air Jordan 3.5 Homme
Adidas John Wall 2 Homme
Adidas Climacool Fresh Bounce

chenmeinv0 said...

toms outlet
coach outlet online
christian louboutin shoes
ugg boots uk
louboutin pas cher
ugg outlet stores
louis vuitton outlet
nike huarache 2015
air max 2016
gucci outlet
20171.23chenjinyan

chenmeinv0 said...

ray bans
ray ban sunglasses outlet
ray bans uk
ralph lauren polo
nike roshe run pas cher
air jordan shoes
borse louis vuitton
ray ban glasses
kate spade handbags
louis vuitton outlet stores
2017.2.23xukaimin

dada xu said...

true religion jeans
cheap oakley sunglasses
coach outlet
christian louboutin shoes
pandora jewelry
coach outlet store
oakley sunglasses
coach outlet
longchamp bags
timberland boots
chenshanshan20170318

caiyan said...

coach factory outlet online
coach outlet online
polo ralph lauren outlet
coach outlet online
coach factory outlet
christian louboutin outlet
rolex replica watches
birkenstock outlet
jordan femmes pas cher
fitflops sale
0405shizhong

chenmeinv0 said...

mlb jerseys cheap
columbia outlet online
adidas neo shoes
coach factory online
ed hardy jeans
kd 8 shoes
nike air max 2016
longchamp handbags
kobe 11 shoes
red bottom shoes
2017.4.28xukaimin