Match.com Post Mortem

Well, I never really talked about how things went with our Match.com 5.0 roll-out and implementation.

Amazingly, it was one of the smoothest, most successful development projects and roll-outs that I've ever been involved with. We recoded an entire (HUGE!) site in about 8 months, including definition and design.

On June 12 at 2:00am in the morning, with a Microsoft representative on site, we went dark, taking all 104 servers (running Match.com 4.0/ASP.OLD) out from behind the load balancer and redeploying 45 new servers (running Windows 2003) into the VIP and going back live.

Holding our breath, we opened our eyes to look at the operations screens and perf mons running... and all was well.

A little initial CPU hit (maybe 10-15% increase) across the web farm while the site JIT compiled, but then leveled out around 20% CPU utilization. The GC did it's merry little job, and memory stays flat. Never moves.

And, at the end of the day, we ended up with 49 assemblies, and just over 500k lines of code.

Again, in a word: Amazing.

At the end of the day, we cut our server farm in half to a site that is twice as fast, and twice as easy to debug, maintain, and build upon. We saved the business hundreds of thousands of dollars, and will continue to be able to do so because we can offer extremely fast development cycles.

One of the greatest accomplishments in my career, and one that I'm very proud of. And, one that says alot about .NET and the thought, planning and implementation that Microsoft has put into it.

Filed under:

Comments

# julie said:

wowee! Congrats!! It's very cool to hear stuff from the trenches of some very well known and very huge sites. I actually have a very happily married friend who met her hubby on match!

Thursday, July 03, 2003 11:12 AM
# Roy Osherove said:

Great to hear!
My boss always asks me if I can point him in some succssfull big .Net projects.

Thursday, July 03, 2003 12:20 PM
# Phil said:

cool. Will I be able to find a date easier, now that you converted to .NET?

Thursday, July 03, 2003 4:10 PM
# TrackBack said:

Jeff Key's .net blog

Thursday, July 03, 2003 4:51 PM
# TrackBack said:

Patrick Steele's .NET Blog

Thursday, July 03, 2003 4:51 PM
# TrackBack said:

Match.com Success Story : IDunno

Thursday, July 03, 2003 4:51 PM
# Dan Bright said:

Heads up Jason, you got /.'ed.

Saturday, July 19, 2003 7:58 AM
# Kevin said:

I was using Match.com last night. It is a great site/service. But I have one major issue, it doesn't work very well with Mozilla. The biggest bummer is that the Match.com Messenger agent (chat) does not even come up when using Mozilla running on Linux. There were a few people online I would have liked chatting too, but couldn't even send them a hello. :(

Hope to see some better support for standards compliant browsers in the future.

Saturday, July 19, 2003 2:32 PM
# TrackBack said:

xL8 - Grant Carpenter

Sunday, August 03, 2003 10:14 PM
# TrackBack said:
Sunday, August 03, 2003 11:07 PM
# TrackBack said:
Friday, August 22, 2003 1:53 AM
# Scott Mitchell said:

Kevin, perhaps Match is trying to send a message to Linux users, namely that no one would want to date your greasy, overweight, non-showing butt.

(I kid, of course, as I use Linux myself. :-)

Friday, August 22, 2003 1:35 PM
# TrackBack said:
Tuesday, February 03, 2004 12:21 PM
# mattb said:

I thought in total you ran 78 servers and including 45 webservers. Not exactly cut in half?!? Assuming you actually perchased new servers, going by the fact you state you brought down you old system and powered up your new ones, then I would expect your new servers to be much faster than the old ones. If this is the case then your CPU improvements aren't really that great.

Wednesday, February 18, 2004 11:33 AM
# Jason Alexander said:

Mattb -

I'm not sure where you get your "78 servers" from, but that certainly wasn't the case. We began with 100+ web servers, and rolled out with 45 (we've since had to add a few extra, due to our traffic steadily increasing).

We did buy new servers, but they weren't significantly different, from what I understand. Dual proc boxes, same as before.

So, I would say that there were significant differences. You make quite a few assumptions in your review, and jump to you own conclusions. Stick to the facts of the case study.


-Jason

Wednesday, February 18, 2004 11:39 AM
# Anonymous said:

Cut server count by 60%, cut available CPU MHz by 20% (servers were 2x1.4 P4, now 2x1.8 Xeon). CPU utilization as an average decreased at least 15%, running on 40% of the hardware as before.

Less servers to manage, easier code to manage, less monthly expenses, way more stable, and much more efficient.

You do the math. It really was an improvement all around.

-JT

Saturday, February 28, 2004 1:10 PM
# Chris Leslie said:

I belive you guys use Vault for your Source Control?
Any comments on it, did you use it during the development cycle of this project?

Friday, March 05, 2004 4:58 PM
# Matt said:

Sorry I didn't make myself very clear.

"I'm not sure where you get your "78 servers" from, but that certainly wasn't the case. We began with 100+ web servers, and rolled out with 45 (we've since had to add a few extra, due to our traffic steadily increasing).
"

You run in total now 78 webservers.

"The farm consists of 45 web servers, 4 file servers, 20 image servers, 6 read-only databases, 2 keyword search databases, and 1 master writeable database"

And you started with 104 webservers. It doesn't say how many other servers you have unfortunately. My reply was based on the fact you ran 104 servers and now run 78, so not cut in half.

I would be interested to know your previous architecture as all MS says is ran 104 webservers and now have :

"4 file servers, 20 image servers, 6 read-only databases, 2 keyword search databases"

This seems to be a more complex solution and to be honest having developed web systems for a good few years your application doesn't actually seem that complicated so to have all those servers smacks of traditional MS overkill and complexity.

Matt

PS I actually use MS products such as ASP, Biztalk etc

Monday, April 26, 2004 9:16 AM
# Ganesh Bhandari said:

Hello Friend ,
how ri usend Match.com Messenger
i don't know about it plz send me detils infrometion plz i'm wating for your mail ok bye see you soon !!! wth mail ok bye !!

Friday, April 30, 2004 12:01 AM
# countrygirl199 said:

have tried to show my pictures but they say countrygirl199 is not a good name

Wednesday, June 16, 2004 11:12 PM
# countrygirl199 said:

have tried to show my pictures but they say countrygirl199 is not a good name

Wednesday, June 16, 2004 11:12 PM
# Barnaby said:

I'd just like to add a few comments...

1) match.com messenger on mozilla is a big pain. I would assume that your code has some sort of check for IE which is annoying. There's no real reason to code for one proprietary platform like IE has become these days

2) the new match.com interface on the main persoinalized page is totally screwed. I don't know who wrote the code but no matter what version of IE I view the site on the twisties still refusse to function properly unless I close and reopen them first...bad code...

as for .net woohoo but really i agree with the preior post...seems like you went from MEGA-kill to overkill...match.com isn't really that complicated

Thursday, July 08, 2004 9:04 PM
# ... said:

Interessante Informationen.

Tuesday, March 03, 2009 4:46 AM
# ... said:

Gute Arbeit hier! Gute Inhalte.

Tuesday, March 03, 2009 4:05 PM
# ... said:

Sehr wertvolle Informationen! Empfehlen!

Saturday, March 14, 2009 6:51 PM