uber1024's WebLog

Firewall in between web and database servers can cause intermittent problems

uber1024 — Mon, 26 Nov 2007 15:37:00 GMT

This story starts, as all stories should, with me sitting on a beach outside of Sao Paolo, Brazil. My team was releasing some bug fixes to our production setup (we do this every week). When I got back, I saw that our sites were having very intermittent problems executing SQL against our database. A trace through the code found that they were able to establish the connection to the database with no problems, but when the code actually ran a query (or stored procedure) using that connection, we'd see various mystifying error messages ... out of memory, SQL Server doesn't exist or access is denied, and some DBNETLIB errors.

Let me step back a step further and outline the setup. We have two load balanced web servers running an application of mixed ASP and ASP.NET. It's a homegrown CMS built over the last 7 years by a series of talented (and many not-so-talented) developers. The data comes from a SQL Server 2005 cluster that resides on a different subnet from the web servers. There are redundant Nokia firewalls that control all traffic into, out of, and between the networks, and the firewalls were managed by a third party.

The most inexplicable thing about this whole scenario was that the sites would experience these problems only during periods of low usage. During peak activity times they clicked along nicely. It was confusing at first. Our research started to point us to the firewall and we found that the source of the problem was that the firewalls would sever connections that showed no activity for 3600 seconds. What was happening was that the web servers would open database connections and pool them. After an hour of not being used, the connections were severed at the firewall level but the web servers still believed them to be valid and usable. When users would use the site, one of these unusable connections would be drawn from the pool and an error would result.

Fortunately for us, our firewalls were configurable. We placed a policy change request into our firewall team and increased the timeout period. The problems immediately ceased. Our current working theory about why this problem began while I was away is that one of the bug fixes that we put live fixed a bug that was subtly causing more traffic between the web and database servers which resulted in the connections in the pool being kept alive by the firewall. We believe this to be true as we noticed a 2500% increase in packets dropped on port 1433 on our firewall after we released the bug fixes.

One guy's view on H1-B visas

uber1024 — Sat, 07 Jul 2007 14:37:00 GMT

I know this is a touchy subject, as well it should be. I'm going to try to approach this from the perspective of a technology executive who is a US citizen and who has worked in the tech industry since 1999. For those who know me, you know that this is not a difficult point of view for me to have. Next time I will write a blog entry from the perspective of a dude who likes watching football and drinking beer ... also not a difficult point of view for me to hold.

The situation in New York City might be different than the situation in Witchita, Kansas, so keep in mind the fact that most of my career has been spent in NYC and my entire executive career has been in this technical gene pool. Also keep in mind the fact that I'm not really trying to represent the industry as a whole, the industry leaders, or the college graduate looking for his first job. I'm trying to represent my own views that have crystalized after sitting on both sides of the interview table dozens of times.

My position is simply that the H1-B visa quota is good, but needs to be raised pretty significantly and maybe be made a little more reactive to the needs of the industry. It's good because there needs to be some effort made to protect US citizens. After all, that's the purpose of laws and governments. We might not always be happy about that, but it's the truth. I wanted to move to Korea and work there for a while, but I found their laws to be exceptionally protectionist and xenophobic so I don't find the US laws to be draconian at all.

However, having been in the industry during the 2000-2002 dot com shakeout, I realized later that this was actually the best thing to happen to the tech industry in a while. There were a lot of people who washed out of the industry in that time and the vast majority of them deserved to. This is a highly-paid field (even with foreign competition dragging down salaries a bit) and, as such, we need the best people we can find in it. My company recently raised our rates to $150 an hour for our technology department (we don't see that $150 an hour, in case you're wondering) and our clients expect to get their money's worth. Now, as the leader of the department it's my job to make sure that our programmers are worth that $150 an hour that our clients are paying.

And that's where we come back to the H1-B visas. At a time like this, competition for the best developers is tight. I'm trying to hire another developer, for example, and the resumes I've been getting are crap. They're not "just okay," they're complete crap. I'm not talking about "I just graduated from college and I'm going to try to be clever in my resume design no matter how many people tell me it's not a good idea" kind of resume. I'm talking "I don't have any clue what I'm doing and I'm going to document that fact in resume form" kind of resume. I met a programmer from South America who just knocked my socks off. She is an outstanding developer, she's well-liked by her peers (ie, she's a great team player), and she's showing signs of developing strong leadership skills. Basically, she's a top-notch candidate for a mid-to-senior level developer role ... exactly what I'm trying to hire. Unfortunately, I can't. It's difficult to understand why we would want to keep a smart, productive person from moving to our country, contributing to the economy, and paying taxes. I understand that this is the way it is, but it needs to change.

I'm trying to hire someone, and US citizens are always easier to hire for many reasons (less paperwork, shared background, familiarity with the American way of doing business). But the pool of US citizens is poor right now. The best people have jobs and are happy there. Scraping the bottom of the barrel is not an attractive business option. If I know someone who is interested in moving and working here that would be a perfect addition to my team ... why shouldn't I be able to hire her?

As a developer, I remember lots of my colleagues being upset as Bill Clinton continually raised the cap on the number of H1-B visas that were available each year. I tried to understand why, and it always came down to one of two reasons. The most common reason was that the people in question flat-out sucked as developers and knew they were not capable of dealing with the competition. The second, definitely less common but still heard often enough to appear on my radar, reason was xenophobia. By which I basically mean "racism." I'm not even going to dignify that sentiment with a rebuttal because it's ridiculous.

I think there's a middle ground here. There were proposals to have the visas be capped but with provisions to increase the cap if necessary. The EB-2 visa has provisions built in that allay the fears of many. To sponsor someone for an EB-2 visa, you have to be able to prove that you attempted to fill the job with a US citizen but were unable to do so. Maybe this is an idea that can be applied to H1-B visas. It would certainly make it easier to bring in talented people to ease tight labor markets and would also help US citizens find jobs during tougher times.

I'll note here that I'm not really involved in setting salaries for my programmers. That's the operations department. I just let them know how good a candidate is and they make the offer. We have one programmer here who is H1-B and I know that she is paid the same as the US citizens. She's my best programmer and I wouldn't have her paid any less than she's worth. The other thing to note, salary-wise, is that my current programmers are going to pay the price for the exclusion of talented foreign workers. They're going to have to cover the slack from the missing developer role while I interview retard after retard.

I'll finish this off by saying that I realize that the problem that many people, including me, with the current system is not the immigrant workers themselves ... but the abuse of the system by large companies. I agree that this is a problem, but punishing companies who work within both the spirit and the letter of the law, and punishing future US citizens is really not the way to go.

Hopefully this entry provokes some thought. I'm sure it's going to provoke some reaction. Whatever your opinion on the matter, I ask two things. The first is to think about why you feel the way you do. REALLY think about it. Look in the mirror and ask yourself why you feel that way. I did, and I didn't always like what I realized. The second is to at least make an effort to be civil when discussing this topic with people. I've found an ENORMOUS amount of ignorance about immigration laws, visas, and the state of the industry and country when discussing immigration. It's a hot-button issue, politically, these days but I wanted to focus more on the high-tech workers and not the low-paid, under-the-table illegal immigrants that most people think about when they think about immigration. I hope that this entry has offered a perspective on this topic that helps people understand the issue.

Linking 64-bit SQL2k5 to 32-bit SQL2k ... or, How to Avoid Wasting 10 Days Waiting For Straight Answers

uber1024 — Sat, 07 Jul 2007 14:04:00 GMT

They say a picture is worth a thousand words, so here's a thousand words. The situation is that we have a 32-bit SQL2k cluster and a linked 32-bit SQL2k machine (they link in both directions) and we need to execute distributed transactions across both machines and in both directions. This works fine, and has been working for many months (after we spent a month trying to get our hosting provider to correctly install MSDTC ... another story entirely).

It was recommended by a consultant (I know, I know, readers of WorseThanFailure aka TheDailyWTF are snickering up their sleeves at me) that we replace the cluster with 64-bit windows and 64-bit SQL2k5. He had the picture of our network and assured us that it wouldn't cause any problems. Predictably, it caused a host of problems immediately. It's been 2 weeks and we're still not done with the project.

The problem is that we were getting a pretty mystifying error message:

The stored procedure required to complete this operation could not be found on the server. Please contact your system administrator.
Msg 7311, Level 16, State 2, Line 1
Cannot obtain the schema rowset "DBSCHEMA_TABLES_INFO" for OLE DB provider "SQLNCLI" for linked server "<LinkedServerName>". The provider supports the interface, but returns a failure code when it is used.

I found the MS knowledgebase article which was, as per usual, not really as helpful as it could have been. I spent 10 days talking to the database guys at our web host (and these guys are typically pretty sharp) and it finally came down to "you are on your own," which is what I figured. We all looked at the KB article and couldn't really figure out what, exactly, we were supposed to do. As a former programmer, I figured I'd dive into some code and lo-and-behold I had a breakthrough in like 45 seconds.

When SQL2k5 64-bit is trying to select data as part of a distributed transaction, it tries to run a stored proc called "sp_tables_info_rowset_64". That jumped out at me immediately. The 32-bit database has a proc called "sp_tables_info_rowset" that does the same thing. So I added the following stored proc to my 32-bit SQL2k instance:

create procedure sp_tables_info_rowset_64 @table_name sysname, @table_schema sysname = null, @table_type nvarchar(255) = null

declare @Result int
select @Result = 0
exec @Result = sp_tables_info_rowset @table_name, @table_schema, @table_type

So ... kancho to the knowledgebase article that was going to have me run a 935KB sql script that would make massive changes to the master database. From the command line. On a Saturday morning. If I would have done it their way, I'd be here all day. Doing it my way, I'm done in time to dash off a blog entry and then have breakfast with a beautiful girl.

Note to Microsoft: This is why people hate upgrading. You make our lives suck when we try it. Remember that we don't want to spend our weekends fixing stuff. It's not about the money. It's about finding out why they call it a Brazilian wax.

New Year's Resolution for 2007

uber1024 — Mon, 08 Jan 2007 21:00:00 GMT

As the technical director for our company, it's up to me to chart a path for the company through the next year and beyond. My current plan involves:

1. Less dependence on Microsoft

Let me explain why. I've been a developer who has been using almost exclusively Microsoft technologies for the past 8 years. When I started, MS development was just easier and faster than anything else out there, and if you needed to upgrade to a new version of a development tool ... it just worked. I upgraded half a dozen VB apps from VB5 to VB6 in an hour and rolled them out the next day to 400 desktops with no problems. Then .NET came around and that was a bigger transition. Fine. Most of our sites don't use .NET because of it. If you have the time to rewrite your website from the ground up in a new technology just because you thought it was cool, that's awesome. I'm happy for you, really, because you've got initiative and time. Meanwhile, I've got 180 sites written in ASP 3.0 that all work, and work well. We're not going to undertake an expensive, non-billable effort to redo the past 6 years of development.

That's understood. When .NET came around, Microsoft flat out said, "yes, we realize it sucks but that's the way it is." And we all kind of forgave them because, up until then, upgrading technology was something that was entirely within the realm of possibility.

Then Windows 2003 came out, and it was a giant kancho(*) for a lot of us. Win2k3 makes too many assumptions about my practices and it's wrong about as often as it's right. But upgrading wasn't terrible. We were able to do so and only lost a couple dozen hours (again, non-billable). We did it because Windows is easier to maintain than Linux and because we still wanted to do ASP and ASP.NET development

Then .NET 2.0 came out. We have a few .NET 1.1 web services that we wanted to use and we hoped to replace some of our ASP pages with ASP.NET 2.0 pages. Can you run them side-by-side? No. Another kancho. I'm sure there's a good reason for it because there are a lot of smart people working for Microsoft. However, I'm not interested in that reason. I'm interested in my team and our work.

Can you easily upgrade a site from 1.1 to 2.0? If you've done it, great. I'm happy for you. It took me yesterday (yes, I came in during a playoff weekend to do this) and today to get almost back to square 1. Except that my web service doesn't recompile because the ConfigurationSettings class is now obsolete?

Kancho on me, I guess.

These things, in and of themselves, suck. The decision makers at MS have let me down repeatedly. But these are just indicative of a lot of things happening at Microsoft that are just a constant stream of urine in the cheerios of developers around the world.

Have you tried to find help on MSDN lately? It used to be one of the best collection of documentation ever assembled. What happened?

Have you tried to get info on licensing? Or being a MS Partner? I don't want to have to spend 45 minutes looking around their site to find out how much money I have to give them.

Really, all I'm saying here is that Microsoft used to be about making life easier for "Developers! Developers! Developers! Developers." As a developer, this isn't true any more. As a technical director, now I have the abililty to look around and say "we're not getting what we used to get from Microsoft, so why are we not only still giving them piles of cash, but why are those piles getting bigger?"

Some of their products are solid. SQL Server is a good example. It just works and you don't really have to worry about it. I never have to "reboot SQL Server." The development tools are also great, provided that you're making small (1000 files or less) sites. I'm just having trouble seeing the value that they're adding to the business that I'm trying to run. I know I'm never going to be able to get us entirely away from MS, but my goal is to make my job less dependent on the vagaries of Microsoft's decision making, which I no longer trust has any of my interests in mind.

(*) - Kancho is a kids game in Japan. It's called Ddong-chim in Korean.

Asia is pretty wired

uber1024 — Mon, 03 Apr 2006 18:24:00 GMT

Japan has recently overtaken South Korea as the most wired country. I think Korea might still have the highest broadband saturation (I understand it's around 75% of households that have broadband ... which is pretty stunning given that 60 years ago Korea was the poorest country on the planet). Somewhere, I also read that Chinese people spend the most amount of time online, on average.

I never lived in Japan or China, but I have lived in South Korea, and Korea is an interesting study in internet technology and its ramifications on the culture and economy. Cyworld minihompis are what myspace hopes to be, except they don't suck like Myspace does. Text messaging is far and away the fastest, easiest way to communicate and it's typically free there. Cellphones work on the subways, which was a huge shock to me, and they're pretty much all web-enabled. Internet cafes are everywhere and cheap (about a dollar an hour), and even girls play games online. As a web developer, it's exciting for me to see a country that so embraces the internet and has figured out ways to make it work for them. They also have kickass food, soju is great after the 1st or 2nd shot but gives the drinker an almost unholy need to sing at noraebang, and the people are awesome if you speak Korean to them. The local beer sucks, though (Hite is pretty bad, you have to admit).

Intermittant problems opening Access files on a web server

uber1024 — Fri, 24 Feb 2006 18:26:00 GMT

One of the more popular features on one of our websites is a page where users can upload Access databases that the website will then open (with ADO -- it's an ASP-based application). This mechanism was in place long before I got here, or I would have tried to find a different way to do this. That being said, we're stuck with it for the forseeable future as the users love it, and users indirectly pay my salary.

This was working fine until we moved our sites onto a Win2k3 server (from Win2k). Now, the feature is accessible intermittantly (as in ... it has about 10% availability). The only thing I can tell is that it fails on the Open method of the Connection object. I checked the temp directory to ensure that the ldb files are going there, and I ensured that the right users have write access to that directory. They would have to, otherwise this would never work. The error message comes back as "Unspecified error".

Microsoft technical support hasn't been much help on this issue. Usually they're very helpful, but on this one they didn't have any ideas. I was just wondering if anyone had come across this before.

Commerce Server vs. Site Server object performance

uber1024 — Mon, 30 Jan 2006 17:16:00 GMT

We recently replaced a Site Server 3.0 site with Commerce Server 2002. Let me step back a minute. The sites don't really use the the server packages as intended. They use the pipelining process and the order form object model and that's about it, so we really miss out on a lot of opportunities here to leverage some of the cooler features of CS2002. However, the process of moving from one platform to another was relatively painless compared to the effort of moving a site that had bought into Site Server completely. The site is old, as in "from about mid-2000", so the architecture was around long before I started here and certainly long before I was in my present, decision-making role.

Anyway, that's where we are. And we're noticing that the Commerce Server 2002 objects are orders of magnitude slower than the Site Server objects. It would be nice to be able to "unroll" those objects and create pure SQL tables out of them rather than leaving all the data in the marshalled baskets, however no one is willing to pay for that effort as it would be a significant amount of work (yes, the developer in me definitely wants to take the time to do it right. However, the director of technology in me doesn't really want to tie up my resources on this project for 3 months. Funny how things change as you move up the ladder. A few years ago, I would have said that my DOT was a dumkopf for not taking the time to do things the best way possible. I'd like to take this time to apologize to all the tech leads and project managers that I've fought with over the past few years.)

Throughout most of the site, this isn't a problem, but there are two places (in administrative pages) on the site where 100 or more of these marshalled baskets get retreived and opened, and these pages are taking minutes longer to load than they were previously. I don't expect that anyone has ever learned anything about this, I just wanted to have documented it somewhere.

File downloads from IIS 6.0

uber1024 — Fri, 20 Jan 2006 19:58:00 GMT

So we've moved 8 websites from Win2k to Win2k3. Some things have gone smoothly, some things have made me want to become a Buddhist monk (which actually wouldn't be that tough since I train in kung fu at a Buddhist temple here in Manhattan). One of the latter was enabling file downloads from the website. Basically, our client has a desktop app that needs to download updates from the website. We have a web service that lists the latest updates and their location and the app is responsible for everything else. Simple yet effective.

So after the switch, the app stopped working for both Mac and PC users. It turns out that Mac users need to download a .DMG file (which apparently is a mac-specific file format that is used all the time) and the Windows version needs to download both .EXE and .DLL files (and a few other types of files, but everything else went pretty smoothly). None of these were working.

The .DMG files we could get to download by opening up the website in the IIS snap in and looking for the MIME type applet and adding .DMG files with the right MIME type. I don't recall off the top of my head what that MIME type was, but you can find it on Google (although the Bush administration will eventually have access to that information). The .DLL and .EXE files were a little tricker. I had to switch the "Execute Permissions" from "Scripts and Executables" to "Scripts only."

For whatever reason, IIS 5 was like Washington Square Park (*) ... you can get anything at any time provided you know where to look. IIS 6 is much more secure, and you have to take steps to explicitly enable certain activities. That's fine and it makes sense. Fortunately there are a number of good bloggers from Microsoft, otherwise this issue would have been much tougher to track down, since the only place I found it talked about was on a blog or two. This tells me three things:

1 -- books are nice, but typically come out too late to be much help for most of us
2 -- keep blogging, everyone! It's a big help to all of us!
3 -- does anyone ever go to MSDN.com any more? It's about as useless as MSN.com. I feel like I'm being sold things rather than being allowed to look up something that's going to help me with my job. I always check MSDN when I've got a problem, but it's literally been years since it was actually useful.

(*) - For those of you who are not old skool New Yorkers, or who are New Yorkers but don't have as many shady friends as I do, Washington Square Park is a park down in Greenwich Village near NYU. It's long been The Place To Go when you're looking for less-than-legal pharmaceuticals. Some days, it reminds me of the old rave scene where it's just a marketplace for drugs of all kinds. These days, there are more police officers patrolling the area so it's not as scruffy as it used to be, but you'll still see people loitering in the evenings saying "smoke, smoke" under their breath. I happen to work 2 blocks away from the park, and one of my friends lives 2 blocks in the other direction, so I've walked through the park many times at all hours of the night.

AJAX to secure servers: semi-resolution

uber1024 — Tue, 10 Jan 2006 21:10:00 GMT

I had previously talked about a problem making XMLHTTP requests from IE to our secure (HTTPS) webserver. Microsoft support was very understanding but in the end it came down to this:

"We can't help you. We've known about this but have no resolution. If you figure something out, let us know."

We wound up rewriting our engine to do the Posts asynchronously and then setting a timer and trying again if the request took too long. It was an ugly, ugly hack and there was money left on the nightstand after that one, for sure. The only thing that would have been uglier (and probably wouldn't have worked) would have been to use IFRAMES. I hope to never type that word again in my professional career.

Email woes when upgrading from Win2K to Win2k3

uber1024 — Tue, 10 Jan 2006 21:04:00 GMT

The SMTP standard, which no one not featured in the mockumentary "Trekkies" has read, states that bare linefeeds ( "\n" ) are not allowed in the SMTP standard. Qmail servers interpret this strictly. There are qmail fanboy articles explaining why this is good. That's fine and the world continues.

However, IIS 5 would catch what amounted to mistakes by developers and would silently rewrite "\n" as "\r\n", which conforms to the SMTP standard. For whatever reason, IIS 6 does not. I can't fathom why this feature wouldn't be included for backwards-compatibility, but it seems that backwards compatibility is aggressively discouraged. This is unfortunate.

I'm having more and more "I hate Microsoft" days since Windows 2003 was adopted. We have our main website running on Win2k3 and we're moving another (although I wish we weren't). Our other 17 websites will remain on Windows 2000 until such time as I no longer work here.