I've spent some time this evening profiling a Python application on Windows,
trying to find out why it was so much slower than on Mac or Linux.
The application is an in-house build tool which reads a number of config files,
then writes some output files.
Using the RunSnakeRun Python profile viewer on Windows,
two things immediately leapt out at me:
we were running os.stat a lot
and file.close was really expensive.
A quick test convinced me that we were stat-ing the same files over and over.
It was a combination of explicit checks and implicit code,
like os.walk calling os.path.isdir.
I wrote a little cache that memoizes the results,
which brought the cost of the os.stats down from 1.5 seconds to 0.6.
Figuring out why closing files was so expensive was harder.
I was writing 77 files, totaling just over 1MB, and it was taking 3.5 seconds.
It turned out that it wasn't the UTF-8 codec or newline translation.
It was simply that closing those files took far longer than it should have.
I decided to try a different profiler, hoping to learn more.
I downloaded the Windows Performance Toolkit.
I recorded a couple of traces of my application running,
then I looked at them in the Windows Performance Analyzer,
whereupon I saw that in each case, the CPU spike of my app
was followed by a CPU spike in MsMpEng.exe.
What's MsMpEng.exe? It's Microsoft's antimalware engine,
at the heart of Windows Defender.
I added my build tree to the list of excluded locations,
and my runtime halved.
The 3.5 seconds of file closing dropped to 60 milliseconds,
a 98% reduction.
The moral of this story is:
don't let your virus checker run on your builds.
Title: Backbone.js Testing
Author: Ryan Roemer
Reading period: October 2013
Backbone.js Testing is a short, dense introduction
Mocha, Chai, and Sinon.JS.
Although the author uses a sample application
of a personal note manager written with Backbone.js
throughout the book, much of the material
Mocha is a test framework that can be executed in the browser or by Node.js,
which runs your tests.
Chai is a framework-agnostic TDD/BDD assertion library.
They complement each other and the author does a good job of explaining
when and how to use each.
I've written a lot of tests in Python (unittest and mock, primarily)
was both limited and years out of date.
with new browser frameworks and Node packages springing up everywhere.
Mocha, Chai, and Sinon meet those challenges, though they can't take away all the pain.
The author describes how to test Backbone models, views, and collections;
dealing with asynchrony;
provides useful testing heuristics, including isolating components to reduce dependencies;
when to use stubs and mocks and fake servers;
and test automation with PhantomJS.
He does not, however, teach you Backbone.js itself; for that, you'll need another book.
There are a few areas which I thought were dealt with too lightly.
There's no real discussion of Test-driven_development or
which provide the intellectual foundations of much of the book.
Nor does he have much to say about testability and how to make
legacy code more testable.
The sample Notes app has plenty of testing seams
(much of this falls naturally out of the architecture of Backbone);
other people's apps are not so lucky.
The chapter on automation is extremely terse—it could be expanded into a very large book!—but it does provide useful indicators to many areas for exploration.
I learned a lot from this book and I have no hesitation in recommending it.
Disclosure: Thanks to Ryan Roemer and Packt for a review copy of this book.
I've spent time over the last three weeks
working on a new website for the Northwest C++ Users' Group.
I blogged about the NWCPP website refresh over there.
In brief, I moved the website
from an instance of the Joomla Content Management System at Just Host
to a static website generated by Pelican and hosted at Github Pages,
and I'm happy with the results.
The Cozi Tech Blog needed some love,
so I wrote a post a couple of weeks ago on
Security 101 for Developers.
I wrote up some lessons that I learned about SQLAlchemy Sharding at the Cozi Tech Blog.
Cozi is hiring. We have positions in Web Development, Software Engineering, and System Engineering at our headquarters in Seattle.
Full details at the Careers Page.
I spent last Wednesday at Benaroya Hall,
attending the Seattle edition of StackOverflow's traveling DevDays conference.
It was well worth $99.
Joel Spolsky, owner of FogCreek Software and co-founder of StackOverflow,
opened the conference with a keynote about the
dichotomy of power and simplicity.
People are happier when not overwhelmed with choices.
Many of the choices that software forces users to make
are essentially meaningless to the users.
However, even though people want simplicity, they also want features
and different people use different features.
Powerful software sells more copies.
He argues that developers and designers should put in the extra work to make good choices
on behalf of the users: don't make users feel bad about themselves.
Undo is better than a confirmation dialog.
You are not in charge of what your users do.
Scott Hanselman spoke about ASP.NET MVC.
We're moving away from ASP.NET to Python,
but if we were to use ASP.NET again, MVC would be a compelling feature.
His presentation was entertaining, if gimmicky.
Rory Blyth introduced iPhone development, in a tone of snarky ambivalence.
He mentioned the Stockholm Syndrome.
He stressed that Apple's Design "Guidelines" are effectively laws:
violate them and you won't make it into the App Store.
Looks like there's a lot of tedious messing around to hook things up in Objective-C.
At the very end, he briefly demoed MonoTouch, which seemeed a little less tedious.
Cody Lindley introduced jQuery.
I've done a lot of work with jQuery, but I still learned a few things.
He worked through five facets of jQuery: Find something, do something;
Create something, do something; Chaining; Implicit iteration; and jQuery parameters.
He has an ebook at jqueryenlightenment.com, which I just picked up.
Daniel Rocha of Nokia talked about the cross-platform Qt (/cute/) toolkit,
which runs on Windows, Mac, and Linux.
More importantly from Nokia's point of view, it runs on their smartphones.
Nokia has changed the licensing of Qt—once very expensive for closed-source apps,
it's now free for apps that don't modify the Qt source.
Qt is for C++, but there are bindings for other languages, such as Python.
Joel Spolsky came back and treated us to a half-hour demonstration
of Fogbugz 7, Evidence-Based Scheduling, and Kiln,
their new hosted Mercurial repository.
Not terribly interesting to me, but the conference was only $99.
Ted Leung gave us a rather dry Hacker's Introduction to Python
from slides rendered unreadable by a poor choice of colors.
I've done a lot of Python, so I didn't learn much new.
pip is an easy_install replacement that uninstalls;
zc.buildout assembles apps from multiple parts;
bpython is a fancy REPL.
Dan Sanderson talked about Google App Engine
and demoed building apps with Java and with Python.
Looked pretty cool and straightforward.
We probably won't go that route, since we're pushing data to
Amazon's S3, so EC2 makes more sense for us.
Finally, Steve Seitz from the University of Washington
gave a cool talk on Modeling the World from Internet Photos.
Some of this technology ended up in Photosynth.
See Building Rome in a Day for some demos.
Title: Pragmatic Version Control Using Git
Author: Travis Swicegood
Publisher: Pragmatic Bookshelf
Reading period: 10–18 October, 2009
As part of my personal conversion to Git, I read Swicegood's Git book.
It's a decent introduction to Git and you learn how to
do all the basic tasks as well as some more advanced topics.
The examples are clear and well-paced.
I would have liked to see more about collaboration and workflow in a DVCS world,
perhaps a few case studies:
how is Git used in the Linux kernel development process;
how a small, distributed team uses Git and GitHub;
how a collocated team migrates from more traditional tools.
The book avoids discussing the lower levels of the Git object model,
which is a reasonable choice for a pragmatic guide.
More Posts Next page »