How Software Development is Like Finishing a Basement

Last year Carrie and I bought and moved into a new house. Our strategy was to move in then finish the unfinished basement ourselves, building up some sweat equity. Now we're about halfway finished and I see some obvious similarities between building a basement and building a software application.

We decided to be agile in our basement finish, rather than work from a predefined blueprint. Some decisions would be put off until necessary; like the colors of carpet and paint, the style of lighting fixtures, and even the placement of some doorways and size of closets. But some things have to be decided right at the start. What will the end product look like generally? (Modern.) What features are definitely required? (Bar, pool table, salon, bathroom, two bedrooms, toy room.) This is like deciding the overall look and feel of your app and its required features.

Once you've got a handle on the overall look and feature set, it's time for the strategy of how to make it happen. How will you design it? You can design the whole project, creating a detailed blueprint before starting work, or wait to design the bathroom until you are ready to start work on it. But keep dependencies in mind. If you cover the main heating duct with drywall before you run a duct to the bathroom, you'll be redoing your ceiling. Likewise, if you write some Javascript that runs on IE then find out you've got some Linux users, you'll be right back into that code.

We ended up doing the basement much like I write software. Start with a good idea of what features must be in the end product, but save the details for later. Know what has to be done first, middle, and last, but break each part into its details when you come to it.

Test each part well before moving on to the next. You want to find out that one of the electrical outlets didn't get a wire run to it BEFORE you put drywall on the walls. And you want to know that your sql stored procedure works correctly before you start calling it from several different web apps.

For me the greatest lesson learned from finishing a basement is that everything doesn't have to be perfect. A wall stud can be a half inch off and you can still make it work. A heating duct can be a few inches too long or short and be fine. Likewise, sometimes the quick and dirty code is good enough. No function will ever be elegant enough and fast enough and memory efficient enough for me to consider it perfect. Put down the level and tape measure, nail the board into place and move on.

Test::Unit Wins in an Upset

Over the last couple of days I decided to write a small program without the benefit of Ruby's Test::Unit module. Why? Well, it's only been in the last year and a half or so that I've been rigorously testing code with frameworks like JUnit and Test::Unit, and I wanted reassure myself that I'm really better off now.

The thing is, even if you haven't created test cases, you still have to test your code. So my coding in the last two days has gone like this:

Make a change
Run the whole program (It worked, yea!)
Run it again with different input (Failed. Boo.)
Interpret the stack trace
Correct the error
Run the whole program again (It worked, yea!)
Run it again with different input (This time it worked, yea!)
Run it again with still another input (Oh, shit.)
Interpret the stack trace
Correct the error
Repeat over and over with every change and every kind of input

By the end of the project I was running the program seven times, to test seven different inputs, each time I made a change. Instead, I should have just run a unit test that tests only the module that was changed.

The time to run the whole program takes a few seconds longer than running a test case of a module. The time to run it seven times to test different inputs takes much longer than running a test suite. The time to type in the different inputs for each run of the program is far more than the time it would take to create the test cases. All these seconds really add up over several hours of coding.

Bottom line: I'm more convinced than ever that by taking the time to create good test cases I'm saving tons of time over the life of a project. Suite!

Email is Efail?

My take on Tantek's Thoughts http://tantek.com/log/2008/02.html#d19t2359.

While I agree there is great benefit to pushing a lot of communication to 1:many forms like blogs, twitter, etc., I find email still works for me in many respects. There are too many kinds of communication to pigeon-hole everything into just a few media.

Some things are best done over the phone, some over IM, some over 1:many media, and some over email.

A friend pointed out that email bloat is almost never a problem, and in fact lack of detail in emails is much more prevalent. I agree. It usually takes an email correspondence of several messages back and forth to get enough detail to proceed with the issue at hand.

Using a great email client, like Gmail, is also a great way to maximize email efficiency.

Flex 3 Pre-release Tour

Last night I attended the Flex and AIR Pre-release Tour hosted by the Salt Lake City Flex/Flash User Group. The presenter was Adobe Evangelist, Kevin Hoyt. He was very good and the meeting was very worthwhile. Hoyt knows his stuff and presents it well and with a sense of humor.

Watching Hoyt use Illustrator to create and style a button confirmed for me just how tedious developing user interfaces is, no matter what tools you use. After several minutes he ended up with a really nice looking button. But thinking about having to go through the same steps to put a skin on each interface widget made my head hurt. I'd rather debug kernel code. I really would. But that's just me.

But it got me thinking, why is the developer taking the time to stylize or skin the app anyway? Why don't developers just create the basic framework of the interface and let the users skin the app the way they want it to look? Well, of course that's not as easy as it sounds and some new tools would be required to make it simple and fun for users to do so. And maybe for another reason which I thought was Hoyt's most interesting idea.

Hoyt's idea, which I had never heard before, is about the way economies evolve. He explained that when the U.S. was young it had an agrarian economy. This evolved into a goods-based economy. Next came the service-based economy, where most 'products' in the economy are really services.

I had heard all this before, but then he said we are moving into an Experience-based economy.

You go to Starbucks and pay $5 not for a cup of coffee, but for the Experience of getting a Starbucks coffee. The smells, atmosphere, music, all that sort of thing. Or you go into an Apple store to get the gadget-buying Experience that you can't get at Best Buy.

I'm not sure I fully buy into the Experience-based economy idea. But, it makes more sense why a developer would spend so much time on the look-and-feel, the skin, of an application. You may want to provide a specific experience to the user. A memorable way of branding your product or company. I'll definitely be thinking more about the overall experience of the user as I develop applications from now on.

Oh yeah, and Flex 3 and Air look like good products to me. I think we made the right decision in our company to develop our next major app for the Air platform.

Speed Up MySQL Query Using Where Clause

This is going to sound obvious but to me it wasn't at first. The situation was, I wanted to see the top 1000 rows of a join on three tables, all of which had many thousands of records in them. The query did not finish in reasonable time until I added a where clause that didn't limit the result set in any way, but certainly sped up the response.

Here's an example to demonstrate the details.

My tables looked similar to this:

CREATE TABLE `Customer` (
`cId` int(11) NOT NULL auto_increment,
`cName` varchar(255) default NULL,
PRIMARY KEY (`cId`)
)

CREATE TABLE `PaymentBatch` (
`pbId` int(11) NOT NULL auto_increment,
`pbStatus` varchar(255) default NULL,
`pbCreatedDate` datetime default NULL,
`pbCompletedDate` datetime default NULL,
PRIMARY KEY (`pbId`)
)

CREATE TABLE `PaymentBatchDetail` (
`pbdId` int(11) NOT NULL auto_increment,
`pbdPaymentBatchId` int(11) NOT NULL,
`pbdCustomerId` int(11) NOT NULL,
`pbdReceivedDate` datetime default NULL,
`pbdAmount` decimal(12,3) default NULL,
PRIMARY KEY (`pbdId`)
)


And this was my first try at the query; I just wanted to see the most recent 1000 rows from the join of these tables:

select *
from Customer left join
(PaymentBatch left join PaymentBatchDetail on pbId = pbdPaymentBatchId)
on cId = pbdCustomerId
order by pbdReceivedDate desc
limit 1000;


Well, this took so long I killed it before it completed.

But adding a simple "where pbId > 0", which doesn't eliminate any rows from the query, made it run in under two seconds. Go figure.

Faster query:

select *
from Customer left join
(PaymentBatch left join PaymentBatchDetail on pbId = pbdPaymentBatchId)
on cId = pbdCustomerId
where pbId > 0
order by pbdReceivedDate desc
limit 1000;


Apache and Subversion

Well, nothing good comes easy. Here's a very abbreviated recipe for setting up Apache as a SVN front-end, listening on port 7382.

In your apache source directory:
./configure --prefix=/usr/local/apache_svn --enable-mods-shared=all --with-included-apr --with-port=7382
make
make install

Following is because subversion puts its apache modules in apache2 by default.
cd /usr/local/apache_svn/modules ; ln /usr/local/apache2/modules/*svn.so .

In /usr/local/apache_svn/conf/httpd.conf uncomment:
Include conf/extra/httpd-dav.conf in httpd.conf

Add to httpd.conf:
LoadModule dav_svn_module modules/mod_dav_svn.so

DAV svn
SVNPath /absolute/path/to/repository

User svn_user #Have apache run as a user with rw access to svn repository.
Group svn_user

Subversion Configuration

I'm setting up a Subversion repository. Hooking up Apache seems too complex right now, so that will come later. I'll set up svn+ssh for now.

Server setup (root)
svnadmin create /usr/local/svn/test
chown -R nucleus:nucleus /usr/local/svn/test
chmod g+w -R /usr/local/svn/test/

User setup (root)
Create each user account as needed, then
usermod -gnucleus -Gusers mike
[Repeat for each svn user]

Each user does their own setup (user)
Append your ssh public key to authorized_keys in your .ssh directory.
ssh-keygen -t dsa [if necessary]
scp ~/.ssh/id_dsa.pub nukson:.ssh/
ssh nukson "cat ~/.ssh/id_dsa.pub >> .ssh/authorized_keys"