Bcc

Failure, science and software testing

npdoty@gmail.com (npdoty) — Fri, 15 Jan 2010 01:39:00 +0000

Brian Houck (Microsoft), VigneshK (Microsoft), Mubarak (Microsoft), TracyMon (Microsoft), Jolie Boushey (Microsoft), Ben Cohen
http://npdoty.name/bcc

Have you guys read this Wired article on failure and science?

I thought it was really reminiscent of the constant failures we run into as programmers, and the particular challenge of being a software tester.

The main premise is that scientists get so comfortable with accepted theory and the status quo that they don't recognize that failures might be breakthroughs instead of just mistakes in their own equipment or experimental method. There's certainly some value there -- the author gives the example of sensitive radio telescope static finally being accepted as cosmic background radiation and not a problem with the dish, and cites some serious ethnographic research of scientists and how they make discoveries. But the article makes it sound like the solution is simply to be skeptical, to assume that every unexpected experimental result is a potential new discovery.

But anyone who's taken high school physics knows that this assumption that it must be your fault not the theory's fault isn't just some elitist fallacy, it's born of experience. Of the hundred times that the results of your high school physics experiment didn't match what theory predicts, how many times was it because the theory was wrong? Zero, of course; it's always a screw-up with your experimental set-up (at least it was in high school, and I bet the percentage doesn't change that much once you're a professional).

As programmers, we learn this lesson even more often. The first rule of programming, after all, is that it's always your fault. This isn't dogma, every one of us has learned it from this quintessential and eternally repeated experience where we write a piece of code, it doesn't work, we assume that it must be a problem with the operating system or the compiler or the other guy's code -- that the computer simply isn't doing what we told it to do -- until we realize the mundane truth when we actually look at our own code and the documentation and find that we'd just made another stupid mistake.

And that's the real trick of software testing: it can be tempting, particularly at first, to file a bug every time something doesn't work. Young confident software testers go to their dev several times a day saying "I found a bug" only to realize that they hadn't called the function with the correct parameters. But this lesson of experience quickly leads to the opposite problem: having become so accustomed to being the cause of our problems (like any programmer), we just fidget until we get the software to work, unconsciously working around bugs that we should be filing.

So I think the real answer, both to the scientific problem and the software-testing one, isn't mere undying skepticism, but in knowing which failures are probably your fault and which ones aren't. And a lot of the techniques that experimental scientists and software testers are the same: the first step for both is reproducing the failure. Lehrer's article also suggests talking to someone who isn't intimately familiar with the experiment, and I think we software testers often understand an unexpected result when we try to explain the bizarre situation to a tester from another team. "Encourage diversity" is also on his list, and I think the Test Apprentice Program at Microsoft was a darn good example of that in action -- being the only non-CS majors on our teams, we often found different bugs.

Maybe experimental science could even learn something from software testers. I thought one of the more valuable things we got from learning test-driven development was that a test wasn't good unless you'd seen it fail. If you've only ever seen a test pass, then how do you know that it really tests what you claim it tests? That must be harder for physicists (they can't briefly turn off a particular universal parameter to ensure that the experiment fails under those conditions), but the same sort of counterfactual thinking (rather than just writing a test and being happy when it turns green, or running an experiment and assuming that the result confirms the theory) seems important to me.

Do we get a lot of good software testers from experimental science backgrounds? Maybe that's where we should be hiring from. Anyway, I highly recommend the Wired article, if only for the comfort that programmers aren't alone in the universe for having their experiments fail constantly.

Hope you're all doing well -- grad school is great, but, as you can see, I still miss software testing from time to time,
Nick

rssCloud meetup Wednesday, September 9th

npdoty@gmail.com (npdoty) — Mon, 07 Sep 2009 00:37:00 +0000

announce@ischool, noise@ischool
Erik Wilde
http://npdoty.name/bcc

I'd like to announce a talk and meeting that may be of interest to iSchoolers.

On Wednesday, September 9th, local blogger Dave Winer, one of the originators of the RSS syndication format, will talk about a new proposal called rssCloud, which allows for Twitter-like instantaneous distribution of short messages in a decentralized way, based on existing RSS technology.

After an overview of that proposal, developers can share their own work in the area and there will be an open discussion.

So if you're interested in an alternative to Twitter that isn't controlled by a single company or want to see how a new standard is designed or implemented in practice, please join us Wednesday at 7 PM in 110 South Hall.

Thanks,
Nick

More info available on Dave's blog and the rssCloud website.

Last call and usage notification

npdoty@gmail.com (npdoty) — Sat, 08 Aug 2009 02:00:00 +0000

public-geolocation@w3.org
http://npdoty.name/bcc, Deirdre Mulligan

Hello all,

I had just a couple of my own comments to follow up on CDT's last call privacy comments and the "intended usage notification" thread that lingered and languished on this list a few months ago.

First of all, I'd like to second CDT's request to hear from other members of this list as to whether implementors of the API or users of the API that don't fulfill all the normative requirements in "Privacy considerations for implementors of the Geolocation API" and "Privacy considerations for recipients of location information" will be officially non-conformant with the API.

For example, Flickr's mobile website provides a "Photos taken nearby" feature which makes use of the draft Geolocation API. But Flickr apparently doesn't clearly and conspicuously disclose how long location data is retained, how location data is secured or whether location data is shared -- the "Your Privacy" link doesn't describe any uses or practices around location data. I might conclude from following another link that the "Yahoo! Privacy Policy" covers my location information, but it's never described explicitly and I couldn't definitively determine if my location information was stored or shared.

What does the WG intend by requiring recipients to "clearly and conspicuously disclose"? Is disclosure within a long Privacy Policy sufficient? Or do we expect location information to be addressed explicitly and before location information is requested? Also, will the W3C have any power to enforce or judge implementations or (ab)uses of the API?

Second (and I bring this up specifically because it might address ambiguities with the normative privacy considerations), I wasn't sure we ever came to a satisfactory conclusion on whether to allow requesters of location information to specify in their request how location information will be used, how long it will be kept or whether location information will be transmitted to 3rd parties. While Doug, Greg, Andrei and Ian proposed that allowing websites to present information about their usage would let them deceive users, Martin, Henning, Max and I thought that some additional context about how location information will be used would be valuable for user privacy.

Could we find some middle ground where requesters can't place arbitrary text which could deceive, but can fill in a timestamp for how long data will be kept and a flag for whether it will be shared? If not in V1, can we open an Issue to reconsider this question in V2? Again, this could help clarify ambiguities around "conspicuous disclosure", address concerns about privacy protection or even provide an easier step towards associating Geopriv-style permissions with location data.

Thanks,
Nick Doty
UC Berkeley School of Information

Twitter character counts

npdoty@gmail.com (npdoty) — Tue, 23 Jun 2009 17:22:00 +0000

Dave Winer
http://npdoty.name/bcc

Hi Dave,

Re my sarcastic tweet:

More seriously though, I think you're right on, but that really you're identifying problems with blogging, not with microblogging.

If it were just as easy to communicate with a blog post as it is with Twitter, plus you could express longer thoughts with a blog post, then there'd be no reason to complain about Twitter's character counts. Twitter would simply be a joke, an inferior product completely dominated (that is, in all dimensions) by blogging software.

But it isn't just as easy. Part of it, as many have pointed out, is the small messages -- it's easy to write and easy to read because neither takes that much commitment. But I think there are other issues too, advantages of Twitter that we wish we had with blog posts.

Replying on Twitter is way easier and more effective than replying with a blog post. Trackbacks are confusing, full of spam and much harder to use than a single character in front of a name.

People are harder to find at arbitrary addresses than they are with a single username after "twitter.com/". (Like @chrismessina and others, I wish this weren't the case, but currently, I believe it is.)

It's easier to be part of a trend just by typing a # and a word than tagging your blog post and hoping technorati picks it up.

Republishing content is a trivially easy and widely-accepted practice. (I think this is why single-click re-blogging on tumblr is so popular too and why Google Reader's "share" feature is so compelling.) You can also push content to particular people with @mentions.

Syndication and reading is handled in the same place as writing -- as soon as you sign up for one, you've signed up for the other. Even though Twitter syndication is inferior to RSS (Twitter is a single unreliable service; there's no tracking of what's read and unread across devices), I suspect more regular people use Twitter to keep track of all their friends than use an RSS aggregator.

I think it's not so much a problem that Twitter has a character limit as it is that blogging platforms don't have all these other advantages. Conversations happen easily and naturally on Twitter, despite the severe limitation of character counts. I'd love to see those same advantages in the blogging platforms we use every day -- something you know about first hand!

Anyway, thanks for starting the conversation, and for using both Twitter and your blog to do it.

Nick

re-blogging is the next big thing

npdoty@gmail.com (npdoty) — Wed, 20 May 2009 05:17:00 +0000

Jessamyn Conell-Price
http://npdoty.name/bcc

Want to know the next big thing? After Twitter and location-based services, it's going to be re-blogging.

Have you been reading your little sister's re-blog? Or her friends' re-blogs?

By allowing re-publishing and commenting with as little as a single click, Tumblr is letting people feel ownership from what is largely curation. I do it a little with delicious and Google Reader, but these girls are doing it voraciously with re-blogging of photos and quotes.

And why not? Although I think technically this is done pretty poorly (copies of things leads to all sorts of problems and I already have a lot of issues with Lynn's friends and trying to track down the original of anything, since it may be several links deep), but the basic idea is smart. Everyone should be out there curating the web, not just a couple of big-name bloggers like Kottke. Add your comments and then (once the technical challenges are solved) compare your comments to everyone else's. Start a conversation on top of the web instead of in it.

Paragraph fingerprinting (CMU) and quotation finding (Google) are some techniques to bolt this on to existing practices. That might be more pragmatic than the Xanadu-style alternative. But I think an enhanced Delicious is where it's at (URLs as actual universal resource locators) -- it's just a question of who gets there first.

Re: My first step towards digital exhibitionism?

npdoty@gmail.com (npdoty) — Fri, 27 Mar 2009 01:51:00 +0000

Sam
Jessamyn, Seth, Ryan, Shawna, Zeina, Brooks, Steph, Andreas
http://npdoty.name/bcc

(Looping back in the digital exhibitionists, in case they have input here.)

Regarding exhibitionism and subjectivity: I'm not sure there's any way around the fact that I control this. Since it's on a web page that I control, I don't see how I could prove to you that it's automatic or genuine, even if it really were. Short of a government-implanted chip, I think there's no way to stop me from potentially lying to you about my location, and if it ever got to the point where I couldn't lie about it, hide my location at certain times, I'd be really unhappy.

But I think I see your point, that there is a difference in degree here. The more automatic the updates are (even if I have the power to turn them off, or distort them), the more realistic the image of myself is portrayed. The more I have to remember to update, choose to only in certain circumstances reveal my location, the more my persona is curated.

There's no choice but for my online persona to be curated, the same way that my "real life" persona is. But the more automatic and implicit I can make these updates, the more realistic (and richer) a persona I can present. That seems like a worthy goal -- I'll work on getting updates to happen more automatically, and on building the habit to press that button each time I look at my phone. And maybe I can document on my page when my location was last updated -- it's not real proof, but it would be a start.

On Feb 23, 2009, at 2:51 PM, Sam Maurer wrote:

Maybe you could make a useful distinction between active and passive engagement with the information? If I have a routine that involves potentially being in the same location as you with any regularity, then I will want pull access to the information. But if I live far away and am just casually intrigued, either because I like to know what my friends are up to (c.f. facebook news feed), or because I have a thing for geospatial information, then I will want push notification of your major location changes. I guess people who live near you could want a combination of active and passive engagement with the information, but people who live far away are more likely to just want passive engagement?

I'm a little bit worried that the updates aren't automatic, though! I think this eliminates a lot of the digital exhibitionism component, because you might start subjectively tweaking your claimed location. And there's nothing to stop someone from using this as just another aspect of a carefully curated online persona. Thoughts?

sam

On Mon, Feb 23, 2009 at 5:10 PM, Nick Doty wrote:

I think since Fire Eagle currently doesn't give access to history I can't write code to do this (compare the current location to a past location). I'm also not sure that Fire Eagle supports notifications like that, though maybe XMPP allows for this. Seth?

But would you want to be notified every time my location changes significantly? I would think my friends would want more of a pull question than a push one: "where is Nick right now?" rather than "let me know whenever Nick moves". The latter also seems a little "creepier", though I'm not completely sure why.

On Feb 23, 2009, at 1:59 PM, Jessamyn Conell-Price wrote:

Can I be notified every time your location changes significantly*?

*standard for significant change to be determined

On Mon, Feb 23, 2009 at 1:54 PM, Nick Doty wrote:

My first step towards digital exhibitionism?

npdoty@gmail.com (npdoty) — Sun, 01 Mar 2009 08:43:00 +0000

Jessamyn, Nathan, Seth, Ryan, Shawna, Sam, Zeina, Brooks, Steph, Andreas
http://npdoty.name/bcc

Friends,

Using the wonders of Fire Eagle (thanks Seth!), and a Google Geo Developer Workshop (thanks iSchool!), I've added a map to my personal website (http://npdoty.name) which is centered around my exact location, as stored in Fire Eagle.

So if you're curious, you can see roughly where I am, any time you have access to a web browser. Go ahead, stalk away. Try it out, let me know what you think.

Of course, updating Fire Eagle is something I have to do explicitly from my phone or laptop, so it won't be as automatic as the way that Andreas does it (though I'm looking at Loki to try to do it automatically via SkyHook). And the way the page is laid out, my exact location is visually obscured -- though those of you comfortable with code should be able to find my latitude and longitude without any trouble.

Nevertheless, I've never made this sort of information public before, so I'm curious to know how you'll use it, whether it borders on digital exhibitionism, if you think it'll lead to my untimely death, etc. So far, I like it, just because the maps are pretty and it provides some context to a webpage about me. If that isn't pretentious and self-aggrandizing, I don't know what is.

Nick

Institutional bugs

npdoty@gmail.com (npdoty) — Sun, 01 Feb 2009 02:23:00 +0000

Brian _____, <_____@microsoft.com>; Jon _____, <_____@google.com>
Mubarak, Bob, TracyMon, Vignesh, NickL
http://npdoty.name/bcc

Gentlemen,

It's pretty exciting to see software testing come so prominently into the news twice in such a short time frame. I know that neither of you can share any of the internal discussion you've heard on these topics, but I sure would have enjoyed watching the threads these events sparked. Are these issues getting talked about a lot outside of the groups immediately impacted?

Really, I've been able to see quite a bit just looking in from the outside. It's pretty neat to be able to see the actual source code of the Zune leap year bug and hear the exact wildcard problem in this morning's Google badware bug -- it makes me feel like I'm not so far away from the industry after all. (Which isn't to say there isn't some advantage from knowing some people on the inside: it was fun when I was at Microsoft last month to hear about how our friend on the Zune test team got a call at 7 AM on a day when most people weren't expected at work telling him he needed to be in the office immediately. That must have been a pretty intense day.)

I've heard conjecture (fueled by the short-lived rumor that StopBadware was somehow responsible rather than Google itself) that the mistake happened because Google got an updated list from StopBadware and just checked it in verbatim, rather than Google mistakenly adding the wildcard in itself.

And it's similar to the discussion I saw around the Zune leapyear issue. Speculation raged about how a Microsoft developer could make such a mistake or how the Zune test team could miss it. Then when it came out that it was actually a bug in Freescale Semiconductor's code, suddenly it made sense to everyone: only the Zune 30 had the problem, none of the newer Zunes have that problem because they no longer rely on a third-party vendor's code. And more significantly, it wasn't that Microsoft developed code with such a glaring hole. Or that Google deployed a file with such an obvious error. It's as if we're comforted by thinking that Google and Microsoft weren't the responsible entities; that at least fits with our understanding of these software companies.

But neither of those explanations helps the Google customer or the Zune customer, nor should they be any solace to them. Microsoft and Google are just as responsible for code they ship that was originally written outside the company. And really, if anything, it's an opportunity for a Microsoft SDET and a Google QA engineer to get a promotion.

Sure, whatever Google engineer checked in the file should be getting a talking to: wouldn't a single manual test have caught the issue? When you're making a change to code that'll be run as part of every Google search, shouldn't you at least have tested it once yourself? But it's much more an issue of why there wasn't an automated check-in test that prevented the change from going in at all. A single negative automated test case would have caught this and relying on all your individual engineers to never make mistakes like this is foolish.

Also, I happen to think that the Zune leapyear bug should have been caught by a developer's unit tests: shouldn't a unit test for a piece of leap year code include a case for the end of a leap year? But a Microsoft SDET could make some significant improvements for his product by proposing a policy to do code reviews of partner code. Collaborations are inevitable, and it would be worse for the company to have the already frustrating Not-Invented-Here syndrome institutionalized as an official company practice under the name of quality assurance. Test plans and code reviews are just as valuable for partner code as for code written internally.

Of course I know that neither of you can speak for either company any more than any single person could represent such a huge group of people, practices and institutions. For that matter, I have faith that both the Zune team and the Google Search team have already come to these conclusions and implemented something along these lines. But I'm curious what your thoughts are, since you might be able to bring this idea up as a reminder in your group and in the next group over and that maybe we can all have a little more discussion about it. And that's exactly the point: we expect Google to not make mistakes like this because we expect such a powerful single entity to be so consistent. But Google isn't such a single entity -- any one engineer will make mistakes and any one partner will be unreliable. But since Google the institution is so powerful, it can be as perfect as we expect, not by being a single infallible entity, but by putting practices in place -- like a culture of quality assurance and a system of unit and check-in testing. In both of these high profile cases, the issues were institutional bugs, not code defects.

Perhaps that's all obvious to you guys; to someone just looking back on the software business, it seemed important.

Anyway, hope you're doing well and that you're enjoying software. Grad school is pretty great, but I miss being more intimately involved.
Nick

Re: programmatic typesetting

npdoty@gmail.com (npdoty) — Thu, 22 Jan 2009 09:21:00 +0000

Sam Maurer
Timothy Paige
http://npdoty.name/bcc

Well, I guess the idea was that it wouldn't need any editing (or only to fix programmatic problems). That raises the question about what the true purpose of the project would be, but at least partially for me it's a statement about our communications -- that they can often be trivial, that the amount is huge, that the pieces are intermixed and formatted identically in my email client despite having such different characters or topics.

This distinction about how email has such a wide variety of topics and styles actually got me started thinking about another idea. What if I wrote a blog that was done in the format of emails that I sent to various people? So each post would be an email (like some of the more significant ones I send to you, or Tim O'Reilly, or friends at Microsoft, or whoever), complete with headers. I really like the idea of blog entries that have more metadata than just a title (who I chose to send to and CC and so on; the content of the email I'm responding to, etc.). Also, it harkens back to publishing an important person's letters as a journal of his life. (http://npdoty.name/bcc, say.)

On Jan 20, 2009, at 11:52 AM, Sam Maurer wrote:

Won't you need to edit the content quite a bit, as well as formatting it? I don't mean that you'll change what you wrote, but you'll need to pick which emails and plans to include, and in what order. Even if you want nearly everything, and you want it chronologically, maybe different themes of writing should be formatted differently, so that personal correspondence stands out from the ideas about information management or about liberal arts education. This part would take even longer than the formatting, right? But you could probably combine the editing with the difficult-to-automate parts of the typesetting, and do both at the same time without much added cost. (Also, this editing process may well be even more enjoyable than reading the finished product, since you'll have to engage with your old ideas in order to organize them.) As long as you do some simple pre-processing to separate entries clearly, format email headers properly, and so on, manual typesetting won't be that hard. You can just define some style sheets and then hit F-keys in BBEdit or InDesign as you go through the content.

I know that I'm subverting your intention to automate the process, but I think this would be a more pragmatic solution!

Sam