For the past 20 years, the State of Hawaii has pursued the development of a high tech industry to counterbalance our economic dependence on tourism, with mixed results. Proponents of software-based companies note that these products are environmentally friendly, can be developed in any location, and bring in highly-skilled, highly paid jobs to the State.
These advantages are all true, and there are some notable high tech success stories. Unfortunately, while Hawaii is well-suited to high tech, high tech may not be well suited to Hawaii. The location-independence of software development means that it is just as easy to migrate high tech jobs out of Hawaii as it is to move them here, as is demonstrated by the frequent relocation of locally started high tech companies to California.
The problem is that the "value proposition" for locating the typical high tech company in Hawaii is weak: our cost of living is higher than the mainland, we are isolated and at least a five hour plane ride away from other companies, the number of qualified high tech professionals here is limited, and the physical geography of Hawaii does not provide a competitive advantage.
As a State and community, we now have an incredible opportunity before us: the development of a new industry that can provide an alternative to tourism, and for which all of the traditional disadvantages of Hawaii suddenly become advantages. That industry is renewable energy. Here are some of the compelling value propositions for this industry in Hawaii:
1. Our geography is an advantage: Hawaii is "world class" with respect to its renewable energy resources. There is no other single place on Earth with Hawaii's simultaneous availability of wind, wave, geothermal, and solar energy resources. That means we can work on multiple fronts, and explore complementary combinations of renewable energy. It also means that renewable energy companies started in Hawaii will tend to stay in Hawaii: there is a geographic disadvantage to moving them elsewhere.
2. Our high cost of living is an advantage: it makes it easier to make renewable energy economically viable. We currently pay $0.25 per kWh, almost twice the cost on the mainland. Furthermore, that cost can rise dramatically with increases in oil prices. This means that alternative energies become cost-effective in Hawaii much sooner than on the mainland, making it easier to start businesses in renewable energy in Hawaii.
3. Our isolation is an advantage: our energy grid is autonomous. On the mainland, all of the electrical grids are interconnected and many times larger than Hawaii's. The fact that our grid is small and isolated makes us better suited to innovation; we provide a natural "laboratory" for experimentation with renewable energy sources.
4. The small number of high tech professionals in Hawaii is not a disadvantage: Renewable energy jobs are not just located in cubicles. Unlike high tech software jobs, renewable energy jobs span the gamut from "high tech" engineering and business to "traditional tech" such as carpentry, electrical, and plumbing. A renewable energy industry creates jobs across the socio-economic spectrum of Hawaii.
5. A renewable energy industry creates a "virtuous circle" of economic development. The development of a renewable energy industry has a singularly positive effect on our economy for one simple but profound reason: every kilowatt-hour of energy created by local, renewable energy sources is one less kilowatt-hour of energy we pay for with foreign oil. In addition to the positive environmental consequences, this means that every dollar generated by renewable energy is a dollar kept in Hawaii and not exported elsewhere. Currently, out of our $60B gross domestic product, almost $8B is "bled away" to pay for foreign oil. Returning almost 15% of our GDP to Hawaii could enable us to improve government services while reducing our tax burden. Renewable energy provides an unparalleled potential for economic development as it can simultaneously create jobs and reduce the flow of money away from our islands.
Creating a renewable energy industry in Hawaii requires vision and leadership from our political representatives and our educational institutions, but it is possible. In general, we must create a legal and regulatory framework that enables energy innovation and provide workforce training for those who wish to pursue careers in this area.
Here are concrete steps we can take, starting today:
1. Ask the political candidates how they will further a renewable energy industry in Hawaii. This election season provides an opportunity to raise the profile of this issue.
2. Lobby your representative for new laws to make renewable energy more affordable. For example, PACE (Property Assessed Clean Energy) financing attaches the cost of solar energy installations to your property taxes. Essentially, your property taxes go up, but that increase is offset by your savings in energy, and if you sell your house, the remainder of the "loan" is paid off by the next owner.
3. If you are a student, investigate renewable energy programs at your school. For example, the UH College of Engineering was recently awarded a $2.4M work force training grant by the Department of Energy to support renewable energy education. Also at UH, the Sustainable UH student group provides a variety of educational opportunities related to renewable energy. The more educated we are, the better we will be positioned to take advantage of opportunities as they occur.
Hawaii is uniquely positioned to be a world leader in renewable energy, with the potential for incredible benefits to us personally and to the wider world. Let's work together to make it reality.
Tuesday, August 24, 2010
Tuesday, February 9, 2010
Soliciting student interns from the UH ICS Department
Several times per semester, local companies contact me to ask if I know of any good students who might be interested in working with them on a project. I am always delighted to receive these emails and want to facilitate these kinds of interactions. In general, even if I happen to know of a student, I will always suggest that they send me a short email that I can resend to our internal student mailing lists. With hundreds of students in our department, there may well be an ideal candidate who I have not had the opportunity to get to know personally.
Here are some hints that you can use to help maximize your chances of connecting with a good candidate:
Here are some hints that you can use to help maximize your chances of connecting with a good candidate:
- Note that your email is unlikely to be the first solicitation our students have received this year. Indeed, your email may not be the first solicitation our students have received this month, or even this week. It is helpful to point out what makes your opportunity special beyond being just a job.
- Our students tend to be busy. Really busy. Most already have part-time jobs in addition to a full academic load, and many are juggling a full-time job with a full-time load. Naturally, pursuing new opportunities requires yet more time and energy, and switching from a currently stable employment situation to a new, unknown situation has real risks for our students. Help them to see the rewards that might come from pursuing your opportunity.
- In your email, the more details you can provide up front, the higher the chances that good students will respond. In addition to the overall intellectual/professional opportunity, students are very interested in logistics. What is the pay? What are the hours, and what level of flexibility is available? Will the student need to work with you on-site, and where is that? Are there citizenship issues? What technical background are you hoping for? What technical skills will the student acquire? Will the student work alone or as part of a team? Could this develop into summer job, or full-time work after graduation?
- While you might be tempted to create a Word document with this information and attach it, resist. To minimize the "barrier to entry", describe your offering as plain text in two to three paragraphs directly in the body of the email. It is always appropriate to provide URLs to further information on your company website.
Wednesday, November 18, 2009
BECC 2009: Day 3
The last day of the 2009 BECC Conference is just a half day, so I attended a session on energy competitions and a keynote regarding energy efficiency in the US economy.
The energy competition session kicked off with a presentation on Oregon's Home Energy Makeover competition. Last year, 4 homes were picked out of 6,000 entries. The presenters talked not just about the winners, but also how important it is to provide ongoing support to the losers, many of whom become motivated to go ahead and do energy improvements on their own dime. For example, the sponsoring organization (Energy Trust) has created a site to evaluate the energy performance of their home. I was reminded quite happily of the Blue Planet Foundation's Hawaii Energy Home Makeover.
The second presentation took it up a notch with the Cool School Challenge. This program is designed for high school students: they perform an energy audit of their school (involving electricity, solid waste, transportation, and heating/cooling), set goals for CO2 emission decreases, track progress, and report results. The challenge website has curriculum materials, spreadsheets that do the carbon calculations, and reports on the various schools participating in the program. It's won national and international awards (they showed a picture of high school students at the White House) and has been used in many states. The program is designed to achieve four goals: (1) reduce carbon emissions; (2) encourage student leadership and empowerment; (3) foster a community of teachers and students; and (4) educate young people and their families. What's more, they've provided a pathway for students to take the knowledge they gained in the classroom and put it into use in the community via a related program called EcoOffice, in which students do a similar energy audit of local businesses and help them determine changes. Wow!
But wait: there's more. The third presentation was on the Energy Smackdown. The founder of this program began by asking the question: what if we could get people as excited about energy savings as they are about football and soccer? Or American Idol? The smackdown began as an energy competition between three households, and has more recently scaled up to a competition between three communities (I recall Medford, MA vs. Cambridge, MA; I forget the other one). He showed some great previews of a reality television show based upon the competition to be released in early 2010. In one scene, folks from Cambridge sneak into a hardware store and buy all the CFL bulbs, then leave chortling about how Medford is going to totally go down. Kailua vs. Manoa? Kaimuki vs. Aiea? This could be so fun in Hawaii.
The day (and the conference) ended with a keynote by Hannah Choi Granade on a report by her organization (McKinsey and Associates) called "Unlocking Energy Efficiency in the US Economy". The report indicates that we as a nation waste an unbelievable amount of energy, so much that even implementation of relatively conservative efficiency measures could reduce energy consumption by 9.1 quadrillion BTUs by 2020, roughly 23% of demand, abating approximately 1.1 gigatons of greenhouse gases. Indeed, we as a nation could actually decrease our aggregate energy use.
That's the good news: the bad news is that there are "staggering" barriers to implementation. What's weird is that these barriers aren't typically "rationale": a straightforward economic analysis would lead most people and businesses to implement the measures. The rather dense report goes into detail on the barriers and ways to approach them. One thing I found interesting is that the authors could not apply the "80/20" rule to energy efficiency: in other words, there were not a small number of actions that would produce a disproportionate amount of benefit. Instead, we need to take a large number of small actions, and these actions cut across residential and industrial settings and all segments of society.
At the conclusion of the conference, one of the organizers took the microphone and asked us to take off our lanyards and leave them on the tables in front of us. They collected them after we left and will reuse them at next year's BECC conference in Sacramento. I'm going to do my best to be there.
Back to Day 2 or Day 1.
The energy competition session kicked off with a presentation on Oregon's Home Energy Makeover competition. Last year, 4 homes were picked out of 6,000 entries. The presenters talked not just about the winners, but also how important it is to provide ongoing support to the losers, many of whom become motivated to go ahead and do energy improvements on their own dime. For example, the sponsoring organization (Energy Trust) has created a site to evaluate the energy performance of their home. I was reminded quite happily of the Blue Planet Foundation's Hawaii Energy Home Makeover.
The second presentation took it up a notch with the Cool School Challenge. This program is designed for high school students: they perform an energy audit of their school (involving electricity, solid waste, transportation, and heating/cooling), set goals for CO2 emission decreases, track progress, and report results. The challenge website has curriculum materials, spreadsheets that do the carbon calculations, and reports on the various schools participating in the program. It's won national and international awards (they showed a picture of high school students at the White House) and has been used in many states. The program is designed to achieve four goals: (1) reduce carbon emissions; (2) encourage student leadership and empowerment; (3) foster a community of teachers and students; and (4) educate young people and their families. What's more, they've provided a pathway for students to take the knowledge they gained in the classroom and put it into use in the community via a related program called EcoOffice, in which students do a similar energy audit of local businesses and help them determine changes. Wow!
But wait: there's more. The third presentation was on the Energy Smackdown. The founder of this program began by asking the question: what if we could get people as excited about energy savings as they are about football and soccer? Or American Idol? The smackdown began as an energy competition between three households, and has more recently scaled up to a competition between three communities (I recall Medford, MA vs. Cambridge, MA; I forget the other one). He showed some great previews of a reality television show based upon the competition to be released in early 2010. In one scene, folks from Cambridge sneak into a hardware store and buy all the CFL bulbs, then leave chortling about how Medford is going to totally go down. Kailua vs. Manoa? Kaimuki vs. Aiea? This could be so fun in Hawaii.
The day (and the conference) ended with a keynote by Hannah Choi Granade on a report by her organization (McKinsey and Associates) called "Unlocking Energy Efficiency in the US Economy". The report indicates that we as a nation waste an unbelievable amount of energy, so much that even implementation of relatively conservative efficiency measures could reduce energy consumption by 9.1 quadrillion BTUs by 2020, roughly 23% of demand, abating approximately 1.1 gigatons of greenhouse gases. Indeed, we as a nation could actually decrease our aggregate energy use.
That's the good news: the bad news is that there are "staggering" barriers to implementation. What's weird is that these barriers aren't typically "rationale": a straightforward economic analysis would lead most people and businesses to implement the measures. The rather dense report goes into detail on the barriers and ways to approach them. One thing I found interesting is that the authors could not apply the "80/20" rule to energy efficiency: in other words, there were not a small number of actions that would produce a disproportionate amount of benefit. Instead, we need to take a large number of small actions, and these actions cut across residential and industrial settings and all segments of society.
At the conclusion of the conference, one of the organizers took the microphone and asked us to take off our lanyards and leave them on the tables in front of us. They collected them after we left and will reuse them at next year's BECC conference in Sacramento. I'm going to do my best to be there.
Back to Day 2 or Day 1.
BECC 2009: Day 2
The second day of this year's Behavior, Energy, and Climate Change conference proved just as interesting as my experiences during the first day.
I began the day by attending a talk by Barbara Farhar from the University of Colorado who presented some initial results from a qualitative study on Boulder residents and their views of the smart grid. Of interest to me was the fact that she developed an energy literacy assessment instrument for the research, and that only (approximately) half of the respondents indicated that they would support demand response in their home (i.e. allow the utility to turn off their dryer or hot water heater). While her sample size is small and not statistically generalizable, this is still an interesting datapoint.
I also saw a talk by Scott Pigg by the Energy Center of Wisconsin dealing with energy behaviors, and a very interesting talk by Hunt Allcott on ways to introduce randomization into energy/behavior research in order to obtain results with both internal and external validity.
The lunch speaker was Doug McKenzie-Mohr who gave a talk related to his book and website "Fostering Sustainable Behavior". He views much of the current energy/behavior applications as "information intensive programs", in which the goal is to simply provide novel forms of information about energy and hope that behavioral change results. He related a case study by Galler on an energy efficiency workshop that illustrates how this approach can fail in a big way.
In contrast, he advocates a more structured approach called "Community based social marketing" with a five step process that can be simplified as: (1) Select behavior; (2) Uncover barriers and benefits; (3) Develop a strategy; (4) Pilot the strategy; and (5) Implement broadly and evaluate. The key insight to me is (2), that without uncovering the barriers/benefits, it is hard to design a strategy (intervention) that is effective.
He illustrated the approach by describing a program in Canada to reduce auto idling time. An initial strategy was to erect signs requesting that drivers not idle their cars (for the environment or for the children). It didn't work, since it did not address two barriers: (1) drivers think that keeping their car engine running is more efficient when the idle period is less than 4-5 minutes (the truth is 10 seconds) and (2) the increased number of car starts will burn out the starter (they won't). So, the more effective strategy was to have a human talk to the car driver to disabuse them of these idling myths, along with a sticker they could put in their windshield to remind them (and others) not to idle. A summary of the project provides more details.
After lunch I attended a talk by John Peterson from Oberlin College. He spoke about his efforts to create "ecological feedback" systems for energy use in buildings, which he says involves the following key features: (1) near real-time; (2) socially comparative within monitored entities and among groups; (3) empathetically connects decision making to nature, community, and future generations; and (4) engaging, entertaining, etc. He illustrated this with his work on the Campus Resource Monitoring System at Oberlin college, with the various dorm energy competitions they've done, and more recent work with the Great Lakes Protection Fund project.
The Campus Resource Monitoring System provides a nice user interface to energy information by the Lucid Design Group. For a good article that overviews many of the current commercial energy dashboard offerings, I recommend the article "Visualizing Building Information" that appears in the Winter 2009 issue of Centerline, the newsletter of the Center for the Built Environment at UC Berkeley.
I then ran over to another session and caught the end of a talk by Jon Froehlich of the University of Washington, who described mechanisms developed by his research group to unobtrusively identify energy use at the appliance level by applying pattern/spectrum analysis techniques. The basic idea is that turning appliances on and off produces "noise" in the power lines for a house which can be detected via some sensors and since different appliances have different signatures, you can disambiguate which appliance has been turned on/off and thus is responsible for the increment in energy consumption. What's pretty cool is that they've gone beyond electrical use to apply this same general technique to water use and natural gas use.
After a few more talks which I won't summarize here, I attended a "Film Festival" session presented by Bill LeBlanc of E-Source. He showed short clips (mostly ads, mostly humorous) regarding energy-related topics, but the best were a series of "PowerWalking" videos he has been doing for several years. To get a taste, here's the PowerWalking 2007 video. We watched the world premier of the 2009 PowerWalking video, which he says will be on the E-Source website eventually.
The day (well, evening) concluded with a poster session. There were lots of good ones, but one I found particularly relevant was by Sam Borgeson and Omar Khan on the Berkeley Campus Dashboard. (Here's a link to their 2008 poster session abstract.) This is intended to be an open, web-based system for displaying energy data that has much in common philosophically with our WattDepot project.
On to Day 3 or back to Day 1.
I began the day by attending a talk by Barbara Farhar from the University of Colorado who presented some initial results from a qualitative study on Boulder residents and their views of the smart grid. Of interest to me was the fact that she developed an energy literacy assessment instrument for the research, and that only (approximately) half of the respondents indicated that they would support demand response in their home (i.e. allow the utility to turn off their dryer or hot water heater). While her sample size is small and not statistically generalizable, this is still an interesting datapoint.
I also saw a talk by Scott Pigg by the Energy Center of Wisconsin dealing with energy behaviors, and a very interesting talk by Hunt Allcott on ways to introduce randomization into energy/behavior research in order to obtain results with both internal and external validity.
The lunch speaker was Doug McKenzie-Mohr who gave a talk related to his book and website "Fostering Sustainable Behavior". He views much of the current energy/behavior applications as "information intensive programs", in which the goal is to simply provide novel forms of information about energy and hope that behavioral change results. He related a case study by Galler on an energy efficiency workshop that illustrates how this approach can fail in a big way.
In contrast, he advocates a more structured approach called "Community based social marketing" with a five step process that can be simplified as: (1) Select behavior; (2) Uncover barriers and benefits; (3) Develop a strategy; (4) Pilot the strategy; and (5) Implement broadly and evaluate. The key insight to me is (2), that without uncovering the barriers/benefits, it is hard to design a strategy (intervention) that is effective.
He illustrated the approach by describing a program in Canada to reduce auto idling time. An initial strategy was to erect signs requesting that drivers not idle their cars (for the environment or for the children). It didn't work, since it did not address two barriers: (1) drivers think that keeping their car engine running is more efficient when the idle period is less than 4-5 minutes (the truth is 10 seconds) and (2) the increased number of car starts will burn out the starter (they won't). So, the more effective strategy was to have a human talk to the car driver to disabuse them of these idling myths, along with a sticker they could put in their windshield to remind them (and others) not to idle. A summary of the project provides more details.
After lunch I attended a talk by John Peterson from Oberlin College. He spoke about his efforts to create "ecological feedback" systems for energy use in buildings, which he says involves the following key features: (1) near real-time; (2) socially comparative within monitored entities and among groups; (3) empathetically connects decision making to nature, community, and future generations; and (4) engaging, entertaining, etc. He illustrated this with his work on the Campus Resource Monitoring System at Oberlin college, with the various dorm energy competitions they've done, and more recent work with the Great Lakes Protection Fund project.
The Campus Resource Monitoring System provides a nice user interface to energy information by the Lucid Design Group. For a good article that overviews many of the current commercial energy dashboard offerings, I recommend the article "Visualizing Building Information" that appears in the Winter 2009 issue of Centerline, the newsletter of the Center for the Built Environment at UC Berkeley.
I then ran over to another session and caught the end of a talk by Jon Froehlich of the University of Washington, who described mechanisms developed by his research group to unobtrusively identify energy use at the appliance level by applying pattern/spectrum analysis techniques. The basic idea is that turning appliances on and off produces "noise" in the power lines for a house which can be detected via some sensors and since different appliances have different signatures, you can disambiguate which appliance has been turned on/off and thus is responsible for the increment in energy consumption. What's pretty cool is that they've gone beyond electrical use to apply this same general technique to water use and natural gas use.
After a few more talks which I won't summarize here, I attended a "Film Festival" session presented by Bill LeBlanc of E-Source. He showed short clips (mostly ads, mostly humorous) regarding energy-related topics, but the best were a series of "PowerWalking" videos he has been doing for several years. To get a taste, here's the PowerWalking 2007 video. We watched the world premier of the 2009 PowerWalking video, which he says will be on the E-Source website eventually.
The day (well, evening) concluded with a poster session. There were lots of good ones, but one I found particularly relevant was by Sam Borgeson and Omar Khan on the Berkeley Campus Dashboard. (Here's a link to their 2008 poster session abstract.) This is intended to be an open, web-based system for displaying energy data that has much in common philosophically with our WattDepot project.
On to Day 3 or back to Day 1.
Tuesday, November 17, 2009
BECC 2009: Day 1
This week I am attending the 2009 Behavior, Energy, and Climate Change conference in Washington, D.C. This is the third year of the conference, and the first time I've attended it. The conference has grown from around 250 people the first year to around 700 this year, and it shows: some of the tracks yesterday were so popular that you had to stand outside the room and listen.
Due to travel arrangements, I missed most of the first set of concurrent sessions, and so the first talk that I attended in its entirety was the lunch speaker: US Representative Brian Baird, who is the Chairman of the Subcommittee on Energy and Environment. He is a former clinical psychologist and so is very open to the idea that behavior is an important issue when thinking about energy use and climate change. (This seems like a no brainer, though the political right is accusing him of mind control.) He had some interesting, low cost ideas on how to improve energy efficiency, such as to require that MLS listings of houses for sale should state the energy usage of the house (electricity and gas) over the past year. This would incentivize home owners to improve the energy efficiency of their home (and use energy wisely), as it would make their house more attractive on the market, and enable home buyers to choose between two houses by picking the one with lower potential energy usage (all other things being equal.)
The afternoon had six concurrent sessions, of which two (Behavior Research and Policy Agenda; Technology Design) were of interest to me. This appears to be a happy problem that I will struggle with for the remainder of the conference: there are always multiple places I want to be at once. I chose the first session and listened to Ed Vine from the California Institute for Energy and the Environment give a review of nine papers sponsored by his organization for the California Public Utilities Commission. They covered a lot of ground, from experimental design to process issues to market segmentation to behavioral issues. You can get links to these and other related papers here.
The day concluded with a panel session on Smart Grid, with Matthew Trevithick from Venrock (a VC firm), Gregory Abowd from Georgia Tech, Omar Khan from Google's PowerMeter program, and Carrie Armel from Stanford's Precourt Energy Efficiency Center. (The Precourt Center website has an excellent section devoted to behavior with an 800+ citation database.)
Each panelist gave a very short presentation giving their perspective on smart grid and behavior, followed by lots of Q&A with the audience. One of the many interested issues touched upon was an informal "stages of use" observed by the Google Powermeter team: users begin by learning about the system, then they engage with it to change their use, then they finally move into a maintenance phase. The learning/engagement phases appear to last only a week or two, where users are actively manipulating the system and discovering issues with their usage and making concrete changes. During this phase, they want to be able to drill down, do experiments, and so forth. In the maintenance phase, they basically only want the system to tell them if something has changed in their energy usage.
On to Day Two or Day Three.
Due to travel arrangements, I missed most of the first set of concurrent sessions, and so the first talk that I attended in its entirety was the lunch speaker: US Representative Brian Baird, who is the Chairman of the Subcommittee on Energy and Environment. He is a former clinical psychologist and so is very open to the idea that behavior is an important issue when thinking about energy use and climate change. (This seems like a no brainer, though the political right is accusing him of mind control.) He had some interesting, low cost ideas on how to improve energy efficiency, such as to require that MLS listings of houses for sale should state the energy usage of the house (electricity and gas) over the past year. This would incentivize home owners to improve the energy efficiency of their home (and use energy wisely), as it would make their house more attractive on the market, and enable home buyers to choose between two houses by picking the one with lower potential energy usage (all other things being equal.)
The afternoon had six concurrent sessions, of which two (Behavior Research and Policy Agenda; Technology Design) were of interest to me. This appears to be a happy problem that I will struggle with for the remainder of the conference: there are always multiple places I want to be at once. I chose the first session and listened to Ed Vine from the California Institute for Energy and the Environment give a review of nine papers sponsored by his organization for the California Public Utilities Commission. They covered a lot of ground, from experimental design to process issues to market segmentation to behavioral issues. You can get links to these and other related papers here.
The day concluded with a panel session on Smart Grid, with Matthew Trevithick from Venrock (a VC firm), Gregory Abowd from Georgia Tech, Omar Khan from Google's PowerMeter program, and Carrie Armel from Stanford's Precourt Energy Efficiency Center. (The Precourt Center website has an excellent section devoted to behavior with an 800+ citation database.)
Each panelist gave a very short presentation giving their perspective on smart grid and behavior, followed by lots of Q&A with the audience. One of the many interested issues touched upon was an informal "stages of use" observed by the Google Powermeter team: users begin by learning about the system, then they engage with it to change their use, then they finally move into a maintenance phase. The learning/engagement phases appear to last only a week or two, where users are actively manipulating the system and discovering issues with their usage and making concrete changes. During this phase, they want to be able to drill down, do experiments, and so forth. In the maintenance phase, they basically only want the system to tell them if something has changed in their energy usage.
On to Day Two or Day Three.
Monday, September 28, 2009
First experiences with the TED 5000
This past weekend I became the proud owner of a TED 5000 home energy monitor. I thought it might be useful to detail some initial experiences for the Renewable Energy and Island Sustainability Project and for our own subproject on From Smart Grids to Smart Consumers.
Overview of the TED 5000
The TED-5000 consists of four components.
MTU: The "MTU" (for Measurement Transmitting Unit) goes inside your electric panel and monitors the power being consumed by your house.
Gateway: The "Gateway" is a small unit that plugs into a wall outlet and connects to your router (in my case, an Airport) using an ethernet cable. The Gateway receives current power consumption data from the MTU every few seconds via a proprietary powerline network protocol.
Display Unit: The "Display Unit" is a small handheld device (like an iPod) that communicates wirelessly with the Gateway using Zigbee and can show you (among other things) a near real-time display of your power usage.
Footprints: Finally, "Footprints" is a web application that runs on the Gateway. You use Footprints to configure your TED, display various graphs and charts, and install Firmware updates.
Installation:
Installing the TED 5000 involves: (1) opening the electric panel and connecting the MTU; (2) plugging in the Gateway and attaching an ethernet cable from it to my Airport; and (3) accessing the Footprints web application and setting some configuration values.
I found that connecting the MTU was pretty straightforward, basically because I had two electrically-akamai friends (Robert Brewer and Tony Querubin) helping and thus my personal involvement was limited to showing them where my electrical panel was located, holding a flashlight, and serving them Apple Huguenot torte when they finished. The TED installation video provides a good approximation to what's involved, though there are some minor changes from the TED 1000 to the TED 5000.
Installation Glitch #1: No http://TED5000
The installation manual says that after hooking up the MTU and Gateway, you can access the Footprints web application using: http://TED5000. That fails for non-Windows systems. Fortunately, Robert was already aware of the problem and how to work around it. First, he looked up my Mac's IP address (10.0.1.3) and invoked "ping 10.0.1.255" to find out all of the devices on the 10.0.1.* subnet. Three devices responded: my Airport (10.0.1.1), my computer (10.0.1.3), and a third "mystery" device at 10.0.1.2. Bingo: the URL to the footprints software on my computer is: http://10.0.1.2/Footprints.html.
You might think that providing a URL that only works on Windows systems (and not even stating that in the installation guide) is pretty lame. But wait, it gets worse:
Installation Glitch #2: No HST.
The Display Unit provides the current time, and to do so, it needs to know the time zone. Part of the configuration process in Footprints involves specifying the time zone, for which it provides a drop down menu with exactly four choices: EST, CST, MST, and PST. No HST for those of us in Hawaii; no AKST for those of us in Alaska. I am hoping the TED programmers get a clue in the near future and issue a firmware update that fixes this problem; in the interim, my Display Unit is laboring under the illusion that it lives in California.
Initial results
Having real-time display of power usage reminds me a great deal of when we first got our Prius: all of a sudden you're getting a whole new dimension of interesting data about your environment. From a safety perspective, the TED has it all over the Prius, since it does not tempt you to monitor whether you're running exclusively on battery power while driving down the Pali Highway at rush hour.
Instead, from the sedentary position of my kitchen table, I can now inform you that our baseline power usage is around 400 Watts. Each of our ceiling fans use about 50 W. Turning on the microwave increases our power usage by about 1000 W. I am too horrified to confess how much power our tiny, "Energy Star", 5000 BTU window air conditioner uses on startup, but it dwarfs every other appliance we own by a mile.
I am sure that we are similar to every other new TED user, in that we tend to run over to the Display Unit whenever we do anything (turn on/off the TV, turn on/off the stove, etc.) to see what effect it has on our power consumption. Similarly, if we're in the vicinity of the Display Unit and our power usage is significantly different from 400W, we start wondering what's going on.
It's also cool to empirically test various things you've heard in the news. For example, Joanne read some place that you should unplug your toaster oven when you're not using it because it uses up power. That seemed suspect to me, so I took the Display Unit over to our toaster oven, unplugged it, and waited for a drop in power consumption. Nada.
Behavioral change, or Is This Just a $250 toy?
The real question is whether the TED 5000 is anything more than an expensive toy, and the answer to that depends upon whether we actually change (i.e. reduce) our energy consumption now that we know what we're doing.
The jury is still out on that question, though of course the jury has only had about 36 hours to deliberate. What I can say at this point is the following:
(1) Our espresso maker is an energy hog. I was surprised to see that it uses over 1000W, and this is significant because we tend to turn it on first thing in the morning and leave it on for a couple of hours until we have finished making all our morning espressos. Given its energy consumption, we should turn it off between uses even though that means we need to wait for it to heat up again.
(2) Our TV/DVR's standby power consumption is fairly low. We have heard horror stories about LCDs TVs that consume many watts even when "off", and we were happy to discover that our TV uses only 1-2W when in standby. The DVR uses about 20-22W in standby. That's not too bad, although that 25W does account for about 6% of our "baseline" 400 W.
(3) We need to replace our remaining incandescent lightbulbs with CFLs without delay. When your baseline is 400W, and you turn on the kitchen lights and it jumps by 25%, you just feel stupid.
I intend to do a bit more detective work on our baseline usage, and perhaps we can cut that down. Come to think of it, we should not make any changes to our behavior for a week so that we can establish a decent baseline dataset of our current-but-soon-to-be-history energetically profligate lifestyle.
For another new TED 5000 user's experience, see here.
Overview of the TED 5000
The TED-5000 consists of four components.
MTU: The "MTU" (for Measurement Transmitting Unit) goes inside your electric panel and monitors the power being consumed by your house.
Installation:
Installing the TED 5000 involves: (1) opening the electric panel and connecting the MTU; (2) plugging in the Gateway and attaching an ethernet cable from it to my Airport; and (3) accessing the Footprints web application and setting some configuration values.
I found that connecting the MTU was pretty straightforward, basically because I had two electrically-akamai friends (Robert Brewer and Tony Querubin) helping and thus my personal involvement was limited to showing them where my electrical panel was located, holding a flashlight, and serving them Apple Huguenot torte when they finished. The TED installation video provides a good approximation to what's involved, though there are some minor changes from the TED 1000 to the TED 5000.
Installation Glitch #1: No http://TED5000
The installation manual says that after hooking up the MTU and Gateway, you can access the Footprints web application using: http://TED5000. That fails for non-Windows systems. Fortunately, Robert was already aware of the problem and how to work around it. First, he looked up my Mac's IP address (10.0.1.3) and invoked "ping 10.0.1.255" to find out all of the devices on the 10.0.1.* subnet. Three devices responded: my Airport (10.0.1.1), my computer (10.0.1.3), and a third "mystery" device at 10.0.1.2. Bingo: the URL to the footprints software on my computer is: http://10.0.1.2/Footprints.html.
You might think that providing a URL that only works on Windows systems (and not even stating that in the installation guide) is pretty lame. But wait, it gets worse:
Installation Glitch #2: No HST.
The Display Unit provides the current time, and to do so, it needs to know the time zone. Part of the configuration process in Footprints involves specifying the time zone, for which it provides a drop down menu with exactly four choices: EST, CST, MST, and PST. No HST for those of us in Hawaii; no AKST for those of us in Alaska. I am hoping the TED programmers get a clue in the near future and issue a firmware update that fixes this problem; in the interim, my Display Unit is laboring under the illusion that it lives in California.
Initial results
Having real-time display of power usage reminds me a great deal of when we first got our Prius: all of a sudden you're getting a whole new dimension of interesting data about your environment. From a safety perspective, the TED has it all over the Prius, since it does not tempt you to monitor whether you're running exclusively on battery power while driving down the Pali Highway at rush hour.
Instead, from the sedentary position of my kitchen table, I can now inform you that our baseline power usage is around 400 Watts. Each of our ceiling fans use about 50 W. Turning on the microwave increases our power usage by about 1000 W. I am too horrified to confess how much power our tiny, "Energy Star", 5000 BTU window air conditioner uses on startup, but it dwarfs every other appliance we own by a mile.
I am sure that we are similar to every other new TED user, in that we tend to run over to the Display Unit whenever we do anything (turn on/off the TV, turn on/off the stove, etc.) to see what effect it has on our power consumption. Similarly, if we're in the vicinity of the Display Unit and our power usage is significantly different from 400W, we start wondering what's going on.
It's also cool to empirically test various things you've heard in the news. For example, Joanne read some place that you should unplug your toaster oven when you're not using it because it uses up power. That seemed suspect to me, so I took the Display Unit over to our toaster oven, unplugged it, and waited for a drop in power consumption. Nada.
Behavioral change, or Is This Just a $250 toy?
The real question is whether the TED 5000 is anything more than an expensive toy, and the answer to that depends upon whether we actually change (i.e. reduce) our energy consumption now that we know what we're doing.
The jury is still out on that question, though of course the jury has only had about 36 hours to deliberate. What I can say at this point is the following:
(1) Our espresso maker is an energy hog. I was surprised to see that it uses over 1000W, and this is significant because we tend to turn it on first thing in the morning and leave it on for a couple of hours until we have finished making all our morning espressos. Given its energy consumption, we should turn it off between uses even though that means we need to wait for it to heat up again.
(2) Our TV/DVR's standby power consumption is fairly low. We have heard horror stories about LCDs TVs that consume many watts even when "off", and we were happy to discover that our TV uses only 1-2W when in standby. The DVR uses about 20-22W in standby. That's not too bad, although that 25W does account for about 6% of our "baseline" 400 W.
(3) We need to replace our remaining incandescent lightbulbs with CFLs without delay. When your baseline is 400W, and you turn on the kitchen lights and it jumps by 25%, you just feel stupid.
I intend to do a bit more detective work on our baseline usage, and perhaps we can cut that down. Come to think of it, we should not make any changes to our behavior for a week so that we can establish a decent baseline dataset of our current-but-soon-to-be-history energetically profligate lifestyle.
For another new TED 5000 user's experience, see here.
Monday, September 7, 2009
Pretty printing code with google-code-prettify
There are many ways to pretty print code in blogs, as a Google search will show you, and all the methods involve a certain amount of hassle. After some fooling around today, I have come across the following approach which is pretty simple and has the advantage that you don't have to preprocess your code to escape certain characters (such as angle brackets).
My currently preferred approach uses google-code-prettify, which I suspect is the package used to perform syntax highlighting in google project hosting. Integrating this package in Blogger requires three steps:
1. Update the <head> section of your Blogger template
Insert the following lines of code:
Change it to:
3. Insert your code using <code class="prettyprint lang-java">
You must be in "Edit Html" mode, not "Compose" mode, when you do this. Here is an example of the results:
OK, if you have followed me so far, and if your Blog template has a light background, then the code should be formatted nicely. However, if your Blogger template has a black background like mine, then the results are terrible, because the default colors in prettify.css are designed to look good against a white background, not a black background.
The reason why my code sample looks relatively reasonable is because I use a custom version of prettify.css where I have changed the colors to those that contrast well with a black background. This file is available to you at:
http://ics-software-engineering.googlecode.com/svn/trunk/prettify/prettify.css
So, use this URL in the <link> element that you inserted into the <head> element of your Blogger template, and you should be fine.
If you don't like my choice of colors (or google's), you can create your own prettify.css, place it somewhere on the web, and reference it to get your own coloring scheme.
There is a bug in Blogger that you should be aware of: if you switch back and forth between "Compose" and "Edit HTML" mode while doing this, at some point Blogger eliminates all indentation in your code. You can avoid this by not using "Compose", or else by inserting your code as the last step in creating your posting.
My currently preferred approach uses google-code-prettify, which I suspect is the package used to perform syntax highlighting in google project hosting. Integrating this package in Blogger requires three steps:
1. Update the <head> section of your Blogger template
Insert the following lines of code:
<link href="http://google-code-prettify.googlecode.com/svn/trunk/src/prettify/prettify.css" rel="stylesheet" type="text/css"/> <script src="http://google-code-prettify.googlecode.com/svn/trunk/src/prettify.js" type="text/javascript" />2. Update the <body> element tag of your Blogger template
Change it to:
<body onload="prettyPrint()">Save these changes to your template.
3. Insert your code using <code class="prettyprint lang-java">
You must be in "Edit Html" mode, not "Compose" mode, when you do this. Here is an example of the results:
/**
* Creates a new CD, provided its title, group, and song list.
* @param title The title.
* @param group The group.
* @param tracks The song titles.
*/
public CompactDisc(String title, String group, String... tracks) {
this.title = title;
this.group = group;
this.tracks = Arrays.asList(tracks);
// Since CompactDiscs are immutable, the hash value will never change for an instance.
this.hashvalue = (new String(title + group)).hashCode();
}
OK, if you have followed me so far, and if your Blog template has a light background, then the code should be formatted nicely. However, if your Blogger template has a black background like mine, then the results are terrible, because the default colors in prettify.css are designed to look good against a white background, not a black background.
The reason why my code sample looks relatively reasonable is because I use a custom version of prettify.css where I have changed the colors to those that contrast well with a black background. This file is available to you at:
http://ics-software-engineering.googlecode.com/svn/trunk/prettify/prettify.css
So, use this URL in the <link> element that you inserted into the <head> element of your Blogger template, and you should be fine.
If you don't like my choice of colors (or google's), you can create your own prettify.css, place it somewhere on the web, and reference it to get your own coloring scheme.
There is a bug in Blogger that you should be aware of: if you switch back and forth between "Compose" and "Edit HTML" mode while doing this, at some point Blogger eliminates all indentation in your code. You can avoid this by not using "Compose", or else by inserting your code as the last step in creating your posting.
Monday, July 13, 2009
Google Summer of Code Midterm Results for Hackystat
I am delighted to announce that all five of the GSoC students working on Hackystat have passed their midterm evaluation and will be continuing with us for the rest of the summer.
Myriam Leggieri is working on the use of RDF to facilitate integration of Hackystat data with other kinds of knowledge sources in the project hackystat-linked-sensor-data.
Rachel Shadoan is working on the integration of social networking with Hackystat data in order to create novel data mining opportunities in the project hackystat-analysis-socnet.
Harvey Weitz is investigating a case-based reasoning approach to management for service-oriented architecture, using Hackystat as his test case, in the project hackystat-service-manager.
Anthony Du is implementing a software engineering educational game in the project hackystat-ui-devcathlon, building upon the initial work done in ICS 414 last spring.
Last but not least, Shaoxuan Zhang is implementing support for Issue sensor data collection and analysis through the hackystat-sensor-ant project.
All of these students have made significant progress on their projects so far, and the summer is only half over. I look forward to see what they have working by September!
Myriam Leggieri is working on the use of RDF to facilitate integration of Hackystat data with other kinds of knowledge sources in the project hackystat-linked-sensor-data.
Rachel Shadoan is working on the integration of social networking with Hackystat data in order to create novel data mining opportunities in the project hackystat-analysis-socnet.
Harvey Weitz is investigating a case-based reasoning approach to management for service-oriented architecture, using Hackystat as his test case, in the project hackystat-service-manager.
Anthony Du is implementing a software engineering educational game in the project hackystat-ui-devcathlon, building upon the initial work done in ICS 414 last spring.
Last but not least, Shaoxuan Zhang is implementing support for Issue sensor data collection and analysis through the hackystat-sensor-ant project.
All of these students have made significant progress on their projects so far, and the summer is only half over. I look forward to see what they have working by September!
Saturday, March 14, 2009
Monday, March 2, 2009
Google Summer of Code 2009
Applications for Google Summer of Code 2009 are coming up soon. I recently gave a short talk on Google Summer of Code to some graduate students at the University of Hawaii. Here is a screencast of that lecture:
Saturday, October 11, 2008
How to guarantee you will not be considered for a student internship
Since I founded the Collaborative Software Development Laboratory in 1991, I have provided research positions and internships to students from across the world, including Germany, Italy, India, China, Japan, Australia, Iceland, and Indonesia. Providing these opportunities to students, and learning from their differing cultural backgrounds is one of the great pleasures of being a professor.
Every semester I receive dozens of emails from students around the world who are requesting consideration for a research position of some sort in my lab. Unfortunately, most of them are quite similar to the one I just received this morning:
Dear Prof.,
To introduce myself, I am a 3rd year student of the Department of [Deleted] Majoring in Statistics and Informatics(5 yr.Integrated) at [Deleted], [Deleted]'s premier research organization, looking at the possibility of obtaining a position for Summer Internship, gelling with my academic background. I am aware of the superior quality of research at your institute ,I have decided that your current research work matches my interests to a remarkable degree.
Enclosed please find a copy of my resume. A number of details about my profile appear in the same. Yet no resume can comprehensively spell out everything.If my profile, prima facie matches with your requirements for a Summer Intern, please revert back, so that I could furnish any more relevant information.May I please also enquire whether some funding may be available for this internship in the form of a grant or scholarship?
Looking forward to a reply in the affirmative.
If selected by your consideration I promise to complete my assignments with utmost sincerity.
--
Thanking you
Your Sincerely....
There are, of course, a number of grammatical errors in this email, but since this student is clearly a non-native speaker, those errors would not deter me in the slightest from considering him for a research internship.
What makes this request a non-starter is the fact that this student sent me a form letter: there is not a single detail in this letter that provides evidence that the student has any clue about my research interests. Indeed, this student does not even take the time to address the email to me personally!
So, with the goal of helping other students who might be interested in academic experiences outside of their current environment, here is a simple guideline:
Send 10 personal, carefully written emails to professors whose research interests really do match your own, with concrete details about their research and how it intersects with your academic interests. These 10 emails have higher odds of success than (for example) 1000 generic emails sent to every professor in the country of your choice.
What might a "personal, carefully written email" include? Here are some ideas:
When I receive this kind of email from a student, I consider it carefully, and even if their interests don't really match mine, I reply out of respect for the energy they clearly put into their request and try to provide pointers to colleagues with better matching interests.
And there you see the key to why this approach is more effective: if you, as a student, devote this kind of energy up front on a small number of letters to a small number of professors, you can enlist our help in the search process. In terms of searching the space of appropriate research institutes, having professors guide your search is significantly more effective than spewing out 1000 generic emails.
Have I ever received such a letter from a student? Yes, several times, and in fact the most recent student who introduced himself this way is arriving in my lab on Monday to start his internship.
Every semester I receive dozens of emails from students around the world who are requesting consideration for a research position of some sort in my lab. Unfortunately, most of them are quite similar to the one I just received this morning:
Dear Prof.,
To introduce myself, I am a 3rd year student of the Department of [Deleted] Majoring in Statistics and Informatics(5 yr.Integrated) at [Deleted], [Deleted]'s premier research organization, looking at the possibility of obtaining a position for Summer Internship, gelling with my academic background. I am aware of the superior quality of research at your institute ,I have decided that your current research work matches my interests to a remarkable degree.
Enclosed please find a copy of my resume. A number of details about my profile appear in the same. Yet no resume can comprehensively spell out everything.If my profile, prima facie matches with your requirements for a Summer Intern, please revert back, so that I could furnish any more relevant information.May I please also enquire whether some funding may be available for this internship in the form of a grant or scholarship?
Looking forward to a reply in the affirmative.
If selected by your consideration I promise to complete my assignments with utmost sincerity.
--
Thanking you
Your Sincerely....
There are, of course, a number of grammatical errors in this email, but since this student is clearly a non-native speaker, those errors would not deter me in the slightest from considering him for a research internship.
What makes this request a non-starter is the fact that this student sent me a form letter: there is not a single detail in this letter that provides evidence that the student has any clue about my research interests. Indeed, this student does not even take the time to address the email to me personally!
So, with the goal of helping other students who might be interested in academic experiences outside of their current environment, here is a simple guideline:
Send 10 personal, carefully written emails to professors whose research interests really do match your own, with concrete details about their research and how it intersects with your academic interests. These 10 emails have higher odds of success than (for example) 1000 generic emails sent to every professor in the country of your choice.
What might a "personal, carefully written email" include? Here are some ideas:
- Include the professor's actual name.
- Include the professor's actual laboratory name.
- Include references to at least two recent publications of the professor, with questions and/or comments about the papers that indicate that you have read the papers and reflected on them.
- Include some simple, but concrete ideas on how you might contribute to the research. These don't have to be feasible or ground breaking. Just demonstrate that you're trying.
- Optionally, include other areas of interest of yours, which might have some interdisciplinary connection.
When I receive this kind of email from a student, I consider it carefully, and even if their interests don't really match mine, I reply out of respect for the energy they clearly put into their request and try to provide pointers to colleagues with better matching interests.
And there you see the key to why this approach is more effective: if you, as a student, devote this kind of energy up front on a small number of letters to a small number of professors, you can enlist our help in the search process. In terms of searching the space of appropriate research institutes, having professors guide your search is significantly more effective than spewing out 1000 generic emails.
Have I ever received such a letter from a student? Yes, several times, and in fact the most recent student who introduced himself this way is arriving in my lab on Monday to start his internship.
Tuesday, August 26, 2008
Reflections on Google Summer of Code 2008
Background
Back in April, we applied Hackystat to the 2008 Google Summer of Code program. We didn't know too much about it, other than that it provided a chance for students to be funded by Google to work on open source projects for the summer.
With great glee, we learned in March that Hackystat was accepted as one of the 140 projects sponsored by Google. The next step was to solicit student applications, which we did by sending email to the Hackystat discussion lists. We ended up with around 20 applications. There were a few that were totally off the wall from people who had no clue what Hackystat was, and a few others that were disorganized, incomplete, or otherwise indicative of a student who would probably not be successful. But, a good dozen of the 20 applications appeared quite promising and deserving of funding.
Google then posted the number of "slots" for each project--the maximum number of students that they would support. Hackystat got 4 slots. The number of slots is apparently based partially on the number of applications received by the project, and partially on the organization's past track record with GSoC. Hackystat had no prior track record, and couldn't compete with the number of applications for, say, the Apache Foundation. The GSoC Program Administrator answered the anguished pleas of new organizations who got less slots than they wanted by basically saying, "Look, we don't want to give you a zillion slots and then have half a zillion projects fail. Do a good job this year with the slots you were given and reapply next year." Sound advice, actually.
We then started ranking the applications to figure out which four students should be funded. It was difficult and frustrating, because there were many good applications. At the end, we came up with four students who we felt had a combination of interesting project ideas and a good chance of success based on their skills and situations.
We were right. Three out of four of the students successfully completed their projects, and the fourth student had to drop out of the program due to sudden illness, which no one could have foreseen.
GSoC requires each student to have a mentor. This summer, Greg Wilson of the University of Toronto and I each took two students. Greg's students were physically sited at the University of Toronto, so he was able to have face-to-face interactions. My students were in China.
Student support took several forms over the summer. First, there was email and the Hackystat developer mailing lists. At the beginning of the summer, I received a few emails from students that I redirected to the mailing list, so that other project developers could respond, and also because the question asked was of general Hackystat interest. Fairly quickly, the students caught on, and started posting most of their general-interest questions to the list. I think this was one conceptual hurdle for the students to get over: they were not in a relationship just with me or Greg, but also with the entire Hackystat developer and user community. While there were certainly issues pertaining to the GSoC program that they discussed privately with their mentors, they were also "real" Hackystat developers and needed to learn how to interact with the wider community. All of the students acclimated to their new role.
We also requested that the students maintain a blog and post an entry at least once a week that would summarize what they'd been working on, what problems they'd run into, and what they were planning to do next. This was also pretty successful. You can see Shaoxuan's, Eva's, and Matthew's blogs for details. Interestingly, the Chinese students found they could not access their (U.S. created) blogs once they were in China, and so had to use Wiki pages.
Finally, I also set up weekly teleconferences via Skype with the two students I was mentoring in China. This was a miserable failure, probably due to my own lameness. Despite the fact that I live in a timezone (HST) shared by very few of my software engineering colleagues, and thus have lots of experience with multi-timezone teleconferencing, the Hawaii-China difference just totally threw me. The international dateline did not help matters. At any rate, we simply fell back to asynchronous communication via blogs and email and that worked fine.
For source code and documentation hosting, we used two mechanisms. The Hackystat project uses Google Project Hosting, and so the students I mentored used this service. Greg is the force behind Dr. Project, and so the students he mentored used that service. As part of the wrapup activities, his students ported their work to Google Project Hosting to conform to the Hackystat conventions.
Results
So, what did they actually accomplish? Matthew Bassett created a sensor for Microsoft Team Foundation server. Here's a screen shot of one page where the user can customize the events the sensor collects:
The sensor itself is available at: http://code.google.com/p/hackystat-sensor-tfs/.
Eva Wong worked on a data visualization system for Hackystat based on Flare.
Her project is available at: http://code.google.com/p/hackystat-ui-sensordatavisualizer/.
Finally, Shaoxuan Zhang worked on multi-project analysis mechanisms for Hackystat using Wicket. Here is a screen shot of the Portfolio page:
His project is available at: http://code.google.com/p/hackystat-ui-wicket/.
Reflections
So, what makes for a successful GSoC?
First, and most obviously, it's important to have good students. "Good", with respect to GSoC, seems to boil down to two essential attributes: a passion for the problem, and the ability to be self-starting. (As long as the student "starts", the mentors and other developers can help them "finish"). It was delightful to read Matthew's blog entries about Team Foundation Server: he obviously likes the technology and enjoyed digging into its internals. At one point in the summer, Shaoxuan sent me an email in which he apologized that he had not been working much for the past week because he just got married, but he'd work extra hard the next week to catch up! We clearly had passionate students.
It also helps to have good mentors. In the Hackystat project, we have an embarrassment of riches on this front, since the project includes a large number of academics who mentor as part of their day jobs. In the end, we only needed two active mentors for the four students, but we easily had mentoring capacity for a couple dozen students.
Establishing effective communication systems is critical. Part of this is technological. We found that email and blogs worked well. Skype did not work well for me, but that was probably operator error on my part. Greg had the additional opportunity to use face-to-face communication, which is certainly helpful but not at all necessary to success. The other part is social. Most of our students needed to learn over the summer to: (a) request help quickly when they ran into problems, and (b) direct their question to the appropriate forum: either the Hackystat developer mailing list or privately to a mentor via email. This wasn't particularly difficult or anything, it was just a part of the process of understanding the Hackystat project culture.
I think I would have more insightful "lessons learned" had any of the student projects crashed and burned, but fortunately for the students (and unfortunately for this blog posting), that simply didn't happen.
For the Hackystat project, participation in GSoC this summer has had many benefits. Clearly, we'll benefit from the code that the students have created and which now is publically visible in the Hackystat Component Directory. We are crossing our fingers that the students will continue to remain active members of the Hackystat community.
GSoC has also helped to create a new "center" of Hackystat expertise at the University of Toronto. We hope to build upon that in the future.
GSoC also catalyzed a number of discussions within the Hackystat developer community about the direction of the project and how students could most effectively participate. These insights will have long term value to the project.
I believe we are now significantly more skillful at mentoring students. I hope we get a chance to participate in GSoC 2009, and that we can build upon our experiences this summer next year.
Back in April, we applied Hackystat to the 2008 Google Summer of Code program. We didn't know too much about it, other than that it provided a chance for students to be funded by Google to work on open source projects for the summer.
With great glee, we learned in March that Hackystat was accepted as one of the 140 projects sponsored by Google. The next step was to solicit student applications, which we did by sending email to the Hackystat discussion lists. We ended up with around 20 applications. There were a few that were totally off the wall from people who had no clue what Hackystat was, and a few others that were disorganized, incomplete, or otherwise indicative of a student who would probably not be successful. But, a good dozen of the 20 applications appeared quite promising and deserving of funding.
Google then posted the number of "slots" for each project--the maximum number of students that they would support. Hackystat got 4 slots. The number of slots is apparently based partially on the number of applications received by the project, and partially on the organization's past track record with GSoC. Hackystat had no prior track record, and couldn't compete with the number of applications for, say, the Apache Foundation. The GSoC Program Administrator answered the anguished pleas of new organizations who got less slots than they wanted by basically saying, "Look, we don't want to give you a zillion slots and then have half a zillion projects fail. Do a good job this year with the slots you were given and reapply next year." Sound advice, actually.
We then started ranking the applications to figure out which four students should be funded. It was difficult and frustrating, because there were many good applications. At the end, we came up with four students who we felt had a combination of interesting project ideas and a good chance of success based on their skills and situations.
We were right. Three out of four of the students successfully completed their projects, and the fourth student had to drop out of the program due to sudden illness, which no one could have foreseen.
GSoC requires each student to have a mentor. This summer, Greg Wilson of the University of Toronto and I each took two students. Greg's students were physically sited at the University of Toronto, so he was able to have face-to-face interactions. My students were in China.
Student support took several forms over the summer. First, there was email and the Hackystat developer mailing lists. At the beginning of the summer, I received a few emails from students that I redirected to the mailing list, so that other project developers could respond, and also because the question asked was of general Hackystat interest. Fairly quickly, the students caught on, and started posting most of their general-interest questions to the list. I think this was one conceptual hurdle for the students to get over: they were not in a relationship just with me or Greg, but also with the entire Hackystat developer and user community. While there were certainly issues pertaining to the GSoC program that they discussed privately with their mentors, they were also "real" Hackystat developers and needed to learn how to interact with the wider community. All of the students acclimated to their new role.
We also requested that the students maintain a blog and post an entry at least once a week that would summarize what they'd been working on, what problems they'd run into, and what they were planning to do next. This was also pretty successful. You can see Shaoxuan's, Eva's, and Matthew's blogs for details. Interestingly, the Chinese students found they could not access their (U.S. created) blogs once they were in China, and so had to use Wiki pages.
Finally, I also set up weekly teleconferences via Skype with the two students I was mentoring in China. This was a miserable failure, probably due to my own lameness. Despite the fact that I live in a timezone (HST) shared by very few of my software engineering colleagues, and thus have lots of experience with multi-timezone teleconferencing, the Hawaii-China difference just totally threw me. The international dateline did not help matters. At any rate, we simply fell back to asynchronous communication via blogs and email and that worked fine.
For source code and documentation hosting, we used two mechanisms. The Hackystat project uses Google Project Hosting, and so the students I mentored used this service. Greg is the force behind Dr. Project, and so the students he mentored used that service. As part of the wrapup activities, his students ported their work to Google Project Hosting to conform to the Hackystat conventions.
Results
So, what did they actually accomplish? Matthew Bassett created a sensor for Microsoft Team Foundation server. Here's a screen shot of one page where the user can customize the events the sensor collects:
The sensor itself is available at: http://code.google.com/p/hackystat-sensor-tfs/.
Eva Wong worked on a data visualization system for Hackystat based on Flare.
Her project is available at: http://code.google.com/p/hackystat-ui-sensordatavisualizer/.
Finally, Shaoxuan Zhang worked on multi-project analysis mechanisms for Hackystat using Wicket. Here is a screen shot of the Portfolio page:
His project is available at: http://code.google.com/p/hackystat-ui-wicket/.
Reflections
So, what makes for a successful GSoC?
First, and most obviously, it's important to have good students. "Good", with respect to GSoC, seems to boil down to two essential attributes: a passion for the problem, and the ability to be self-starting. (As long as the student "starts", the mentors and other developers can help them "finish"). It was delightful to read Matthew's blog entries about Team Foundation Server: he obviously likes the technology and enjoyed digging into its internals. At one point in the summer, Shaoxuan sent me an email in which he apologized that he had not been working much for the past week because he just got married, but he'd work extra hard the next week to catch up! We clearly had passionate students.
It also helps to have good mentors. In the Hackystat project, we have an embarrassment of riches on this front, since the project includes a large number of academics who mentor as part of their day jobs. In the end, we only needed two active mentors for the four students, but we easily had mentoring capacity for a couple dozen students.
Establishing effective communication systems is critical. Part of this is technological. We found that email and blogs worked well. Skype did not work well for me, but that was probably operator error on my part. Greg had the additional opportunity to use face-to-face communication, which is certainly helpful but not at all necessary to success. The other part is social. Most of our students needed to learn over the summer to: (a) request help quickly when they ran into problems, and (b) direct their question to the appropriate forum: either the Hackystat developer mailing list or privately to a mentor via email. This wasn't particularly difficult or anything, it was just a part of the process of understanding the Hackystat project culture.
I think I would have more insightful "lessons learned" had any of the student projects crashed and burned, but fortunately for the students (and unfortunately for this blog posting), that simply didn't happen.
For the Hackystat project, participation in GSoC this summer has had many benefits. Clearly, we'll benefit from the code that the students have created and which now is publically visible in the Hackystat Component Directory. We are crossing our fingers that the students will continue to remain active members of the Hackystat community.
GSoC has also helped to create a new "center" of Hackystat expertise at the University of Toronto. We hope to build upon that in the future.
GSoC also catalyzed a number of discussions within the Hackystat developer community about the direction of the project and how students could most effectively participate. These insights will have long term value to the project.
I believe we are now significantly more skillful at mentoring students. I hope we get a chance to participate in GSoC 2009, and that we can build upon our experiences this summer next year.
Saturday, August 23, 2008
Clean code
There is a persistent, subtle perception that "coding is for the young": that as you progress as a technical professional, you "outgrow" coding. Perhaps this is because most organizations pay senior managers way better than their technical staff. Perhaps this is because many software developers hit a glass ceiling and decide they've learned all there is to know about code. Perhaps this is because coding is seen as a low-level skill and vulnerable to outsourcing.
Other disciplines lack this perception: no one would ever want Itzhak Perlman to give up playing violin for a management position, or believe that his musical development ended while he was in his 20's, or that a symphony would outsource his position to a young virtuoso, no matter how talented, on the basis that they could get equivalent quality for less money.
I think part of the reason for this difference in perception is a difference in visibility: one can immediately hear the quality of a great violinist, even if one does not play violin themselves. The quality of the work produced by a great coder is, unfortunately, almost invisible: how do you "hear" code that is simultaneously flexible, maintainable, understandable, and efficient? How do you hear it if you are a senior manager who doesn't even know how to code?
Clean Code: A Handbook of Agile Software Craftsmanship is a nice attempt to make the quality of great code visible, and in so doing makes some other points as well: that great code is very, very difficult to write; that even apparently well-written code can still be significantly improved; and that the ability to consistently write great code is a goal that will take most of us decades of consistent practice to achieve.
Clean Code is written by Robert Martin and his colleagues at Object Mentor. It begins with a chapter in which Bjarne Stroustrup, Grady Booch, Dave Thomas, Michael Feathers, Ron Jeffries, and Ward Cunningham are all asked to define "clean code". Their responses, and Martin's synthesis, would make a stunning Wikipedia entry for "clean code".
However, as he points out, knowing what clean code is, and even recognizing it when you see it, is far different (and far easier) than being able to actually create it yourself. The most interesting parts of the book are case studies where code from various open source systems (Tomcat, Apache Commons, Fitnesse, JUnit) is reviewed and improved.
Along the way, the "Object Mentor School of Clean Code" emerges. I found much to agree with, along with some controversial points. For example, I am a great believer in the use of Checkstyle to ensure that all public methods and classes have "fully qualified" JavaDoc comments (i.e. that all parameters and (if present) return values) are documented. The OMSCC actually has a fairly low opinion of comments: that they should be eliminated whereever possible in favor of code that is so well-written that comments are redundant, even for public methods. As a result, I don't think they could use an automated tool such as Checkstyle for QA on their JavaDocs.
In some cases, they create "straw man" examples: such as using three lines for a JavaDoc comment that could easily be contained on one line, and then complaining that the JavaDoc takes up too much vertical space.
From a truth in advertising standpoint, the book should have the word "Java" in its title. All of the examples are in Java, and while the authors attempt to generalize whenever possible, it is clear that many aspects of cleanliness are ultimately language-specific. While extremely useful to Java programmers, I am not sure how well these lessons would translate to Perl, Scala, or C.
One final nit: some of the chapters are quite small---dare I say too small, such as the five page chapter on Emergence. On the other hand, the chapter on concurrency gets a 30 page appendix with additional guidelines. If there is a second edition (and I hope there will be), I expect that the topics will get more balanced and even treatment.
Despite these minor shortcomings, I found this book to be well worth reading. I was humbled to see just how much better the authors could make code that already seemed perfectly "clean" to me. And I am happy that someone has made an eloquent and passionate argument for remaining in the trenches writing code for 10, 20, or 30 years, and the maturity and beauty that such discipline and persistence can yield.
Other disciplines lack this perception: no one would ever want Itzhak Perlman to give up playing violin for a management position, or believe that his musical development ended while he was in his 20's, or that a symphony would outsource his position to a young virtuoso, no matter how talented, on the basis that they could get equivalent quality for less money.
I think part of the reason for this difference in perception is a difference in visibility: one can immediately hear the quality of a great violinist, even if one does not play violin themselves. The quality of the work produced by a great coder is, unfortunately, almost invisible: how do you "hear" code that is simultaneously flexible, maintainable, understandable, and efficient? How do you hear it if you are a senior manager who doesn't even know how to code?
Clean Code: A Handbook of Agile Software Craftsmanship is a nice attempt to make the quality of great code visible, and in so doing makes some other points as well: that great code is very, very difficult to write; that even apparently well-written code can still be significantly improved; and that the ability to consistently write great code is a goal that will take most of us decades of consistent practice to achieve.
Clean Code is written by Robert Martin and his colleagues at Object Mentor. It begins with a chapter in which Bjarne Stroustrup, Grady Booch, Dave Thomas, Michael Feathers, Ron Jeffries, and Ward Cunningham are all asked to define "clean code". Their responses, and Martin's synthesis, would make a stunning Wikipedia entry for "clean code".
However, as he points out, knowing what clean code is, and even recognizing it when you see it, is far different (and far easier) than being able to actually create it yourself. The most interesting parts of the book are case studies where code from various open source systems (Tomcat, Apache Commons, Fitnesse, JUnit) is reviewed and improved.
Along the way, the "Object Mentor School of Clean Code" emerges. I found much to agree with, along with some controversial points. For example, I am a great believer in the use of Checkstyle to ensure that all public methods and classes have "fully qualified" JavaDoc comments (i.e. that all parameters and (if present) return values) are documented. The OMSCC actually has a fairly low opinion of comments: that they should be eliminated whereever possible in favor of code that is so well-written that comments are redundant, even for public methods. As a result, I don't think they could use an automated tool such as Checkstyle for QA on their JavaDocs.
In some cases, they create "straw man" examples: such as using three lines for a JavaDoc comment that could easily be contained on one line, and then complaining that the JavaDoc takes up too much vertical space.
From a truth in advertising standpoint, the book should have the word "Java" in its title. All of the examples are in Java, and while the authors attempt to generalize whenever possible, it is clear that many aspects of cleanliness are ultimately language-specific. While extremely useful to Java programmers, I am not sure how well these lessons would translate to Perl, Scala, or C.
One final nit: some of the chapters are quite small---dare I say too small, such as the five page chapter on Emergence. On the other hand, the chapter on concurrency gets a 30 page appendix with additional guidelines. If there is a second edition (and I hope there will be), I expect that the topics will get more balanced and even treatment.
Despite these minor shortcomings, I found this book to be well worth reading. I was humbled to see just how much better the authors could make code that already seemed perfectly "clean" to me. And I am happy that someone has made an eloquent and passionate argument for remaining in the trenches writing code for 10, 20, or 30 years, and the maturity and beauty that such discipline and persistence can yield.
Tuesday, March 18, 2008
From Telemetry to Trajectory
Cam Moore stopped by my office to chat yesterday, and in 15 minutes he managed to completely revolutionize my thinking about how to visualize software project information over time. (Not bad, most visitors usually need at least 30 :).
One outcome of our prior research in Hackystat was the idea of Software Project Telemetry, which Cedric Zhang explored for his Ph.D. thesis. Software Project Telemetry defines a nifty domain specific language for defining the kinds of software measures you want to consider, how to combine them together, and how to display them as trend lines. One thing that's neat about telemetry is that it enables you to discover covariances among measures. For example: "Hey, when my percentage of TDD-compliant episodes drops from 85% to 40%, my coverage drops precipitously too!" (That really happened.) It also serves as a way to create an "early warning system" for projects in trouble. For example, if coverage is steadily dropping, complexity is steadily increasing, and coupling is steadily increasing too, then it looks like your design or architecture is in trouble because the trends for three orthogonal static measures of structural quality are all deteriorating. Here's an example of a telemetry chart:
As the above chart illustrates, the X axis is always time, and there can be multiple Y axes for each kind of trend line.
Telemetry turns out to work really well for in-process monitoring of a single project, and we definitely want to continue supporting and enhancing software project telemetry.
Recently, we have started to think about software project "portfolio management": what happens when an organization has 100 or more projects under development and wants insight not only into individual projects, but into the "portfolio" as a whole? For example, what projects are similar to each other? What projects constitute "outliers"? What kind of management or organizational changes appear to make broad impacts across multiple projects?
These questions pose difficulties for a conventional telemetry oriented viewpoint. For example, how would you compare the telemetry for a project that is six months in duration with the telemetry for a project that is 12 months in duration? What if one project started in January and another project started in June? The notion of having multiple X axes in addition to multiple Y axes seems problematic at best.
Enter Cam. His idea is to think about "trajectory" instead of "telemetry". To make it simple, let's consider a situation where we are collecting three measures for a project: coupling, complexity, and coverage. Instead of using a 2D plot with the X axis being time, we use a 3D plot, where the three axes are coupling, complexity, and coverage. Now time is implicit in the "trajectory" of the plots through space.
Here's an exceedingly lame mockup using 3D VOPlot with a little powerpoint post-processing:
I actually took some astronomical data to make this image, so you should ignore the axis values and pretty much everything else about this image except the basic notion that we are now focusing on trajectory, rather than telemetry, and this makes certain things much easier.
First, since the "time" dimension is now implicit, we can much more easily compare projects that start/end at different times and/or have different durations. Just plot their trajectories using different color points for different projects, and the scaling/displacing comes "for free". My mockup illustrates two projects, one with a sequence of blue dots, and one with a sequence of green dots. The visualization makes a couple of things pretty obvious: (a) for a while, Project Blue and Project Green have pretty similar trajectories, except that Project Green's complexity is below Project Blue's, and (b) something weird happened to Project Blue near the end that didn't happen to Project Green.
Note that we have no idea about the relative durations of Project Green or Blue, nor about their start/end dates. And in many cases abstracting away those details might be exactly the right thing to do.
Second, we can now ask ourselves a whole new set of interesting questions about the trajectories associated with different projects, such as:
One outcome of our prior research in Hackystat was the idea of Software Project Telemetry, which Cedric Zhang explored for his Ph.D. thesis. Software Project Telemetry defines a nifty domain specific language for defining the kinds of software measures you want to consider, how to combine them together, and how to display them as trend lines. One thing that's neat about telemetry is that it enables you to discover covariances among measures. For example: "Hey, when my percentage of TDD-compliant episodes drops from 85% to 40%, my coverage drops precipitously too!" (That really happened.) It also serves as a way to create an "early warning system" for projects in trouble. For example, if coverage is steadily dropping, complexity is steadily increasing, and coupling is steadily increasing too, then it looks like your design or architecture is in trouble because the trends for three orthogonal static measures of structural quality are all deteriorating. Here's an example of a telemetry chart:
As the above chart illustrates, the X axis is always time, and there can be multiple Y axes for each kind of trend line.
Telemetry turns out to work really well for in-process monitoring of a single project, and we definitely want to continue supporting and enhancing software project telemetry.
Recently, we have started to think about software project "portfolio management": what happens when an organization has 100 or more projects under development and wants insight not only into individual projects, but into the "portfolio" as a whole? For example, what projects are similar to each other? What projects constitute "outliers"? What kind of management or organizational changes appear to make broad impacts across multiple projects?
These questions pose difficulties for a conventional telemetry oriented viewpoint. For example, how would you compare the telemetry for a project that is six months in duration with the telemetry for a project that is 12 months in duration? What if one project started in January and another project started in June? The notion of having multiple X axes in addition to multiple Y axes seems problematic at best.
Enter Cam. His idea is to think about "trajectory" instead of "telemetry". To make it simple, let's consider a situation where we are collecting three measures for a project: coupling, complexity, and coverage. Instead of using a 2D plot with the X axis being time, we use a 3D plot, where the three axes are coupling, complexity, and coverage. Now time is implicit in the "trajectory" of the plots through space.
Here's an exceedingly lame mockup using 3D VOPlot with a little powerpoint post-processing:
I actually took some astronomical data to make this image, so you should ignore the axis values and pretty much everything else about this image except the basic notion that we are now focusing on trajectory, rather than telemetry, and this makes certain things much easier.
First, since the "time" dimension is now implicit, we can much more easily compare projects that start/end at different times and/or have different durations. Just plot their trajectories using different color points for different projects, and the scaling/displacing comes "for free". My mockup illustrates two projects, one with a sequence of blue dots, and one with a sequence of green dots. The visualization makes a couple of things pretty obvious: (a) for a while, Project Blue and Project Green have pretty similar trajectories, except that Project Green's complexity is below Project Blue's, and (b) something weird happened to Project Blue near the end that didn't happen to Project Green.
Note that we have no idea about the relative durations of Project Green or Blue, nor about their start/end dates. And in many cases abstracting away those details might be exactly the right thing to do.
Second, we can now ask ourselves a whole new set of interesting questions about the trajectories associated with different projects, such as:
- What set of measures create interesting trajectories?
- Which projects have similar trajectories?
- Which projects have anomalous trajectories?
- What about higher dimensionalities, where we want to compare trajectories involving more than three measures?
Thursday, January 17, 2008
Hackystat Version 8
Hackystat Version 8 is now in public release. Hackystat is an open source framework for collection, analysis, visualization, interpretation, annotation, and dissemination of software development process and product data. This eighth major redesign of the system is intended to retain the advantages of previous versions while incorporating significant new capabilities:
RESTful web service architecture. The most significant change in Version 8 is its re-implementation as a set of web services communicating using REST principles. This architecture facilitates several of the features noted below, including scalability, openness, and platform/language neutrality.
Sensor-based. A primary means of data collection in Hackystat is through "sensors": small software plugins to development tools that unobtrusively collect and transmit low-level data to the Hackystat sensor data repository service called the "SensorBase".
Extensible. Hackystat can be extended to support new development tools by the creation of new sensors. It can also be extended to support new analyses by the creation of new services.
Open. All sensors and services communicate via HTTP PUT, GET, POST, and DELETE, according to RESTful web service principles. This "open" API has two advantages: (1) it makes it easy to extend the Hackystat Framework with new sensors and services; and (2) it makes it easy to integrate Hackystat sensors and services with other information services. Hackystat can participate as just one part of an "ecosystem" of information services for an organization.
High performance. The default version of the SensorBase uses an embedded Derby RDBMS for its back-end data store. Initial performance evaluation of this repository, in combination with our multi-threaded client-side SensorShell, has been been quite encouraging: we have achieved sustained transmission rates of approximately 1.2 million sensor data instances per hour. The SensorBase is designed to allow "pluggable" back-end data stores. One organization, for example, is using Microsoft SQL server as the back-end data store.
Scalable. A natural outcome of a web service architecture is scalability: one can distribute services across multiple services or aggregate them on a single server depending upon load and resource availability. Hackystat is also scalable due to the fact that each organization can run its own local SensorBase, or even multiple SensorBases if required. Finally, Hackystat can exploit HTTP caching as yet another scalability mechanism.
Secure. While Hackystat maintains a "public" SensorBase and associated services for use by the community, we expect that most organizations adopting Hackystat will choose to install and run the SensorBase and associated services locally and internally. This facilitates data security and privacy for organizations who do not wish sensitive product or process information to go beyond their corporate firewalls.
Platform and language neutrality. Hackystat's implementation as a set of RESTful web services makes it language and platform neutral. For example, a sensor implemented in .NET and running on Windows might send information to a SensorBase written in Java running on a Macintosh, which is queried by a user interface written in Ruby on Rails web application hosted on a Linux machine.
Open Source. Hackystat is hosted at Google Project Hosting, and distributed among approximately a dozen individual projects. The "umbrella" Hackystat project includes a Component Directory page with links to all of the related subprojects. Since most subprojects correspond to independent Hackystat services, they are typically free to choose their own open source license, though most have chosen GNU V2.
Out of box support for process and product data collection and analysis. The standard Hackystat includes a variety of process and product data collection and analyses, including: editor events and developer editing time, coverage, unit test invocations, build invocations, code issues discovered through static analysis tools, size metrics, complexity metrics, churn, and commits. Of course, the Open API makes it possible to extend this list with more.
When we began work on Hackystat in 2001, we thought of it primarily as a software metrics framework. Seven years later, we find that vision limiting, because it tends to focus one on the collection and display of numbers. Our vision for Hackystat now is broader: we believe that the collection and display of numbers is just the first step in an ongoing process of collaborative sense-making within a software development organization. An organization needs numbers, but it also needs ways to get those numbers to the right people at the right time. More importantly, it needs ways to incrementally interpret, re-interpret, and annotate those numbers over time to build up a collective consensus as to their meaning and implications for the organization. Our goal for Hackystat Version 8 is to be an effective infrastructure for participation in the broader knowledge gathering and refinement processes of an organization, or even the software development community as a whole. If successful, it can play a role in creating new mechanisms for improving the collective intelligence of a software development group.
RESTful web service architecture. The most significant change in Version 8 is its re-implementation as a set of web services communicating using REST principles. This architecture facilitates several of the features noted below, including scalability, openness, and platform/language neutrality.
Sensor-based. A primary means of data collection in Hackystat is through "sensors": small software plugins to development tools that unobtrusively collect and transmit low-level data to the Hackystat sensor data repository service called the "SensorBase".
Extensible. Hackystat can be extended to support new development tools by the creation of new sensors. It can also be extended to support new analyses by the creation of new services.
Open. All sensors and services communicate via HTTP PUT, GET, POST, and DELETE, according to RESTful web service principles. This "open" API has two advantages: (1) it makes it easy to extend the Hackystat Framework with new sensors and services; and (2) it makes it easy to integrate Hackystat sensors and services with other information services. Hackystat can participate as just one part of an "ecosystem" of information services for an organization.
High performance. The default version of the SensorBase uses an embedded Derby RDBMS for its back-end data store. Initial performance evaluation of this repository, in combination with our multi-threaded client-side SensorShell, has been been quite encouraging: we have achieved sustained transmission rates of approximately 1.2 million sensor data instances per hour. The SensorBase is designed to allow "pluggable" back-end data stores. One organization, for example, is using Microsoft SQL server as the back-end data store.
Scalable. A natural outcome of a web service architecture is scalability: one can distribute services across multiple services or aggregate them on a single server depending upon load and resource availability. Hackystat is also scalable due to the fact that each organization can run its own local SensorBase, or even multiple SensorBases if required. Finally, Hackystat can exploit HTTP caching as yet another scalability mechanism.
Secure. While Hackystat maintains a "public" SensorBase and associated services for use by the community, we expect that most organizations adopting Hackystat will choose to install and run the SensorBase and associated services locally and internally. This facilitates data security and privacy for organizations who do not wish sensitive product or process information to go beyond their corporate firewalls.
Platform and language neutrality. Hackystat's implementation as a set of RESTful web services makes it language and platform neutral. For example, a sensor implemented in .NET and running on Windows might send information to a SensorBase written in Java running on a Macintosh, which is queried by a user interface written in Ruby on Rails web application hosted on a Linux machine.
Open Source. Hackystat is hosted at Google Project Hosting, and distributed among approximately a dozen individual projects. The "umbrella" Hackystat project includes a Component Directory page with links to all of the related subprojects. Since most subprojects correspond to independent Hackystat services, they are typically free to choose their own open source license, though most have chosen GNU V2.
Out of box support for process and product data collection and analysis. The standard Hackystat includes a variety of process and product data collection and analyses, including: editor events and developer editing time, coverage, unit test invocations, build invocations, code issues discovered through static analysis tools, size metrics, complexity metrics, churn, and commits. Of course, the Open API makes it possible to extend this list with more.
When we began work on Hackystat in 2001, we thought of it primarily as a software metrics framework. Seven years later, we find that vision limiting, because it tends to focus one on the collection and display of numbers. Our vision for Hackystat now is broader: we believe that the collection and display of numbers is just the first step in an ongoing process of collaborative sense-making within a software development organization. An organization needs numbers, but it also needs ways to get those numbers to the right people at the right time. More importantly, it needs ways to incrementally interpret, re-interpret, and annotate those numbers over time to build up a collective consensus as to their meaning and implications for the organization. Our goal for Hackystat Version 8 is to be an effective infrastructure for participation in the broader knowledge gathering and refinement processes of an organization, or even the software development community as a whole. If successful, it can play a role in creating new mechanisms for improving the collective intelligence of a software development group.
Friday, December 14, 2007
If Collective Intelligence is the question, then dashboards are not the answer
I've been thinking recently about collective intelligence and how it applies to software engineering in general and software metrics in particular.
My initial perspective on collective intelligence is that provides an organizing principle for a process in which a small "nugget" of information is somehow made available to one or more people, who then refine it, communicate it to others, or discard it as the case may be. Collective Intelligence results when mechanisms exist to prevent these nuggets from being viewed as spam (and the people communicating them from being viewed as spammers), along with mechanisms to support the refinement and elaboration of the initial nugget of information into an actual "chunk" of insight or knowledge. Such chunks, as they grow over time, enable long-term organization learning from collective intelligence processes.
Or something like that. What strikes me about the software metrics community and literature is that when it comes to how "measures become knowledge", the most common approach seems to be:
When we started work on Hackystat 8, it became clear that there were new opportunities to integrate our system with technologies like Google Gadgets, Twitter, Facebook, and so forth. I suspect that some of these integrations, like Google Gadgets, will turn out to be little more than neat hacks with only minor impact on the usability of the system. My conjecture is that the Hackystat killer app has nothing to do with "dashboards"; most modern metrics collection systems provide dashboards and people can ignore them just as easily as they ignore their web app predecessors.
On the other hand, integrating Hackystat with a communication facility like Twitter or with a social networking application has much more profound implications, because these integrations have the potential to create brand new ways to coordinate, communicate, and annotate an initial "nugget" generated by Hackystat into a "chunk" of knowledge of wider use to developers. It could also work the other way: an anecdotal "nugget" generated by a developer ("Hey folks, I think that we should all run verify before committing our code to reduce continuous integration build failure") could be refined into institutional knowledge (a telemetry graph showing the relationship between verify-before-commit and build success), or discarded (if the telemetry graph shows no relationship).
My initial perspective on collective intelligence is that provides an organizing principle for a process in which a small "nugget" of information is somehow made available to one or more people, who then refine it, communicate it to others, or discard it as the case may be. Collective Intelligence results when mechanisms exist to prevent these nuggets from being viewed as spam (and the people communicating them from being viewed as spammers), along with mechanisms to support the refinement and elaboration of the initial nugget of information into an actual "chunk" of insight or knowledge. Such chunks, as they grow over time, enable long-term organization learning from collective intelligence processes.
Or something like that. What strikes me about the software metrics community and literature is that when it comes to how "measures become knowledge", the most common approach seems to be:
- Management hires some people to be the "Software Process Group"
- The SPG goes out and measures developers and artifacts somehow.
- The SPG returns to their office and generates some statistics and regression lines.
- The SPG reports to management about the "best practices" they have discovered, such as "The optimal code inspection review rate is 200 LOC per hour".
- Management issues a decree to developers that, from now on, they are to review code at the rate of 200 LOC per hour.
When we started work on Hackystat 8, it became clear that there were new opportunities to integrate our system with technologies like Google Gadgets, Twitter, Facebook, and so forth. I suspect that some of these integrations, like Google Gadgets, will turn out to be little more than neat hacks with only minor impact on the usability of the system. My conjecture is that the Hackystat killer app has nothing to do with "dashboards"; most modern metrics collection systems provide dashboards and people can ignore them just as easily as they ignore their web app predecessors.
On the other hand, integrating Hackystat with a communication facility like Twitter or with a social networking application has much more profound implications, because these integrations have the potential to create brand new ways to coordinate, communicate, and annotate an initial "nugget" generated by Hackystat into a "chunk" of knowledge of wider use to developers. It could also work the other way: an anecdotal "nugget" generated by a developer ("Hey folks, I think that we should all run verify before committing our code to reduce continuous integration build failure") could be refined into institutional knowledge (a telemetry graph showing the relationship between verify-before-commit and build success), or discarded (if the telemetry graph shows no relationship).
Thursday, December 6, 2007
Social Networks for Software Engineers
I've been thinking lately about social networks, and what kind of social network infrastructure would attract me as a software engineer. Let's assume, of course, that my development processes and products can be captured via Hackystat and made available in some form to the social network. Why would this be cool?
The first reason would be because the social network could enable improved communication and coordination by providing greater transparency into the software development process. For example:
I'm not sure, however, that this is enough for the average developer. Where things get more interesting is when you realize that Hackystat is capable of developing a fairly rich representation of an individual developer's skills and knowledge areas.
As a simple example, when Java programmer edits a class file, the set of import statements reveal the libraries being used in that file, and thus the libraries that this developer has some familiarity with, because he or she is using those libraries to implement the class in question. When a Java programming edits a class file, they are also using some kind of editor---Emacs, Eclipse, Idea, NetBeans, and thus revealing some level of expertise with that environment. Indeed, Hackystat sensors can not only capture knowledge like "I've used the Eclipse IDE over 500 hours during the past year", but even things like "I know how to invoke the inspector and trace through functions in Eclipse", or "I've never once used the refactoring capabilities." Of course, Hackystat sensors can also capture information about what languages you write programs in, what operating systems you are familiar with, what other development tools you know about, and so forth. Shoots, Hackystat could even keep a record of the kinds of exceptions your code has generated.
Let's assume that all of this information can be processed and made available to you as, say, a FaceBook Application. And, you can edit the automatically generated profile to remove any skills you don't want revealed. You might also be able to annotate the information to provide explanatory information. You can provide details about yourself, such as "Student" or "Professional", and also your level of "openness" to the community. After all that's done, you press "Publish" and this becomes part of your FaceBook or OpenSocial profile.
So what?
Well, how about the following scenarios:
[1] I'm a student and just encountered a weird Exception. I search the community for others with experience with this Exception. I find three people, send them an IM, and shortly thereafter one of them gives me a tip on how to debug it.
[2] I'm interested in developing a Fortress mode for Emacs, but don't want to do it alone. I search the community for developers with both expertise in Fortress and Emacs, and contact them to see if they want to work with me on such a mode.
[3] I'm an employer and am interested in developers with a demonstrated experience with compiler development for a short-term, well paying consulting position. I need people who don't require any time to come up to speed on my problem; I don't want to hire someone who took compilers 10 years ago in college and hasn't thought about it since. I search the community, and find a set of candidates who have extensive, recent experience using Lex, YACC, and JavaCC. I contact them to see if they would be interested in a consulting arrangement.
[4] I'm a student who has been participating in open source projects and making extensive contributions, but has never had a "real" job. I need a way to convince employers that I have significant experience. I provide a pointer in my resume to my profile, showing that I have thousands of hours of contributions to the Apache web server and Python language projects.
Hackystat is often thought of as a measurement system, and indeed all the above capabilities result from measurement. However, the above doesn't feel like measurement, it feels like social coordination and communication of relatively sophisticated and efficient nature.
The first reason would be because the social network could enable improved communication and coordination by providing greater transparency into the software development process. For example:
- Software project telemetry would reveal the "trajectory" of development with respect to various measures, helping to reveal potential bottlenecks and problems earlier in development.
- Integration with Twitter could support automated "tweets" informing the developers when events of interest occur.
- An annotated Simile/Timeline representation of the project history could help developers understand and reflect upon a project and what could be done to improve it.
I'm not sure, however, that this is enough for the average developer. Where things get more interesting is when you realize that Hackystat is capable of developing a fairly rich representation of an individual developer's skills and knowledge areas.
As a simple example, when Java programmer edits a class file, the set of import statements reveal the libraries being used in that file, and thus the libraries that this developer has some familiarity with, because he or she is using those libraries to implement the class in question. When a Java programming edits a class file, they are also using some kind of editor---Emacs, Eclipse, Idea, NetBeans, and thus revealing some level of expertise with that environment. Indeed, Hackystat sensors can not only capture knowledge like "I've used the Eclipse IDE over 500 hours during the past year", but even things like "I know how to invoke the inspector and trace through functions in Eclipse", or "I've never once used the refactoring capabilities." Of course, Hackystat sensors can also capture information about what languages you write programs in, what operating systems you are familiar with, what other development tools you know about, and so forth. Shoots, Hackystat could even keep a record of the kinds of exceptions your code has generated.
Let's assume that all of this information can be processed and made available to you as, say, a FaceBook Application. And, you can edit the automatically generated profile to remove any skills you don't want revealed. You might also be able to annotate the information to provide explanatory information. You can provide details about yourself, such as "Student" or "Professional", and also your level of "openness" to the community. After all that's done, you press "Publish" and this becomes part of your FaceBook or OpenSocial profile.
So what?
Well, how about the following scenarios:
[1] I'm a student and just encountered a weird Exception. I search the community for others with experience with this Exception. I find three people, send them an IM, and shortly thereafter one of them gives me a tip on how to debug it.
[2] I'm interested in developing a Fortress mode for Emacs, but don't want to do it alone. I search the community for developers with both expertise in Fortress and Emacs, and contact them to see if they want to work with me on such a mode.
[3] I'm an employer and am interested in developers with a demonstrated experience with compiler development for a short-term, well paying consulting position. I need people who don't require any time to come up to speed on my problem; I don't want to hire someone who took compilers 10 years ago in college and hasn't thought about it since. I search the community, and find a set of candidates who have extensive, recent experience using Lex, YACC, and JavaCC. I contact them to see if they would be interested in a consulting arrangement.
[4] I'm a student who has been participating in open source projects and making extensive contributions, but has never had a "real" job. I need a way to convince employers that I have significant experience. I provide a pointer in my resume to my profile, showing that I have thousands of hours of contributions to the Apache web server and Python language projects.
Hackystat is often thought of as a measurement system, and indeed all the above capabilities result from measurement. However, the above doesn't feel like measurement, it feels like social coordination and communication of relatively sophisticated and efficient nature.
Monday, November 5, 2007
Measurement as Mashup, Ambient Devices, Social Networks, and Hackystat
The new architecture of Hackystat has me thinking about new metaphors for software engineering measurement. Indeed, it has me wondering if where we are heading is even characterized best as "measurement" or even "software engineering". Alistair Cockburn, for example, has written an article on The End Of Software Engineering in which he challenges the use of the term "software engineering" as an appropriate description for what people do (or should do) when developing software.
Similarly, when we began work on Hackystat six years ago, I thought in fairly conventional terms about this system: it was basically a way to make it simpler to gather more accurate measures that could be used for traditional software engineering measurement activities: baselines, prediction, control, quality assessment.
One interesting and unintended side effect of the Hackystat 8 architecture, in which the system becomes a seamless component of the overall internet information ecosystem via a set of RESTful web services, is a re-examination of my fundamental conceptions of what the system could and should be. In particular, two Web 2.0 concepts: "mashup", and "social network", provide interesting metaphors.
Measurement as Mashup
Hackystat has always embraced the idea of "mashup". From the earliest days, we have pursued the hypothesis that there is a "network effect" in software process and product metrics; that the more orthogonal measures you could gather about a system, the more potential you would gain for insight as you obtained the ability to compare and contrast them. Thus, we created a system that was easily extensible with sensors for different tools that could gather data of different types.
Software Project Telemetry is an early result of our search for ways to obtain meaning within this network effect. In Software Project Telemetry, we created a language that enables users to easily create "measurement mashups" consisting of metrics and their trends over time. The following screen image shows an example mashup, in which we opportunistically discovered a covariance between build failures and code churn over time for a specific project :
Hackystat 8 creates new opportunities for mashups, because we can now integrate this kind of data with other data collection and visualization systems. As one example, we are exploring the use of Twitter as a data collection and communication mechanism. Several members of the Hackystat 8 development group "follow" each other with Twitter and post information about their software development activities (among other things) as a way to increase awareness of project state. Here's a recent screen image show some of our posts:
There are at least two interesting directions for Twitter/Hackystat mashups. Assuming that members of a project team are twitter-enabled, we can provide a Hackystat service that monitors the data being collected from sensors and sends periodic "tweets" that answer the question "What are you doing now?" for individual developers and/or the project as a whole. Going the other direction, we can gather "tweets" as data that we can display on a Simile/Timeline with our metrics values, which provides an interesting approach to integrating qualitative and quantitative data.
A second form of mashup is the use of internet-enabled ambient devices such as Ambient Orbs or Nabaztag. The idea here is to get away from the use of the browser (or even the monitor) as the sole interface to Hackystat information and analyses. Instead, we could move toward Mark Weiser's vision of calm technology, or ""that which informs but doesn't demand our focus or attention".
The net of all this is that Hackystat is evolving from a kind of "local" capability for mashups represented by software project telemetry to a new "global" capability for mashups in which Hackystat can act as a first class citizen of the internet information infrastructure.
Software development as social network
Google is releasing an API for social networking called OpenSocial. This API essentially enables you to (a) create profiles of users; (b) publish and subscribe to events, and (c) maintain persistent data. You can use Google Gears to maintain data on client computers, and thus create more scalable systems. Google intends this as a way for developers to create third party applications that can run within multiple social networks (MySpace, Orkut), as well as enable users to maintain, transfer, and/or integrate data across these networks.
So. What would Hackystat look like, and what would it do, if it was implemented using OpenSocial?
First, I think that in contrast to the current analysis focus of Hackystat, in which the concept of a "project" as an organizing principle is very important, in an OpenSocial world you might not be so interested in a project-based orientation for analyses. Instead, I think the emphasis would be much more on the individual and their behaviors across, and independent of, projects.
For example, your Hackystat OpenSocial "profile" might include analysis results like: "I worked for three hours hacking Java code yesterday", or "I have a lot of experience with the Java 2D API", or "I use test driven design practices 80% of the time". All of these might be interesting to others in your social network as a representation of what you are doing currently and/or are capable of doing in future. The process/product characteristics of the projects that you work on might be less important in an OpenSocial profile for, I think, two reasons: (a) it is harder to understand the individual's contributions in the context of project-level analyses; and (b) project data might "give away" information that the employer of the developer does not want published.
Which brings me to a second conjecture: issues of data privacy or "sanitization" will become much more important for social network software engineering using a system like OpenSocial. To make the example analyses I listed above, it must be possible to collect detailed data about your activities as a developer (sufficient, for example, to infer TDD behaviors), yet publish them at an abstract enough level that no proprietary information is being revealed. That is a fascinating trade-off that will require a great deal of study and research. The implications are both technical and social.
Similarly, when we began work on Hackystat six years ago, I thought in fairly conventional terms about this system: it was basically a way to make it simpler to gather more accurate measures that could be used for traditional software engineering measurement activities: baselines, prediction, control, quality assessment.
One interesting and unintended side effect of the Hackystat 8 architecture, in which the system becomes a seamless component of the overall internet information ecosystem via a set of RESTful web services, is a re-examination of my fundamental conceptions of what the system could and should be. In particular, two Web 2.0 concepts: "mashup", and "social network", provide interesting metaphors.
Measurement as Mashup
Hackystat has always embraced the idea of "mashup". From the earliest days, we have pursued the hypothesis that there is a "network effect" in software process and product metrics; that the more orthogonal measures you could gather about a system, the more potential you would gain for insight as you obtained the ability to compare and contrast them. Thus, we created a system that was easily extensible with sensors for different tools that could gather data of different types.
Software Project Telemetry is an early result of our search for ways to obtain meaning within this network effect. In Software Project Telemetry, we created a language that enables users to easily create "measurement mashups" consisting of metrics and their trends over time. The following screen image shows an example mashup, in which we opportunistically discovered a covariance between build failures and code churn over time for a specific project :
Hackystat 8 creates new opportunities for mashups, because we can now integrate this kind of data with other data collection and visualization systems. As one example, we are exploring the use of Twitter as a data collection and communication mechanism. Several members of the Hackystat 8 development group "follow" each other with Twitter and post information about their software development activities (among other things) as a way to increase awareness of project state. Here's a recent screen image show some of our posts:
There are at least two interesting directions for Twitter/Hackystat mashups. Assuming that members of a project team are twitter-enabled, we can provide a Hackystat service that monitors the data being collected from sensors and sends periodic "tweets" that answer the question "What are you doing now?" for individual developers and/or the project as a whole. Going the other direction, we can gather "tweets" as data that we can display on a Simile/Timeline with our metrics values, which provides an interesting approach to integrating qualitative and quantitative data.
A second form of mashup is the use of internet-enabled ambient devices such as Ambient Orbs or Nabaztag. The idea here is to get away from the use of the browser (or even the monitor) as the sole interface to Hackystat information and analyses. Instead, we could move toward Mark Weiser's vision of calm technology, or ""that which informs but doesn't demand our focus or attention".
The net of all this is that Hackystat is evolving from a kind of "local" capability for mashups represented by software project telemetry to a new "global" capability for mashups in which Hackystat can act as a first class citizen of the internet information infrastructure.
Software development as social network
Google is releasing an API for social networking called OpenSocial. This API essentially enables you to (a) create profiles of users; (b) publish and subscribe to events, and (c) maintain persistent data. You can use Google Gears to maintain data on client computers, and thus create more scalable systems. Google intends this as a way for developers to create third party applications that can run within multiple social networks (MySpace, Orkut), as well as enable users to maintain, transfer, and/or integrate data across these networks.
So. What would Hackystat look like, and what would it do, if it was implemented using OpenSocial?
First, I think that in contrast to the current analysis focus of Hackystat, in which the concept of a "project" as an organizing principle is very important, in an OpenSocial world you might not be so interested in a project-based orientation for analyses. Instead, I think the emphasis would be much more on the individual and their behaviors across, and independent of, projects.
For example, your Hackystat OpenSocial "profile" might include analysis results like: "I worked for three hours hacking Java code yesterday", or "I have a lot of experience with the Java 2D API", or "I use test driven design practices 80% of the time". All of these might be interesting to others in your social network as a representation of what you are doing currently and/or are capable of doing in future. The process/product characteristics of the projects that you work on might be less important in an OpenSocial profile for, I think, two reasons: (a) it is harder to understand the individual's contributions in the context of project-level analyses; and (b) project data might "give away" information that the employer of the developer does not want published.
Which brings me to a second conjecture: issues of data privacy or "sanitization" will become much more important for social network software engineering using a system like OpenSocial. To make the example analyses I listed above, it must be possible to collect detailed data about your activities as a developer (sufficient, for example, to infer TDD behaviors), yet publish them at an abstract enough level that no proprietary information is being revealed. That is a fascinating trade-off that will require a great deal of study and research. The implications are both technical and social.
Monday, October 29, 2007
The Mismeasurement of Science
Peter Lawrence has written an interesting article on the (mis)use of measurement to assess "quality" and/or "impact" of scientists. It's called The Mismeasurement of Science, and appeared in Current Biology, August 7, 2007: 17 (15), r583. You can download it here.
Highly recommended reading, not only for scientists, but also as another interesting example of how a simple-minded approach to measuring "quality" or "productivity" has a wide range of dysfunctional implications. I particularly liked the following:
The measures seemed, at first rather harmless, but, like cuckoos in a nest, they have grown into monsters that threaten science itself. Already, they have produced an “audit society” in which scientists aim, and indeed are forced, to put meeting the measures above trying to understand nature and disease.
I suspect that similarly simple minded application of software engineering measures (such as Active Time in Hackystat) would have similarly disastrous consequences were anyone to actually take them seriously.
Highly recommended reading, not only for scientists, but also as another interesting example of how a simple-minded approach to measuring "quality" or "productivity" has a wide range of dysfunctional implications. I particularly liked the following:
The measures seemed, at first rather harmless, but, like cuckoos in a nest, they have grown into monsters that threaten science itself. Already, they have produced an “audit society” in which scientists aim, and indeed are forced, to put meeting the measures above trying to understand nature and disease.
I suspect that similarly simple minded application of software engineering measures (such as Active Time in Hackystat) would have similarly disastrous consequences were anyone to actually take them seriously.
Sunday, October 7, 2007
Hackystat and Crap4J
The folks at Agitar, who clearly have a sense of humor in addition to being excellent hackers, have recently produced a plug-in for Eclipse called Crap4J that calculates a measure of your code's "crappiness".
From their web page:
Immediately after sending this link to the Hackystat Hackers, a few of us started playing with it. While the metric seems intuitively appealing (and requires one to use lots of bad puns when reporting on the results), its implementation as an Eclipse plugin is quite limiting. We have found, for example, that the plugin fails on the SensorBase code, not through any fault of the SensorBase code (whose unit tests run quite happily within Eclipse and Ant) but seemingly because of some interaction with Agitar's auto-test invocation or coverage mechanism.
Thus, this seems like an opportunity for Hackystat. If we implement the CRAP metric as a higher level analysis (for example, at the Daily Project Data level), then any combination of tools that send Coverage data and FileMetric data (that provides cyclomatic complexity) can produce CRAP. Hackystat can thus measure CRAP independently of Eclipse or even Java.
The Agitar folks go on to say:
We are also aware that the CRAP formula doesn’t currently take into account higher-order, more design-oriented metrics that are relevant to maintainability (such as cohesion and coupling).
Here is another opportunity for Hackystat: it would be trivial, once we have a DPD analysis that produces CRAP, to provide a variant CRAP calculation that factors in Dependency sensor data (which provides measures of coupling and cohesion).
Then we could do a simple case study in which we run these two measures of CRAP over a code base, order the classes in the code base by their level of crappiness according to the two measures, and ask experts to assess which ordering appears to be more consistent with the code's "True" crappiness.
I think such a study could form the basis for a really crappy B.S. or M.S. Thesis.
From their web page:
There is no fool-proof, 100% objective and accurate way to determine if a particular piece of code is crappy or not. However, our intuition – backed by research and empirical evidence – is that unnecessarily complex and convoluted code, written by someone else, is the code most likely to elicit a “This is crap!” response. If the person looking at the code is also responsible for maintaining it going forward, the response typically changes into “Oh crap!”
Since writing automated tests (e.g., using JUnit) for complex code is particularly hard to do, crappy code usually comes with few, if any, automated tests. The presence of automated tests implies not only some degree of testability (which in turn seems to be associated with better, or more thoughtful, design), but it also means that the developers cared enough and had enough time to write tests – which is a good sign for the people inheriting the code.
Because the combination of complexity and lack of tests appear to be good indicators of code that is potentially crappy – and a maintenance challenge – my Agitar Labs colleague Bob Evans and I have been experimenting with a metric based on those two measurements. The Change Risk Analysis and Prediction (CRAP) score uses cyclomatic complexity and code coverage from automated tests to help estimate the effort and risk associated with maintaining legacy code. We started working on an open-source experimental tool called “crap4j” that calculates the CRAP score for Java code. We need more experience and time to fine tune it, but the initial results are encouraging and we have started to experiment with it in-house.
Here's a screenshot of Crap4J after a run over the SensorShell service code:
Immediately after sending this link to the Hackystat Hackers, a few of us started playing with it. While the metric seems intuitively appealing (and requires one to use lots of bad puns when reporting on the results), its implementation as an Eclipse plugin is quite limiting. We have found, for example, that the plugin fails on the SensorBase code, not through any fault of the SensorBase code (whose unit tests run quite happily within Eclipse and Ant) but seemingly because of some interaction with Agitar's auto-test invocation or coverage mechanism.
Thus, this seems like an opportunity for Hackystat. If we implement the CRAP metric as a higher level analysis (for example, at the Daily Project Data level), then any combination of tools that send Coverage data and FileMetric data (that provides cyclomatic complexity) can produce CRAP. Hackystat can thus measure CRAP independently of Eclipse or even Java.
The Agitar folks go on to say:
We are also aware that the CRAP formula doesn’t currently take into account higher-order, more design-oriented metrics that are relevant to maintainability (such as cohesion and coupling).
Here is another opportunity for Hackystat: it would be trivial, once we have a DPD analysis that produces CRAP, to provide a variant CRAP calculation that factors in Dependency sensor data (which provides measures of coupling and cohesion).
Then we could do a simple case study in which we run these two measures of CRAP over a code base, order the classes in the code base by their level of crappiness according to the two measures, and ask experts to assess which ordering appears to be more consistent with the code's "True" crappiness.
I think such a study could form the basis for a really crappy B.S. or M.S. Thesis.
Subscribe to:
Posts (Atom)