ZenLinux Blog

This is a summary of some of the sessions I attended at the Open Source Bridge conference in June 2010. The conference was held this year at the Portland Art Museum. I was both a speaker and a volunteer at this event, and had a great time. Over 300 people attended, which gave the conference an intimate feel, and allowed for many opportunities to meet and share ideas with other open source developers.

Return of Command-Line Kung Fu

Return of Command-Line Kung Fu presentation slides

Hal Pomeranz gave a talk on useful shell tricks you may not have heard of. I learned a lot in this presentation. Some of those things included:

Bash has a neat way of interacting with network services using a filesystem-like syntax. For example, if you wanted to send the output of a command over a network port, you can actually do this:

df > /dev/tcp/target_hostname/port_number

This does not rely on anything in the Linux filesystem – that filesystem path is handled by bash. target_hostname can be either an IP address or DNS hostname. And it works with tcp or udp connections. Note that Debian and some older Ubuntu distros disabled this feature in bash in their default packages. As another example, you could test whether a port is open with:

echo > /dev/tcp/localhost/22

Hal uses tricks like these when doing forensic analysis of potentially compromised servers, where he wants to capture diagnostic information but cannot write to the filesystem.

Sometimes, when output redirection in bash doesn’t appear to be working properly, you may need to run the command in a subshell – which you can do by enclosing the command in parenthesis, like so:

(echo > /dev/tcp/localhost/22) 2>/dev/null && echo live || echo dead

Also, I didn’t know you can redirect output from an entire loop – just add your redirection syntax on your “done” line.

Hal did some things with FIFOs that were neat but not immediately useful to me. A shell FIFO is essentially a “pipe that looks like a file.”

There was a good refresher on the strings (which can extract ASCII or Unicode), tee, and lsof commands. Also the use of time to do basic benchmarks.

Did you know:

• You can kill network processes by their port number with the lsof command? kill `lsof -t -i :22` (note: this will work on both source and destination ports, so be careful)
• There’s no need to uncompress text files to view them quickly when you use the zless command
• date -d @epoch_time is a quick way of converting from epoch time to human readable format
• !$ is shorthand for the last argument from the previous command in your history

The pkill command can kill a process by its process name, username, or even do fun things like kill the parent of a process name. I have been guilty of killing processes with a ps aux | grep process_name | grep -v grep too many times. I will never do this again – pkill is the way to go.

When trying to use the find command to filter filenames by timestamp, keep in mind that the -mtime family of options only work in granularities of a day or more. To work around this, you can use the touch command to create a file with a particular time stamp and then use find’s -newer option.

I was also reminded that I need to understand xargs better – it’s a pretty crucial shell tool.

Finally, Hal co-authors a blog at commandlinekungfu.com where he shares useful and more challenging things you can do with the shell. I immediately subscribed to his RSS feed.

Creating Embedded Linux Products with OpenEmbedded

This was the talk I gave – my slides can be found here.

Just over 20 people showed up. Only a handful had much experience with embedded Linux systems, and most were interested in learning how to get started with embedded Linux. As such, it was a perfect way to introduce some new folks to OpenEmbedded and Poky, and I received a lot of positive comments afterward from people who seemed ready to get started with it.

Organizing User Groups – A Panel Discussion

Several leaders of Portland-area user groups had a panel discussion on the care and feeding of user groups. There weren’t a lot of earth-shattering insights to me, but I made note of the following:

http://pdxgroups.pbworks.com is a wiki of Portland user groups.

New user groups should definitely post their meetings on Calagator and Upcoming.

Free Geek seems to be a pretty supportive meeting location for newly formed user groups before they get particularly large.

A successful user group usually has one or more “cheerleaders” who encourage involvement and celebrates local projects.

Shorter presentations (10-20 minutes) are a good way of getting your membership to present topics regularly – it makes the process much less intimidating. Also, giving a token gift to presenters – such as a mug with the user group logo on it – is a nice way of encouraging involvement.

Make member introductions meaningful – if you just go around the room and have everyone say their name, by the time you’re done most of that information will be forgotten. Ask a new question every month and have members answer it during introductions to help make identities “stick” and improve interaction between members.

Being a Catalyst in Communities

Download the presentation slides and notes.

This was a presentation by RedHat’s Karsten Wade on how to foster a healthy open source community.

Karsten started the talk with the infamous IRC chat log someone wrote up when RedHat announced the Fedora project, and that RHEL would be a product users would have to pay for. It was an example of how things were done poorly during this transition, and since then RedHat has done a much better job of being an open source community leader and member.

Some companies believe releasing their code as open source will result in a Tom Sawyer-like effect. Meaning, your codebase is like the fence that Tom Sawyer convinced his friends to paint for him while he sat under a tree eating an apple. This is not how things work at all. Instead, a successful open source project is more like a barn-raising, where people come together to work, socialize, and celebrate successes along the road to creating an important community resource. You don’t buy all the materials to build the barn first, lay out the blueprints, and invite everyone over to start working. It needs to be more of an organic and incremental process for things to work.

Even if there is a core group of people making important decisions about an OSS project, it is absolutely critical that those decisions be made transparently in an open forum, and that there be clear and credible way that any community member could become a core contributor through merit-based processes. If people view the core group as a special clique that will always have privileges over everyone else, the project is not likely to sustain any significant contributors.

Relevant history note: When Fedora was first released, and up until Fedora 7, the build system used was something RedHat didn’t release and this prevented the distribution from developing features people wanted, such as LiveCD support. This bothered the community so much that they ended up re-implementing their own distro build system for Fedora that was so good RedHat finally adopted it in Fedora 7. This then also led to the increase in third-party Fedora repositories, which have helped close the gap in the breadth of packages available to users in comparison with Debian’s offerings.

When measuring the health of your community, focus on the community and its interactions, not on metrics such as the number of downloads or bug reports.

Keep your processes simple and open to evolution – e.g, RedHat’s decision to stop fighting with the community over their distro build system and adopting the community version.

Make sure you are inviting diverse levels of participation. It is rare that experts are going to show up at your door to improve your project. Make project “janitors” feel important, because their work is important. As people feel more valued they will participate more, develop more experience, and become the experts of your project.

Combine familiarity and excitement – familiarity meaning a community rhythm with regular meetings and software releases, and excitement with new project goals that aren’t boring and which people have something worth working toward.

The second half of the talk focused largely on projects intended to help educators (particularly college professors) teach students through involvement in open source software projects. POSSE is one of these efforts.

Karsten is also an author of The Open Source Way book, which is relevant reading if this subject matter is important to you.

Using Modern Perl

This was a presentation by chromatic on some Perl best practices.

He wrote a perl module called Modern::Perl which turns on a lot of the best-practice defaults such as use strict / use warnings, etc.

The slides were pretty clear in showing off several other modules that help clean up the use of Perl and make it more readable. I will post a link to the slides when they become available.

He has also written a book on Modern Perl Programming.

How to Report Bugs and Receive Bug Reports

Michael Schwern gave a very entertaining talk on the dysfunctional state of bug reporting today.

Michael’s main point (which he drove home in many humorous ways throughout the talk) was that our way of handling bug reports discourages end users from doing so. Instead, the bug reporting process is something intended to make the developer’s life easier. But is it easier when developers aren’t fixing bugs that are not being reported?

Many projects have page-long descriptions of “bug reporting policies and procedures” that force users, who have an issue they are currently already frustrated with, to jump through more frustrating hoops before they are even “given permission” to report it. Everything from searching for bugs first to having exact version numbers and environment configuration data are often required up-front before a bug can be reported. Then the user has to register at yet another bug-tracking web site before they can log in, fill out a bug reporting form as complicated as their tax forms, and then end up with a bug in the system that they have no idea if/when anyone will look at it or notify them when it’s fixed.

It’s not that all this data isn’t critically important to the bug report, it’s that you’re raising barriers to entry that are too much for the majority of end users to handle. Instead, Michael felt that reporting a bug should be simple and easy to start with, and foster a dialogue which can then build up the necessary information to allow developers to fix the bug. This has the important side-effect of engaging the end user and lets them know that human beings are aware of this issue and working on it.

Michael’s litmus test for bug trackers is that they should be “as easy as email,” and suggested a paradigm where bugs can be reported via email and automatically imported into a web-based system where other bug metrics can be modified as the legitimacy of the bug becomes apparent.

While he didn’t mention this explicitly, I felt what he was also getting at is that community bug “mentors” get involved to herd these bug reports along and act as positive PR for the project at the same time.

Michael also mentioned a software project called SD that can migrate bugs between different bug tracking systems.

Finally, he gave examples of how end-users attack developers in bug reports and bad developer reactions to bug reports, which also destroy the trust and cooperation necessary for software to improve. A lot of this is being extremely careful with the wording in your emails/messages. For example, avoid the use of the following words:

• obvious
• just
• simply
• you/your
• why
• did you
• any profanity

and avoid joking around in your emails, which are likely to be mis-interpreted.

Thursday Morning Keynote – Portland Mayor Sam Adams

Sam Adams introduced the Civic Apps project, which is an effort sponsored by the Portland city government to encourage the development of software to empower citizens to get involved in government or help make city government work more efficiently. Through this the city has been releasing a lot of government data via web-accessible APIs, to allow for reporting of potholes using your cell phone, for example. He also announced some seed funding the city is offering to technology start-ups, and his commitment to encouraging the growth of companies working in open source.

Throughout the day people got together to work on various applications, mostly web-based. Some city IT and software development employees were around to act as resources and answer questions.

Puppet for Beginners

Puppet is a configuration management system you can use to manage and configure your server infrastructure. It is a domain-specific language (DSL) written in Ruby and I was impressed with its readability and simplicity – a stark contrast to my understanding of cfengine, which is an older tool in this domain.

One of the neat things about Puppet is you can use querying tools to inspect the current state of resources (user accounts, etc) and it will output the information in Puppet syntax. This can simplify the process of getting started with it and learning how to use it. There is also a –no-op flag you can set when testing puppet configurations to ensure it only shows what changes *would* be made instead of making them directly, allowing for incremental development of Puppet configurations without modifying live systems.

You generate dynamic configuration files using ERB, a simple templating system also used by default in Ruby on Rails. Puppet resources can have dependencies and you can dump the dependency info into a graphviz format similar to what BitBake uses (-g). You can define behaviors easily such as telling a parent service to re-start when a configuration file for it changes (e.g, restart apache when one of the vhost config files is modified).

Security in Puppet client/server communication is done with SSL certificates. They seemed to have a number of simple utilities for creating and managing SSL certs, which is otherwise a bit complicated.

USB 3.0 Open Source Support

Sarah Sharp (also of Intel’s OTC) gave a talk on the Linux kernel support she works on for USB 3.0. It was a pretty high-level overview, covering some new features of USB 3.0, including:

• Increased speed (10x more than USB 2.0, ~ 5 Gb/sec)
• Link-level power management which can put devices to sleep automatically
• A scatter-gather transmission protocol to improve throughput
• Stream ids which allow devices to prioritize I/O (e.g, to allow spinning hard disks to collate data transfers close to where the head is currently positioned)
• Basic USB 3.0 support is in 2.6.31, many more features should make it in by 2.6.36
• Sarah is currently working on some UAS (USB Attached SCSI) drivers for next-generation NAS devices

Your Internets Are Leaking

This was a fun end-of-the-day presentation which demonstrated ways that attackers can snoop on wireless traffic to collect data and passwords. During the presentation, various tools such as tshark (a command-line wireshark app) and urlsnarf were used to display traffic currently happening at the conference. Data such as IRC “private” messages, IM traffic, URLs people were visiting, etc were displayed.

This was then followed by tips on how to mitigate some of these practices. One thing I didn’t realize was that many web applications offer https support but just don’t advertise it. So just by changing your bookmarks to use https (after verifying it works) can help increase your security. Likewise for AIM logins. Another tool for IM is Off-The-Record (OTR), which encrypts your communication from point to point.

An overview of SOCKS proxies and commercial privacy VPN services were also discussed.