Jack is Going, Going, Going...Gone!

Hey Everyone,

I think I told most people that I wanted to tell privately so it's time to tell the list and out myself to the public. August 14th will be my last day at Red Hat and of temporary daily direct involvement in the Fedora project.

In 1997 I got my first taste of Linux, Red Hat Linux 4.2, to be exact. It was in the basement lab of the university that I was doing research at during the second half of my freshman year in high school. It was at that point the most fun and challenging thing I had ever done, struggling to get the kernel to work with the crappy Matrox (I think) video card that was the only spare piece of anything in that lab. I aimlessly wandered down that path I had no idea that jumping down the rabbit hole would lead to the 12 most pleasantly wondrous and amazing years of my life.

Over the last 12 years this love affair has grown stronger and I have had the unbelievable good fortune to travel the world, see amazing places, explore amazing ideas, meet and work with some of the planet's greatest, smartest and most passionate people and play my part to help turn Linux, Open Source, Red Hat, Fedora and the concepts of free open and democratic commons of content and technology from relatively unknowns into the great revolution of our age. I have spent the better part of the last 6 years working for Red Hat on Fedora and Fedora-related projects in directed efforts to improve both the state and awareness of those things I mentioned. Red Hat has been a warm home and family to me and I am as much glad as I am in awe of how ferociously dedicated we have been to our noble principles of freedom and truth, while having accomplished, ascertained and executed and what I have been able to imbibe, about so many diverse concepts, over these last few years. What niche and facet have we not touched? What direction or device have we not influenced? What proclivity have we not affected? For this, I am proud.

Fedora has been my brother since the day it was conceived. The more energy and time I invested into Fedora, to help it grow and mature, the more it paid me back by proving to be the best platform for innovation, and letting me be involved in that cause. Starting a community is no small order and keeping it going all these years take passion on the part of those willing to undertake the task. We have learned what it means to be a community, to live, breathe, eat and be true to community. To provide, so that others can have, to build so that others can build upon and to be selfless so that we can embrace others and more importantly so that others can embrace us, virtual strangers, and feel welcome. It has been my distinct pleasure to work with every single precious member of the Fedora community, from all over the world to help build a very deep and intimate relationship with the concept of community. We have accomplished such great feats, arising from a turbulent and tumultuous genesis and virtually transformed and flipped the world and the hearts and minds of people in a few short years. We have become the paramount archetype of community. How many have communities emulated and continue to emulate our success? How many have our ideas spawned? How many have been lucky to be as true and real as we have? For this, I am grateful.

The best part has been the people. I can't count on 100 sets of hands the number and names of all the wonderful people that have affected me. When I was on the Fedora University Tour, my speech was called "Crash: How a Billion Little Collisions Defines Everything," and it was about how working in a community and in real life, we are the sum total of the people we interact with. I don't think one can find a better metaphor and if I stick to my axiom then I can truly consider myself rich. Every person I met and spent time with in the office, at a meeting, show, conference or elsewhere, and online has helped shape my character, both personal and professional, for the better. As a lover of people I am both thankful for the interactions we have had and excited for what the future holds. I owe thanks to many, like I said, even 100 hands can't count, but I will try and pay homage to some of my closest, dearest and most influential friends over the last few years.

First and foremost, Tom "Spot" Callaway, for urging me to get involved way back when things started and helping me score a gig at Red Hat.

Greg DeKoenigsberg, for being a friend, a mentor and a visionary; if I can say one thing about Greg it's that he "gets it" when no one else does, he can put it in words, and above all else, he's real.

Max Spevack, because I can write a whole book of reasons to thank Max, who has been a dear friend, a true buddy, a team player and a team leader.

Karsten Wade, for being the most chillin guy you will ever find, and for being my west coast trade show and conference booth buddy.

Jim Gleason, for being first a friend for 9+ years of NYLUG and then a mentor and being someone who cares.
Michael Tiemann, for being a genius, for always giving me something to think about and someone to look up to.
John Flanagan, for being my first manager at Red Hat, always commenting on my hats and being an all around great guy and Jeff Needle, for being the guy who would let me wander into his cubicle and talk about nothing for hours on end.
Mo Duffy, for being the best artist and designer in the world. Go Kitty Go!
The original Red Hat QA team, Ed Rousseau, Bill Peck, Marty, John Poelstra, John Lockhart and Zack Cerza for letting me encroach on their cube area and steal one when I was an intern.

Jesse Keating, for being awesome, for being the workhorse upon much of which the foundations of Fedora are built, and for being a cool guy who I spoke to for almost 2 years online and helped me with everything before I ever got a chance to meet him and buy him a drink.

Luke Macken, my brother and friend, for all those games of Star Wars pinball on the 3rd floor and for being the most uber hacker the world has ever seen.

Arlinton Bourne, for being a true friend throughout college and following my advice to join Red Hat, where the hood at?

Paul Frields, for being a great leader and a real sweetheart while still secretly being 007.
Yaakov Nemoy, for being my intern and not complaining and for being a friend who will always listen to my crazy ideas.
Arjun Roy and Mohammed Morsi and James Leddy, for being great interns as well and for accepting offers to come to Red Hat as well. Mo, who knew real Red Hatters wear Orange?
Bill Nottingham, because I like him.
Moshe Bar, for being my international hangout buddy, a great entrepreneur and being an all around great human being.
The Red Hat Anaconda team including Jeremy Katz and Peter Jones, the Desktop team including Ray Strode and Dan Williams, Fedora kernel team (a.k.a. Dave Jones), the Fedora Ambassadors, the Fedora Infrastructure team including Mike McGrath, Dennis Gilmore and Toshio Kuratomi, anyone who was ever been on the Fedora board including Rex Dieter, and Matt Domsch, anyone who ever volunteered to help at an event or show, everyone in the Westford office, everyone in the NYC office.
The Fedora Marketing team including Steven Moix, David Nalley, Bob Jensen, Jon Stanley, Rahul Sundaram, John Rose, Larry Cafeiro, Clint Savage and anyone else I'm forgetting...we done good, real good.
To the next generation of leaders in Fedora, Mel Chua, Ricky Zhou, Ian Weller and crew. It's been a pleasure getting to know and working with you, and it gives me pride to know that we have attracted a fine crew to pass the torch on to. The future is in good hands--kick some butt!
Last and certainly not least, to Matthew Szulik, a true lion, who believed in us and led us finely as a teacher and friend and Jim Whitehurst, who keeps the flame alive, the train running (although he used to run an airline) and still making time for the little guys in between calling the shots. You have both exemplified leadership and hold a sacred and hallowed place in my mind and heart. We could have only gotten this far because we stood on the shoulders of giants.

Thanks everyone for an amazing time and ride. As I move on to other ventures, I wish everyone blessing and success and hope to keep in touch. I can be reached via email jack@jackfoundation.com, Freenode IRC as themayor, and various and sundry social networks.

Fedora Marketing TNG: Project FooBar

Just wanted to keep people posted as to what's going on in Marketing and the outcome of my trip to Westford last week. As many of you know, I've been thinking about what the next steps we need to take in Fedora Marketing should be. I feel that we have come along way in terms of improving process and that we can go even further while also putting a fresh spin on things. For some time there has been discussion of a "Fedora Magazine" concept; this goes back a couple of years. I really liked the idea and it sort of stuck in my mind all these years and I was thinking we can centralize things around that format. I had a few rough ideas for kickstarting this, but mainly my motivations were to solidify policies and process for what content Marketing creates, who we create it for, the content creation schedule, and how we distribute it.

Last week, I went up to Westford to and I met with Mo and Mel and Stephen Smoogen who was in the house for a visit. We ran through a bunch of ideas and goals for Marketing's future and the idea of a Fedora journal/magazine type setup. The results can be found here: https://fedoraproject.org/wiki/Marketing/Project_FooBar. Mo came up with the name, lol.

There are 5 main goals that we are striving for as you can read on the wiki page. Centralization of Content, well scheduled, recurring and prepared content, design which is consistent with the philosophy of the Design team, standardized "official" feeds for distribution of different forms of content, mechanisms for localization and sharing the media with press or on social news sites.

Feel free to read the wiki page and add/edit, etc. I would like to know what people think of this type of format, and I know people will have many question so I would like those to be voiced now so that we can answer them and make sure we are working to build something that is useful for the community with input from the community.

By the way, here is a screenshot:

Fedora 11 and Ext4: The Straight Bits

Let's face it--We're addicted! To files that is. More importantly, we are addicted to the massively large and ever increasing storage devices upon which we store those files. Make no mistake though, like any addiction, storing content comes at a cost and usually those costs are paid at the filesystem level. We all want more space and we all want better performance when it comes to disk I/O and a junkie's wishlist never ends.

Fedora 11, when released tomorrow, will be the first distribution to boast the inclusion of ext4, the latest incarnation in the extended file system family, as default. Ext4 brings with it support for larger filesystems, larger single file size and many improvements in almost every imaginable facet. Join me for an interview with Eric Sandeen, renown file system hacker, Red Hat Engineer and Fedora Contributor as he takes on a little trip down Filesystem Alley and explains what filesystems are, where did they come from, why should we care and why they along with Fedora 11 are prepping to take over the WOOOOORLD!

1. Please give is a quick self-introduction and how you got started working on Fedora/Red Hat and filesystems.

I was an electrical engineer by education & career years ago, but in the course of that work I started fiddling with Linux - Red Hat Linux (5?) was actually the first distro I ever used. I worked at SGI for about 6 years on the XFS filesystem, and then moved to Red Hat to work on ext3, ext4, XFS, and other filesystem related bits. I feel lucky to be able to have turned a fun hobby into a paid gig. :)

2. Practically, what is a filesystem and why should the average user care about what filesystem they are using?

A filesystem is the detailed format of how the operating system stores data on disk, and how it manages reading and writing of that data. The filesystem's job, first and foremost, is to keep the user's data intact and accessible, but beyond that, extra features and speed on certain workloads may influence a user's choice of which filesystem to use.

3. Can you give us a brief history on filesystems in Linux? What have been the major milestones?

Linux started out with a very simple filesystem, the Minix filesystem. This was replaced with the "ext" filesystem around 1992. Ext2 showed up around 1993, and the later ext* filesystems been developed from that basic lineage. Around 2000-2001, there was a bit of an explosion of new journaling filesystems for Linux, including ext3, xfs, jfs, and reiserfs. Of those, I'd say that ext3, ext4, and XFS have remained in most active development to this day.

Ext4 development was started about 3 years ago to address scalability & functional limitations of ext3, working on top of the ext3 codebase. Some of the basic features came from work that ClusterFS and Bull had done for Lustre, and other development has happened on top of that. It's been a joint effort by several entities upstream, and we've all worked together to make a good filesystem.

4. In Fedora 4, the default filesystem will be ext4. Fedora 11 will be the first distro to offer ext4 as the default FS. Why is that significant?

I think Fedora has always taken pride in helping to develop new features for Linux, and pushing them as part of the distribution to get these features out to a user base. It's always a bit of a balancing act, because new software inherently has bugs, and users expect any distribution to work well, of course.

The open development process of Fedora has allowed early adopters to test & provide bug reports and feedback on ext4, and the end result, I think, is that we have a very solid ext4 codebase for F11. It was a little rough in the beginning but thanks to all the testers, and the hard work by all the upstream ext4 developers, I feel confident that we're in good shape.

5. What limitations was ext4 developed to overcome and what benefits can we expect to see? There are also new features like the addition of extents and pre-allocation. These specific features are a big win over previous filesystems. Can you tell us more?

One of the primary limitations of ext3, and motivators for ext4, was the relatively small maximum file size (2T) and filesystem size (16T). The allocator in ext3 wasn't particularly efficient either, and the direct/indirect block layout scheme caused some performance bottlenecks.

The ext4 on-disk format allows for up to 1EB filesystems with 4k blocks, although due to user space tool limitations we're still at a 16T maximum filesystem size. Work is currently underway to address this.

Ext4 also has a new allocator, called "mballoc" which can be much more efficient than ext3's old block at a time allocator.

One of the other real bottlenecks to scalability is how well a very large filesystem can be checked and repaired, and modifications to ext4's metadata layout have yielded some very impressive speedups in e2fsck's check times.

Features like extents and delayed allocation have honestly been around for a very long time on other Linux filesystems like XFS, and ext4 implemented these features in part based on that proven track record. Together these features can help give us very efficient allocation patterns.

One other thing that the extent format brings us is much faster deletion of large files compared to ext3 - something which anyone who has had to enable the "slow delete" feature of MythTV may appreciate.

Extents also allow filesystems like ext4 to efficiently track preallocated disk space, allowing applications which use preallocation calls to get more efficient allocation. The transmission bittorrent client and the libvirt tools are a couple of packages in Fedora which make use of this.

6. Fedora has been using LVM and other volume management layers for a while. In fact, Fedora helped pioneer technologies like LVM. How does ext4 play well with these? How does it facilitate use of those technologies?

To be honest, there's a lot more work to be done in this area. One of the things which has just recently been addressed upstream is LVM's ability to pass write barriers from the filesystem down to the underlying block device. Write barriers prevent write reordering by the drive. They have a bit of a performance hit, but they're needed to ensure a journaling filesystem's consistency whenever power is lost to a disk with a volatile write cache. Until very recently, LVM didn't pass these barrier requests down at all; this now works upstream for simple LVM volumes, and work is ongoing in this area.

The other area where filesystems and volume managers really need to communicate is in the geometry of the aggregate block device - ideally the filesystem wants to know about the stripe unit and stripe width of a raid5 device, for example, so that it can do efficient, well-aligned allocation and IOs. The XFS userspace utilities are able to extract this information from software raid devices and use it at mkfs time, and honestly this is something that needs to be added to e2fsprogs as well. Again, there is more work going on upstream to address this issue.

7. What are your thoughts on the future of filesystems? What do you think are the features that we should be focusing on? Are we working on pioneering any of these efforts in Fedora?

One of the big pushes is for more active protection of the user's data via checksumming at all levels, as well as management features, such as better ability to use multiple devices for a filesystem. In Linux, a lot of this type of work is being done in the new BtrFS filesystem.

Fedora 11 is a pretty exciting release for filesystems overall, because it also includes an early preview of BtrFS. Josef Bacik, one of our filesystem developers, has been putting a lot of effort into BtrFS upstream. Adventurous users who want to try out BtrFS can do so in F11, and even install the distro onto it by booting the installer with a "secret" boot argument - "icantbelieveitsnotbtr". This is a very early preview, and isn't yet suitable for more than testing for most users, but early testing and bug reporting will be very useful.

8. Do you like any other filesystems that are being used/developed, such as ZFS, which seems to be a big fan favorite and others like BtrFS, Tux3?

ZFS has a lot of nice advertised features, but it's not really available for Linux primarily due to license issues - and I'm not sure the userspace fuse implementation is optimal, but I may be biased as a kernel filesystem developer! BtrFS shows a lot of promise, and Chris Mason and his crew have been developing it at an amazing pace, in my opinion. BtrFS is a pretty fundamental re-thinking of what a Linux filesystem should be.

I can only keep so much in my brain at once, and so have not really kept up with Tux3. The other filesystem that I still think is interesting is XFS, because it has the scalability and feature set that ext4 is striving for in a mature, well-tested (though pretty complex...) code base. Of course, like anything else, it has its strengths and weaknesses. It's also a pretty different beast administratively compared to what people are used to with ext2 and ext3.

9. What is your day to day development cycle look like? Surely, work continues on ext4, but what else are you working on? What do you spend your free time doing?

I do a fair amount of work on ext4 and XFS on a daily basis, and a lot of my time is taken addressing various Fedora and Red Hat Enterprise Linux user & customer bugs. I maintain a few other filesystem-related tools for Fedora and RHEL as well, including e2fsprogs, xfsprogs, xfsdump, blktrace, fio, ffsb, fs_mark, seekwatcher.... this keeps me plenty busy!

I've recently been working on making the xfs regression test suite filesystem-agnostic so that other filesystems can use this basic framework for regression testing; it's been hugely useful for XFS development. We have about 30 tests running on other filesystems now.

There are many other bits and pieces that compete for attention every day, so there's a lot of juggling of priorities. Any filesystem corruption bugs or oopses usually rise to the top.

Free time? I have a family and a 2 kids, so there's not a lot of that! I bike and swim when I can, and to be honest some of my free time is spent.... hacking filesystems. I guess it's in my blood.

10. How are you planning to celebrate the Fedora 11 release tomorrow?

Hm, I'll probably be working on what needs to be done for F12. :)

Fedora 11 Tour

Anxious for Fedora 11. Can't wait until tomorrow to get your hands on it? Well you can take a sneak preview of what's coming by visiting the Fedora 11 Tour page. On it you'll find an overview of Fedora 11, the Fedora 11 screenshot tour, links to the various pieces of Fedora multimedia we've produced and of course information how and where to download Fedora 11 from starting tomorrow at 10am Eastern.

Fedora 11: Raise thy Mighty... Finger?

Authentication is an aspect of computing which many take for granted. What's all the fuss? you think. Username, password and that's that. In the following Q&A session with Bastien Nocera, long time Fedora Contributor and Desktop Renaissance Man, we discover that when it comes to authentication, there is more than meets the finger!

With fingerprint and other biometric authentication options gaining more popularity, its time to get more creative regarding their use. Many laptops have had built-in fingerprint readers for upwards of two years now and Fedora 11, thanks to Bastien and crew, does a solid job of making that option a viable one for Linux desktop user. How did we make this happen for Fedora 11? Will your Fedora laptop one day be able to authenticate you on the web using your finger? Will we ever get GNOME keyring to unlock using a fingerprint? What will Bastien work on next? All this and more if you keep reading below!

1. Can you please give us a quick self introduction and how you got started in Fedora.

Hey, I'm Bastien Nocera, I work for Red Hat, and I've been a GNOME contributor for 10 years. I started using Fedora when I joined Red Hat in 2002, and I've been hooked since :)

2. For at least a couple of years now, many laptop models have had built-in fingerprint readers. They never seemed to work well under Linux, despite various bits and pieces of drivers being out there. Can you tell us more about how this feature came about in Fedora 11? [note: PAM is the pluggable authentication system used on Linux machines to authenticate users. D-Bus is a message bus system, a simple way for applications to talk to one another.]

I've had a Dell laptop with the omnipresent Thomson fingerprint reader for a couple of years, and I was looking at how I could use it, and make it work out-of-the-box in Fedora. At that time, as far as I remember, the only options were the proprietary Upek bits, and thinkfinger, which was a very PAM specific solution.

Around that time, Daniel Drake mentioned that he was working on ‘libfprint’, a library to fold the support of different fingerprint readers, with different capabilities, into one supported API, for his BSc in Computer Science.

I got in at about that point. Daniel and I already had a pretty good idea on how we should be architect support for the fingerprint readers, and Daniel wrote a first pass at the ‘fprintd’ D-Bus daemon to present it at his final year project presentation.

When Daniel presented his project, he put all his code up, and I started working on the D-Bus daemon, cleaning up the API, and implementing various front-ends on top of it.

3. In order to accomplish a lot of this some significant modifications were necessary to other parts of the distro, i.e. DBus, PAM and authentication dialogs. Can you talk to us a little about what type of work needed to be done to get all the pieces to work together.

It was pretty fun getting to use some new technology. We fixed some bugs in ‘libfprint,’ re-did the public API, added developer documentation, added PolicyKit integration, added a PAM module, and wrote a nice UI for all that in the GNOME control-center.

We were pretty much done, and then Ray Strode added support to GDM to get multiple PAM stacks. This meant that the user could choose between logging in with a password, or using the fingerprint reader.

4. What are some of the issues that remain to be worked on if any?

Most of the remaining problems fall slightly outside the scope of this project. ‘libusb1’ needs a bit of reworking to handle devices appearing and disappearing more gracefully. ‘libfprint’ needs bug fixes for existing drivers and more drivers (never-ending story). Finally, we need PAM to die die die (or add multiple PAM stacks support to more front-ends).

5. Where do you see the future of this going? Do you expect that we will one day down the line see encrypted filesystems which require biometric authentication to decrypt? How about extending this capability to authentication on the web?

Hehe. The architecture is there to support those, although security concerns will possibly override that. We're still thinking of ways to integrate LDAP authentication, and get the PAM module to unlock the GNOME keyring for us.

6. You are well known as a long time and very involved Fedora contributor. What are some of the other projects you worked on for this release?

I worked on the (oh-so-controversial) volume control, updated Bluetooth management tools, and wrote/updated a driver for Wacom Bluetooth tablets.

But work has already started for Fedora 12. With Dan Williams, we already added Bluetooth PAN support to NetworkManager, and we're working on the front-end bits now. I'd like to do some more work on my old flames, Totem and Rhythmbox.

I also have a drawer full of Bluetooth devices that I need to work on. I'm half-way done adding Geolocation to Firefox, for Linux platforms, using GeoClue. Hopefully I'll be able to finish that and work on some more devices.

7. What are you going to do to celebrate the release of Fedora 11?

Probably raise my glass to it, and getting cracking on Fedora 12!

Add This

I recently added something new to my blog and feed which seems very useful to me and I figured I would share it. I used to have a bunch of buttons to help people submit stories to different social news sites like digg, link aggregators like delicio.us and or to save bookmark them online or other similar services. This took up a lot of space under each post and barely ever worked correctly. Certain links would always end up missing and it would never render right.

Today, I added an "addthis" bar to the bottom of each post. You can see it down below. See it? Its really handy and really cool and when you click on it it will open new overlay or page and let you select where you want this post to go, whether its email, print or sites like digg, facebook, newsvine, technorati, etc. Its pretty cool I suggest everyone give it a try to help simplifying you blog and feed management.

Fedora 11 Podcast Series #5 - Presto with Jonathan Dieter

Presto! Wow, what just happened? Was that a magic trick? Well there is no magic trick here today, but what we do have is the latest in the Fedora 11 Podcast Series, an Interview with Fedora Contributor Jonathan Dieter on one of the coolest new features on Fedora 11 - presto! Presto allows you to use deltarpms to download only binary ‘diffs’ from whatever packages you already have on your system. For example, if only one file in a released update was change, all you would be downloading would be that one changed file as opposed to the whole new RPM. Starting in Fedora 11 you can use presto and the yum-presto plugin to enable this functionality when downloading updates.

Presto with Jonathan Dieter [7.2MB Ogg Vorbis]

Remember when I said no magic was involved? We actually did have to use some magic to record this podcast because Jon lives far far away--in Lebanon! Using some of that good ole’ Fedora voodoo we were able to get Jon’s thoughts on Presto, how it and deltarpms work, how this idea came to be and what working in and being a Fedora contributor has meant and continues to mean to him and just a little bit about what’s goin' on in Lebanon. Just another example of how great people all around the world come together to build a project, a community and a home called Fedora.

Fedora 11: Virtual(ization) Reality

Cutting edge virtualization technology has always been one of Fedora's strong suits and Fedora 11 looks to continue that trend. In an interview with Daniel P. Berrange, Red Hat Virt Team Engineer and Fedora Virtualization guru, we talk about the many key upgrades to virt technology in F11 focusing on areas of usability, performance and security. Fedora 11 will premiere the latest in secure and powerful virtualization technology available to users and developers. With so much to look forward to Fedora 11, it's sure to make your virtualization dreams a reality.

1. Please introduce yourself, and tell us about your work in virtualization and how you got started.

I'm one of the lead developers for the libvirt project and am actively involved in many related areas of open source development (qemu/kvm, xen, gtk-vnc, virt-manager, to name but a few). I also co-maintain many of these packages in Fedora and RHEL, along with many others in Red Hat's virtualization team.

More than three years ago (shortly after transferring into Red Hat's Engineering team, from consulting services) I was working on the OLPC project. We needed a way to easily test the OS images we were building without needing real hardware. As a proof of concept, I hacked up a simple GTK application to run images them under QEMU. At around the same time Daniel Veillard had started the libvirt project and there was a desire for a desktop application to manage Xen using libvirt. So I switched over to the virtualization team, wrote virt-manager for Fedora 6, and my involvement in all areas of open source virtualization grew from there.

2. Many people view the work being done on virtualization as a feature set of major importance and significance. Can you give us a brief overview of some of the changes we can expect to see in Fedora 11?

The open source virtualization development effort is so large now, that it is useful to discuss each stream in turn.

At the lowest layer is obviously the Linux kernel and KVM/QEMU. There has been a major acceleration of development in QEMU and push to merge KVM into the official QEMU source repository. There's ever continuing work on performance, stability, scalability and reliability in KVM. PCI device passthrough is one new feature we're highlighting for Fedora 11. The return of Xen Dom0 was not to be, as the Dom0 paravirt_ops merge with the upstream Linux kernel is still an ongoing process.

At the middle layer is libvirt, providing a consistent management API across different virtualization technologies. New features in libvirt, since F10, include PCI device passthrough for Xen and KVM, the sVirt security driver using SELinux to protect KVM guests from each other, thread safety of all libvirt APIs, improved scalability, reliability and debugging for the libvirtd daemon and support for SCSI HBAs and copy-on-write volumes in the storage management APIs.

The top layer covers end user tools such as virt-install and virt-manager. virt-manager is undergoing a significant (and ongoing) overhaul of its user interface. The first improvements arriving for Fedora 11 are in the guest installation process and storage management capabilities. As guest installation is first task most users try, ensuring this is simple and reliable is key to making a good first impression. Guest desktop interaction is another historical pain point which has been a focus for improvements in Fedora 11.

With every release we also try to make a significant step forward in security of the virtualization stack. In Fedora 11 the focus has been on SELinux to protect guests from each other and SASL to authenticate VNC users.

3. There have been some large changes in virt-manager and libvirt, which are at the core of the user experience when it comes to virtualization. Can you talk to us more about those?

The guest installation process and desktop interaction are the most critical areas for making a good first impression. In the virt-manager re-design the wizard used for installing new guests has been streamlined, cutting out three redundant steps. Where possible, it will automatically detect the type of operating system being installed and choose the best configuration options to optimize for this OS, no longer requiring the user to figure this out for themselves.

The installation process now directly utilizes the libvirt storage management APIs to allow easy creation of files in a variety of formats (raw, qcow2, vmdk, etc), allocation of new local disk partitions or LVM volumes and access to LUNs exported by iSCSI targets. This is particularly useful when remotely managing virtualization hosts, allowing regular administrator tasks to be performed from the virt-manager UI without resorting to command line SSH sessions.

The mouse pointer has been a constant source of trouble for virtualization management applications. Getting the guest mouse pointer to track the host pointer is essentially impossible with the standard emulated PS/2 mouse. The solution is to provide a pointer device that supports absolute motion events, instead of relative events that the PS/2 mouse provides. For KVM and Xen, this means adding a USB tablet device, but historically Xorg has not been able to automatically configure this correctly. This is resolved with Fedora 11 guests, finally providing a guest pointer that moves in perfect sync with the host, not requiring the pointer to be confined to the guest window.

Users with non-US layout keyboards have also had a hard time getting their guests to support input of accented/special characters. The VNC protocol has now been extended to allow the hardware keycodes to be passed directly from the VNC client to the guest OS without any intermediate translation step. This should allow the guest OS complete control over the keyboard layout mapping, without a need for any special settings on the host.

The final piece of work was to increase the guest desktop resolution. The real Cirrus video card that QEMU emulates would never have done more than 800x600, but there are tricks that can be done in a virtual world. Thus a simple change to the Xorg cirrus driver allows it to detect that it is using a Cirrus card emulated by QEMU and increase the guest desktop resolution to 1024x768. Still not great by today's standards, but better than before. Longer term plans involve replacing the cirrus driver in QEMU with something more virtualization friendly.

4. Also, as people should note, there has been a lot of work done surrounding KVM and getting that well integrated into Fedora. How has that work been going, and has anything significant been done in that area in this release?

Fedora was the first major Linux distribution to integrate KVM back in the Fedora 7 release. It became the default virtualization technology in Fedora 9, when it became clear we could no longer maintain the separate Xen host kernel until it was merged in the upstream Linux kernel. The great benefit of KVM from an distro integration point of view, is that it is there by default in all new Linux kernels. All that was required in Fedora was to turn on the module build and make sure the modules are always loaded when compatible CPUs are found. libvirt and virt-manager have also both supported KVM since it was first added to Fedora. Thus there hasn't been a need for much additional integration work for KVM. The focus has simply been on improving features available to KVM users via libvirt and virt-manager.

5. Glauber Costa has also done significant work merging KVM and QEMU. Can you explain to us what QEMU is, and why the choice was made to merge it with QEMU? How that is of benefit to the user base?

Earlier Fedora releases have suffered from the divergence of QEMU and KVM code bases. Upstream QEMU has had releases almost 1 year apart, while KVM has been releasing at least once a month, if not more, using snapshots of the QEMU source repository. Thus the features available in QEMU were far behind those available in KVM even though they both shared the same fundamental code base and upstream development stream. It also doubled the work package maintainers had to do for security and bug fixes.

Since Fedora 10 though, the upstream QEMU community has accelerated its release schedule significantly and many of the KVM features have been merged back into the main QEMU code base. Thus we judged that the time was right to attempt to ship a single package containing both QEMU and KVM built from a single code base. For users this means that parity of features between QEMU and KVM, while the reduced burden on our Fedora package maintainers, ensures more timely security and bug fixes. Glauber also took the opportunity to split out all the virtual BIOS files and ROMs from QEMU into separate packages and ensure all are fully built from source using appropriate upstream source releases.

6. Virtualization and security are discussed hand in hand these days, and as the ability to create and use virtualized machines expands there are many security risks involved. Can you speak a bit to the work that was done to improve security both at the kernel and user levels?

In each Fedora release we try to make at least one significant step forward in the security of our virtualization technology. In Fedora 8, libvirt gained support for secure remote management using TLS for encryption and x509 client certificates for authentication, while GTK-VNC, QEMU, KVM and Xen were also all extended to add a VNC extension for TLS encryption providing a secure remote desktop. In Fedora 9 libvirt was further extended to support SASL enabling secure remote management with Kerberos for authentication and PolicyKit for local desktop authentication. Fedora 9 and 10 also increased use of SELinux to protect the host operating system from a compromised or malicious QEMU/KVM process.

The latter still did not provide any protection between guests, so one compromised QEMU process would still potentially be able to compromise another on the same host. Thus James Morris started work on a project known as sVirt, the first results of which are appearing in Fedora 11. The focus has been to provide isolation between guests running on a single host. libvirt directly integrates with SELinux to ensure every QEMU process it launches runs within a dedicated security context, only able to access its own assigned disk images. This protection is enabled by default on all Fedora 11 hosts using libvirt for management. As well as the security benefits, the end user experiance is improved because libvirt will automatically manage SELinux labelling for all guest disks, eliminating a major source of bug reports from previous Fedora releases.

7. These features have all evolved over time in previous Fedora releases and Fedora, as a distribution, has always been a leader in the virtualization realm. Let's talk a bit now about the actual process of developing these features. How many of the improvements and enhancements to virtualization have come about as a direct result of the work done previously? Also, what does the future look like?

Virtualization technology in Fedora is reaping the benefit of very active upstream projects and the significant developer resources of Red Hat's Virtualization Engineering team. The combination of these provide great opportunities for new features to have their debut in Fedora releases.

The ideas for new features come from many sources, some from Fedora end-user experiences and consequent bug reports, some magically arrive on cue from upstream projects, while others are things that look to be important for future RHEL releases. With the PCI device passthrough feature in F11, the core support was all already done by the upstream KVM community. This is a important feature for future RHEL, so Red Hat put resources into a F11 feature to add support to libvirt for PCI passthrough with KVM and Xen and then expose this in virt-manager.

The feature aiming to improve the guest desktop interaction was a result of the persistent stream of bug reports from Fedora users. We had been considering ways to address this over course of several Fedora releases, but it was not until Fedora 11 that all the pieces of the solution finally came together from the various upstream projects.

The security improvements in virtualization have a different origin. Very few end users ever file explicit bug reports asking for the addition of more authentication / encryption features or to use more SELinux. If anything users ask for ability to more easily turn off existing security features. This is a case where the user is not always right. They do want more security, but they just don't know it yet!

As a developer it is necessary to be very proactive with security improvements. This can be particularly challenging work because the solutions often involve working across multiple upstream communities.

Take the VNC SASL authentication feature in Fedora 11 as an example. The first step was to write a specification for a new VNC security extension, have it reviewed and get a code for it allocated by RealVNC. Work on QEMU was required to implement the server side. Work on GTK-VNC was needed for client side. For management tools, work on libvirt was required to get the new security type enabled for guests it launches and finally virt-manager was extended such that it knows how to login. That's five different projects involved for one feature. This is only practical by having a close working relationship with all the upstream communities and carefully coordinating the work there so it all arrives in time for the next Fedora release.

For the future we're happy that libvirt gained support for managing VirtualBox recently and likely to soon have a driver for the Open Nebular cloud management project. Expect more advancements in sVirt, allowing for tighter controls on what a virtual machine can do, for example, ability to restrict network access of guests. libvirt will also gain the ability to manage host network configuration in Fedora 12, removing the need to manually configure bridge devices. Container based virt may make a more formal appearance in Fedora 12 as the native Linux container (LXC) support improves in the kernel and libvirt. The overhaul of the virt-manager user interface also continues.

8. Working on virtualization must be awfully time consuming and involved. Do you enjoy it? What do you do to get away from the pressures of hacking?

Working on open source virtualization technology is a great experience because it is a really interesting and challenging field, having plenty of talented developers to work with and learn from. There is plenty of work still to be done at all levels of the stack from kernel/hypervisor right through to end user applications and not nearly enough time to do it all. I'm fortunate to be able to spread my work between upstream projects, the Fedora community and RHEL releases and maintenance.

As for free time? What free time? :-) I try to find time for a photography, with 4 out of my 5 cameras still using film, rather than digital.

The Sound of Fedora 11

An Interview on Fedora 11's enhanced Audio Control with Lennart Poettering

Where would we be without sound? It's the most primitive of communication methods, and yet it has spawned so much technology around it. Whether you're a musician, a DJ, riding a bus to work, or even just stuck in a cubicle listening to the radio somewhere, sound has become an integral part of your daily experiences. When Fedora 11 lands, along with it will land a number of enhancements to the sound subsystem, including unified volume control, per stream and per device monitoring, and proper Bluetooth audio support. I recently caught up with Lennart Poettering, Red Hat Desktop Team Engineer and resident audio guru. Here's what he had to say about the upcoming improvements and what the future holds:

1. Please introduce yourself and give us a brief intro to how you started working on the upcoming audio improvement in F11.

I am Lennart Poettering and have been working for Red Hat in the Desktop Group for two years now this month. I live in Berlin, Germany.

PA has been part of Fedora since F8. Since then we used to ship two volume control appications: the GNOME volume control and a PA (Pulse Audio) specific tool (pavucontrol). The latter was mostly a showcase what can be done with PA and I wrote it mostly as a demo, not because I thought it was any good as an UI.

Of course having these two volume control UIs in Fedora was a situation that badly needed fixing. Especially since both UIs exposed too many unnecessary options: the GNOME volume control exposed a lot of low-level hardware-specific features that only a tiny minority of people actually really understood, and the PA volume control exposed a lot of low-level software features that a slightly larger minority of people only actually really understood.

Now during the last year we reached a point were the feature set of PA for volume controls became very complete (with such things as arbitrary meta data on every stream/device, per-stream and per-device monitoring, hardware volume range extension, "flat" volumes and lots of other stuff) and Jon McCann with help from Bastien Nocera finally took up the work to
fix the UI situation.

They basically designed the new UI from scratch with input from usability experts. It implements many of the features the old pavucontrol tool did, but in a much nicer, streamlined way. Also it integrates sound theme/event sound control with general audio configuraton and volume control in a single UI tool.

2. Can you give us some background on the upcoming changes to the audio subsystem in the Fedora 11 Release?

If you want to know more about the Volume Control, I'd just refer to the Feature page:

https://fedoraproject.org/wiki/Features/VolumeControl

We moved PA 0.9.15 into F11, a nice overview over the new features you can find here:

http://0pointer.de/blog/projects/oh-nine-fifteen.html

However that overview is a bit out-of-date. There are quite a few additional features that went into 0.9.15, most prominently full Bluetooth Audio support: Together with Bastien Nocera and the BlueZ guys I worked to make Bluetooth audio easily accessible -- the bluetooth applet now exposes an easy dialog that allows you to pair and activate a bluetooth headset. After that is done it will automatically appear in PulseAudio. If you need to reactivate it later, you can do that with a simple click in the applet menu. It works surprisingly well. It even works fine for lip-sync video. Which is kind of magic, given that Bluetooth Audio doesn't actually offer any timing interfaces, so syncing up audio with video is not really possible. I spent a lot of time to make sure it does work nonetheless, and it seems to work on the majority of headphones although I cannot say for sure if it does for all of them.

3. Where did the ideas to change all this stuff come from? Didn't audio always work in Fedora?

Depends what you mean by 'work'. Sure, basic audio output worked. But in many ways what we had on Linux was not comparable to what MacOS or Windows supported. And it still isn't in many ways. However in other ways we have now surpassed those competitors.

A lot of the changes we introduced with PA are not directly visible to the user. For example the so called 'glitch-free' logic in PA is very important for a modern audio stack, however the normal user will never notice it -- except maybe because when we introduced it initially a lot of driver bugs got exposed that people were not aware of before because that driver functionality (usually timing related) was not really depended on by any application. In fact even now many of the older drivers expose broken timing that makes usage with PA not as much fun as it could be.

A more detailed explanation of this 'glitch-free' logic you may find here:

http://0pointer.de/blog/projects/pulse-glitch-free.html

Both Windows Vista and MacOS X have similar g-f logic in their audio stacks, however with PA we brought it to the next step. For example, we implemented this logic in a zero-copy fashion and with arbitrary sample types. This allows us to pass PCM data through our pipelines without ever having to copy/convert it unless we really have to.

So yes, as you might have noticed I spend a lot of time to get low-level internals right. And I like to speak about it, even though most people are not aware of all those technical details and how awesome this all is. ;-) That said, this stuff isn't perfect yet and could need more improvements.

But it's not all just in the low-level details. Also on higher levels we got inspired by how our competitors do things. For example the new "flat" volume logic was pioneered in Vista, and we have now adopted a similar logic in PA. It's a great way to reduce the complexities of volume control by 'merging' a few of the sliders in the pipeline. It thus solves the "So which slider is now causing my volume to be too low?" a bit. But also here, there's more work to be done.

It's not all just getting inspired by our competitors. There are a lot of genuinely new features in PA that none of them have (at least to my knowledge). For example, in PA we have 'spatial' event sounds. I.e. if an event sound sound is triggered by a mouse click/dialog at the left side of the screen the sound is generated more from the left speakers, and similar for the right side. This is of course mostly a toy. But I think a useful one ;-) .

Listing all the fancy features PA has would certainly be a bit too much for this interview. So I'll leave it with this... ;-)

Generally, we get inspiration from everywhere. And sure, as long as the most basic music playback was enough for you audio did always work in Fedora. But OTOH, when we started with the integration of all of these new audio features into Fedora two years ago the audio stack was still at a point of what was modern in the 90's. With the new features of the new volume control and PA we are working on bringing Linux audio to what is modern today.

4. Can you also give us a comparison of our new audio framework in reference to other audio frameworks and audio subsystem models that are out there?

There are many frameworks out there. On Free Software systems PA doesn't really have any competitor. Some people think that JACK is one, but it actually is not. JACK is clearly focussed on audio production and not very useful on the desktop otherwise. For example, it is strictly designed to provide very low-latency at the price of power consumption. This is the right thing to do for audio production but not on the general desktop. Logic like 'glitch-free' (see above) makes a lot of sense for the usual desktop audio since it allows flexible adjusting of the latency to what is needed. If used properly it can be used to decrease the interrupt rate to 1/s, while still allowing instant reaction to user input. Since most PCs these days are laptops theses kind of power consumption related features are very important.

One of the current weaker points of Audio on Linux is that we have this clear separation of JACK for audio production and PA for desktop/embedded. Other operating systems have managed to make this a bit smoother by having a single stack for both. This however actually has both advantages and disadvantages.

To improve the situation for now we focussed on making PA and JACK cooperate better. In F11 when JACK needs low-level access to an audio device it will tell PA so and PA will comply and release the device.

This should make switching between the two sound systems easier though of course this is no perfect solution. Given the lack of manpower further integration is unlikely to happen anytime soon -- though both the JACK guys and I seem not generally opposed to something like that.

Now, if you compare our audio stacks with those of the big other operating systems (Windows and MacOSX), then besides the fact that they usually integrate desktop audio and audio production better than we do (as mentioned) there are many things we are better in and many they are better in. We certainly have more flexibility: i.e. depending on your application you can access audio on a lot of different levels: you can access ALSA directly if you need very low-level control, or via PA for desktop level control. You have APIs like GStreamer for media streaming and so on.

This flexibility however translates to more complexity in many ways, and a hodgepodge of API styles. (OTOH Apple's CoreAudio actually isn't as streamlined as many MacOS proponents like us to think.) The documentation for our APIs is usually much worse then theirs. We really need some improvements in that area. Featurewise, PA usually has better networking related features then those counterparts. But there's a lot of features they have right now we lack.

Other Unixes, such as FreeBSD and OpenSolaris are still stuck with OSS (Open Sound System) audio. In F11 we finally switched OSS off by default (though you can still reenable it via some minor hackery). OSS was the predecessor of ALSA. Thankfully it is now fully obsolete on Linux. OSS is mostly a design from the early nineties. It has received only minor updating since then. It is no way comparable to what we now have on Linux or even what MacOS or Windows provide. (Although is has some very vocal fans which like to write me hate mails because I say things like this)

5. This work all started in earlier releases dating all the way to even Fedora 8, if I am correct. How has all this stuff progressed and evolved from then? What was done in previous releases that enabled building upon for this release?

Fedora 8 was the first release where we integrated PA. In Fedora 9 we stabilized PA support. In F10 we integrated the 'glitch-free' logic which turned out to be quite a bumpy ride given that it exposed a lot of timing related driver bugs. In F11 g-f has now been made more robust and most of the more modern audio drivers should now be fixed. Also we have now started to push PA support more into the UI, like with this new volume control.

6. What are the plans for the future, if any, in this particular space in the distro?

I am working on multiple things for F12. Firstly there will be a couple of more low-level changes to PA. The core will be made more threaded. Right now, we run most things in one 'main' thread and do low-level audio I/O in one thread for each audio card. My plan for F12 is to split that one 'main' thread up into as many threads as possible. This should make PA more robust for a couple of operations, and make latencies more reliable.

Then, I am working on considerably beefing up PA's usage of the low-level hardware volume controls. For example, many cards have seperate low-level volume sliders for "Speaker", "Master", "PCM" (and more) that are in the line from the PCM data we stream to the speakers. PA currently exposes only one of those sliders (usually "Master"). My plan is to 'multiply' those sliders and create a single 'product' virtual slider from them that has a better granularity and a larger range. This rework will also introduce input/output switching and probably more.

What has already landed in PA's git repository is support for UPnP A/V. When used in conjunction with Zeeshan Ali's Rygel UPnP MediaServer implementation this allows streaming any application's audio to a any UPnP MediaRenderer (including PS3/Xboxes and all those 'Internet Radio' devices). This is actually pretty neat. Later on we hope to make PA a Media Renderer as as well as a MediaServer. This nicely compliments our current Apple RAOP support.

And there's a lot of other things planned. We'll see how much of that will be ready for F12. I don't like to talk too much about upcoming features and planned code if I don't have anything to show yet, so I'll leave it at this.

And then there's always a little project of mine that is called 'libsydney' that is intended to be a portable, modern and friendly PCM API. During the last months I focussed more on PA itself though.

7. Do you feel that work like this helps enhance the desktop experience on Linux in general and strengthens the cause of the Linux Desktop, or is it more all in day's work?

I think that PA is the way forward for audio on the Linux desktop. It may have its deficiencies -- but everything has. We still have some way to go, but I believe that a modern audio layer is really important for the Linux Desktop to succeed.

And no, it doesn't feel at all in a day's work. It always is a great feeling to see how PA got incorporated into so many distributions and how it is now used by so many people. I am pretty sure that only if you hack on Linux software you get this in this ways.

8. Speaking of all in a days work, what are things do you usually work on? What do you most enjoy doing outside of work.

Red Hat basically hired me to help improving audio on Linux. So that's what I am doing during work.

Outside of work spend my time with photopgraphy. And I am trying my best to travel to interesting places as much as I can and my time off allows.

Thank you Lennart for an excellent interview, ideas and insight. We look forward to hearing more from you. Get it--hearing more, he works on sound, okay I give up.