Fedora Marketing TNG: Project FooBar

Just wanted to keep people posted as to what's going on in Marketing and the outcome of my trip to Westford last week. As many of you know, I've been thinking about what the next steps we need to take in Fedora Marketing should be. I feel that we have come along way in terms of improving process and that we can go even further while also putting a fresh spin on things. For some time there has been discussion of a "Fedora Magazine" concept; this goes back a couple of years. I really liked the idea and it sort of stuck in my mind all these years and I was thinking we can centralize things around that format. I had a few rough ideas for kickstarting this, but mainly my motivations were to solidify policies and process for what content Marketing creates, who we create it for, the content creation schedule, and how we distribute it.

Last week, I went up to Westford to and I met with Mo and Mel and Stephen Smoogen who was in the house for a visit. We ran through a bunch of ideas and goals for Marketing's future and the idea of a Fedora journal/magazine type setup. The results can be found here: https://fedoraproject.org/wiki/Marketing/Project_FooBar. Mo came up with the name, lol.

There are 5 main goals that we are striving for as you can read on the wiki page. Centralization of Content, well scheduled, recurring and prepared content, design which is consistent with the philosophy of the Design team, standardized "official" feeds for distribution of different forms of content, mechanisms for localization and sharing the media with press or on social news sites.

Feel free to read the wiki page and add/edit, etc. I would like to know what people think of this type of format, and I know people will have many question so I would like those to be voiced now so that we can answer them and make sure we are working to build something that is useful for the community with input from the community.

By the way, here is a screenshot:

Fedora 11 and Ext4: The Straight Bits

Let's face it--We're addicted! To files that is. More importantly, we are addicted to the massively large and ever increasing storage devices upon which we store those files. Make no mistake though, like any addiction, storing content comes at a cost and usually those costs are paid at the filesystem level. We all want more space and we all want better performance when it comes to disk I/O and a junkie's wishlist never ends.

Fedora 11, when released tomorrow, will be the first distribution to boast the inclusion of ext4, the latest incarnation in the extended file system family, as default. Ext4 brings with it support for larger filesystems, larger single file size and many improvements in almost every imaginable facet. Join me for an interview with Eric Sandeen, renown file system hacker, Red Hat Engineer and Fedora Contributor as he takes on a little trip down Filesystem Alley and explains what filesystems are, where did they come from, why should we care and why they along with Fedora 11 are prepping to take over the WOOOOORLD!

1. Please give is a quick self-introduction and how you got started working on Fedora/Red Hat and filesystems.

I was an electrical engineer by education & career years ago, but in the course of that work I started fiddling with Linux - Red Hat Linux (5?) was actually the first distro I ever used. I worked at SGI for about 6 years on the XFS filesystem, and then moved to Red Hat to work on ext3, ext4, XFS, and other filesystem related bits. I feel lucky to be able to have turned a fun hobby into a paid gig. :)

2. Practically, what is a filesystem and why should the average user care about what filesystem they are using?

A filesystem is the detailed format of how the operating system stores data on disk, and how it manages reading and writing of that data. The filesystem's job, first and foremost, is to keep the user's data intact and accessible, but beyond that, extra features and speed on certain workloads may influence a user's choice of which filesystem to use.

3. Can you give us a brief history on filesystems in Linux? What have been the major milestones?

Linux started out with a very simple filesystem, the Minix filesystem. This was replaced with the "ext" filesystem around 1992. Ext2 showed up around 1993, and the later ext* filesystems been developed from that basic lineage. Around 2000-2001, there was a bit of an explosion of new journaling filesystems for Linux, including ext3, xfs, jfs, and reiserfs. Of those, I'd say that ext3, ext4, and XFS have remained in most active development to this day.

Ext4 development was started about 3 years ago to address scalability & functional limitations of ext3, working on top of the ext3 codebase. Some of the basic features came from work that ClusterFS and Bull had done for Lustre, and other development has happened on top of that. It's been a joint effort by several entities upstream, and we've all worked together to make a good filesystem.

4. In Fedora 4, the default filesystem will be ext4. Fedora 11 will be the first distro to offer ext4 as the default FS. Why is that significant?

I think Fedora has always taken pride in helping to develop new features for Linux, and pushing them as part of the distribution to get these features out to a user base. It's always a bit of a balancing act, because new software inherently has bugs, and users expect any distribution to work well, of course.

The open development process of Fedora has allowed early adopters to test & provide bug reports and feedback on ext4, and the end result, I think, is that we have a very solid ext4 codebase for F11. It was a little rough in the beginning but thanks to all the testers, and the hard work by all the upstream ext4 developers, I feel confident that we're in good shape.

5. What limitations was ext4 developed to overcome and what benefits can we expect to see? There are also new features like the addition of extents and pre-allocation. These specific features are a big win over previous filesystems. Can you tell us more?

One of the primary limitations of ext3, and motivators for ext4, was the relatively small maximum file size (2T) and filesystem size (16T). The allocator in ext3 wasn't particularly efficient either, and the direct/indirect block layout scheme caused some performance bottlenecks.

The ext4 on-disk format allows for up to 1EB filesystems with 4k blocks, although due to user space tool limitations we're still at a 16T maximum filesystem size. Work is currently underway to address this.

Ext4 also has a new allocator, called "mballoc" which can be much more efficient than ext3's old block at a time allocator.

One of the other real bottlenecks to scalability is how well a very large filesystem can be checked and repaired, and modifications to ext4's metadata layout have yielded some very impressive speedups in e2fsck's check times.

Features like extents and delayed allocation have honestly been around for a very long time on other Linux filesystems like XFS, and ext4 implemented these features in part based on that proven track record. Together these features can help give us very efficient allocation patterns.

One other thing that the extent format brings us is much faster deletion of large files compared to ext3 - something which anyone who has had to enable the "slow delete" feature of MythTV may appreciate.

Extents also allow filesystems like ext4 to efficiently track preallocated disk space, allowing applications which use preallocation calls to get more efficient allocation. The transmission bittorrent client and the libvirt tools are a couple of packages in Fedora which make use of this.

6. Fedora has been using LVM and other volume management layers for a while. In fact, Fedora helped pioneer technologies like LVM. How does ext4 play well with these? How does it facilitate use of those technologies?

To be honest, there's a lot more work to be done in this area. One of the things which has just recently been addressed upstream is LVM's ability to pass write barriers from the filesystem down to the underlying block device. Write barriers prevent write reordering by the drive. They have a bit of a performance hit, but they're needed to ensure a journaling filesystem's consistency whenever power is lost to a disk with a volatile write cache. Until very recently, LVM didn't pass these barrier requests down at all; this now works upstream for simple LVM volumes, and work is ongoing in this area.

The other area where filesystems and volume managers really need to communicate is in the geometry of the aggregate block device - ideally the filesystem wants to know about the stripe unit and stripe width of a raid5 device, for example, so that it can do efficient, well-aligned allocation and IOs. The XFS userspace utilities are able to extract this information from software raid devices and use it at mkfs time, and honestly this is something that needs to be added to e2fsprogs as well. Again, there is more work going on upstream to address this issue.

7. What are your thoughts on the future of filesystems? What do you think are the features that we should be focusing on? Are we working on pioneering any of these efforts in Fedora?

One of the big pushes is for more active protection of the user's data via checksumming at all levels, as well as management features, such as better ability to use multiple devices for a filesystem. In Linux, a lot of this type of work is being done in the new BtrFS filesystem.

Fedora 11 is a pretty exciting release for filesystems overall, because it also includes an early preview of BtrFS. Josef Bacik, one of our filesystem developers, has been putting a lot of effort into BtrFS upstream. Adventurous users who want to try out BtrFS can do so in F11, and even install the distro onto it by booting the installer with a "secret" boot argument - "icantbelieveitsnotbtr". This is a very early preview, and isn't yet suitable for more than testing for most users, but early testing and bug reporting will be very useful.

8. Do you like any other filesystems that are being used/developed, such as ZFS, which seems to be a big fan favorite and others like BtrFS, Tux3?

ZFS has a lot of nice advertised features, but it's not really available for Linux primarily due to license issues - and I'm not sure the userspace fuse implementation is optimal, but I may be biased as a kernel filesystem developer! BtrFS shows a lot of promise, and Chris Mason and his crew have been developing it at an amazing pace, in my opinion. BtrFS is a pretty fundamental re-thinking of what a Linux filesystem should be.

I can only keep so much in my brain at once, and so have not really kept up with Tux3. The other filesystem that I still think is interesting is XFS, because it has the scalability and feature set that ext4 is striving for in a mature, well-tested (though pretty complex...) code base. Of course, like anything else, it has its strengths and weaknesses. It's also a pretty different beast administratively compared to what people are used to with ext2 and ext3.

9. What is your day to day development cycle look like? Surely, work continues on ext4, but what else are you working on? What do you spend your free time doing?

I do a fair amount of work on ext4 and XFS on a daily basis, and a lot of my time is taken addressing various Fedora and Red Hat Enterprise Linux user & customer bugs. I maintain a few other filesystem-related tools for Fedora and RHEL as well, including e2fsprogs, xfsprogs, xfsdump, blktrace, fio, ffsb, fs_mark, seekwatcher.... this keeps me plenty busy!

I've recently been working on making the xfs regression test suite filesystem-agnostic so that other filesystems can use this basic framework for regression testing; it's been hugely useful for XFS development. We have about 30 tests running on other filesystems now.

There are many other bits and pieces that compete for attention every day, so there's a lot of juggling of priorities. Any filesystem corruption bugs or oopses usually rise to the top.

Free time? I have a family and a 2 kids, so there's not a lot of that! I bike and swim when I can, and to be honest some of my free time is spent.... hacking filesystems. I guess it's in my blood.

10. How are you planning to celebrate the Fedora 11 release tomorrow?

Hm, I'll probably be working on what needs to be done for F12. :)

Fedora 11 Tour

Anxious for Fedora 11. Can't wait until tomorrow to get your hands on it? Well you can take a sneak preview of what's coming by visiting the Fedora 11 Tour page. On it you'll find an overview of Fedora 11, the Fedora 11 screenshot tour, links to the various pieces of Fedora multimedia we've produced and of course information how and where to download Fedora 11 from starting tomorrow at 10am Eastern.

Fedora 11: Raise thy Mighty... Finger?

Authentication is an aspect of computing which many take for granted. What's all the fuss? you think. Username, password and that's that. In the following Q&A session with Bastien Nocera, long time Fedora Contributor and Desktop Renaissance Man, we discover that when it comes to authentication, there is more than meets the finger!

With fingerprint and other biometric authentication options gaining more popularity, its time to get more creative regarding their use. Many laptops have had built-in fingerprint readers for upwards of two years now and Fedora 11, thanks to Bastien and crew, does a solid job of making that option a viable one for Linux desktop user. How did we make this happen for Fedora 11? Will your Fedora laptop one day be able to authenticate you on the web using your finger? Will we ever get GNOME keyring to unlock using a fingerprint? What will Bastien work on next? All this and more if you keep reading below!

1. Can you please give us a quick self introduction and how you got started in Fedora.

Hey, I'm Bastien Nocera, I work for Red Hat, and I've been a GNOME contributor for 10 years. I started using Fedora when I joined Red Hat in 2002, and I've been hooked since :)

2. For at least a couple of years now, many laptop models have had built-in fingerprint readers. They never seemed to work well under Linux, despite various bits and pieces of drivers being out there. Can you tell us more about how this feature came about in Fedora 11? [note: PAM is the pluggable authentication system used on Linux machines to authenticate users. D-Bus is a message bus system, a simple way for applications to talk to one another.]

I've had a Dell laptop with the omnipresent Thomson fingerprint reader for a couple of years, and I was looking at how I could use it, and make it work out-of-the-box in Fedora. At that time, as far as I remember, the only options were the proprietary Upek bits, and thinkfinger, which was a very PAM specific solution.

Around that time, Daniel Drake mentioned that he was working on ‘libfprint’, a library to fold the support of different fingerprint readers, with different capabilities, into one supported API, for his BSc in Computer Science.

I got in at about that point. Daniel and I already had a pretty good idea on how we should be architect support for the fingerprint readers, and Daniel wrote a first pass at the ‘fprintd’ D-Bus daemon to present it at his final year project presentation.

When Daniel presented his project, he put all his code up, and I started working on the D-Bus daemon, cleaning up the API, and implementing various front-ends on top of it.

3. In order to accomplish a lot of this some significant modifications were necessary to other parts of the distro, i.e. DBus, PAM and authentication dialogs. Can you talk to us a little about what type of work needed to be done to get all the pieces to work together.

It was pretty fun getting to use some new technology. We fixed some bugs in ‘libfprint,’ re-did the public API, added developer documentation, added PolicyKit integration, added a PAM module, and wrote a nice UI for all that in the GNOME control-center.

We were pretty much done, and then Ray Strode added support to GDM to get multiple PAM stacks. This meant that the user could choose between logging in with a password, or using the fingerprint reader.

4. What are some of the issues that remain to be worked on if any?

Most of the remaining problems fall slightly outside the scope of this project. ‘libusb1’ needs a bit of reworking to handle devices appearing and disappearing more gracefully. ‘libfprint’ needs bug fixes for existing drivers and more drivers (never-ending story). Finally, we need PAM to die die die (or add multiple PAM stacks support to more front-ends).

5. Where do you see the future of this going? Do you expect that we will one day down the line see encrypted filesystems which require biometric authentication to decrypt? How about extending this capability to authentication on the web?

Hehe. The architecture is there to support those, although security concerns will possibly override that. We're still thinking of ways to integrate LDAP authentication, and get the PAM module to unlock the GNOME keyring for us.

6. You are well known as a long time and very involved Fedora contributor. What are some of the other projects you worked on for this release?

I worked on the (oh-so-controversial) volume control, updated Bluetooth management tools, and wrote/updated a driver for Wacom Bluetooth tablets.

But work has already started for Fedora 12. With Dan Williams, we already added Bluetooth PAN support to NetworkManager, and we're working on the front-end bits now. I'd like to do some more work on my old flames, Totem and Rhythmbox.

I also have a drawer full of Bluetooth devices that I need to work on. I'm half-way done adding Geolocation to Firefox, for Linux platforms, using GeoClue. Hopefully I'll be able to finish that and work on some more devices.

7. What are you going to do to celebrate the release of Fedora 11?

Probably raise my glass to it, and getting cracking on Fedora 12!

Add This

I recently added something new to my blog and feed which seems very useful to me and I figured I would share it. I used to have a bunch of buttons to help people submit stories to different social news sites like digg, link aggregators like delicio.us and or to save bookmark them online or other similar services. This took up a lot of space under each post and barely ever worked correctly. Certain links would always end up missing and it would never render right.

Today, I added an "addthis" bar to the bottom of each post. You can see it down below. See it? Its really handy and really cool and when you click on it it will open new overlay or page and let you select where you want this post to go, whether its email, print or sites like digg, facebook, newsvine, technorati, etc. Its pretty cool I suggest everyone give it a try to help simplifying you blog and feed management.

Fedora 11 Podcast Series #5 - Presto with Jonathan Dieter

Presto! Wow, what just happened? Was that a magic trick? Well there is no magic trick here today, but what we do have is the latest in the Fedora 11 Podcast Series, an Interview with Fedora Contributor Jonathan Dieter on one of the coolest new features on Fedora 11 - presto! Presto allows you to use deltarpms to download only binary ‘diffs’ from whatever packages you already have on your system. For example, if only one file in a released update was change, all you would be downloading would be that one changed file as opposed to the whole new RPM. Starting in Fedora 11 you can use presto and the yum-presto plugin to enable this functionality when downloading updates.

Presto with Jonathan Dieter [7.2MB Ogg Vorbis]

Remember when I said no magic was involved? We actually did have to use some magic to record this podcast because Jon lives far far away--in Lebanon! Using some of that good ole’ Fedora voodoo we were able to get Jon’s thoughts on Presto, how it and deltarpms work, how this idea came to be and what working in and being a Fedora contributor has meant and continues to mean to him and just a little bit about what’s goin' on in Lebanon. Just another example of how great people all around the world come together to build a project, a community and a home called Fedora.