One of the things I’ve been working on since I got back from Taipei has been helping with the implementation of vertical-text in Gecko which WEBVTT needs to support. The small way in which I helped out was to implement the “writing-mode” property in the style system. Basically this is just getting the Gecko CSS parser/scanner to understand the new property and be able to process it correctly. This was fairly easy to implement due to Daniel Holbert (:dholbert) giving me an awesome walkthrough of the things that would need to change in order for this to work and also an example bug where he did a similar thing. So after half a day doing the initial code and a few rounds of review it landed in the tree and we’re now one step closer to vertical text!
If you’d like to learn more about how the style system works you can head over to the style section of the Gecko overview page which also has a lot of really good information on how Gecko works in general. I won’t duplicate information here, but I will talk about a couple of the things I found interesting in the style system.
- CSS property values are stored in a computed form and there are translation mechanisms to go back and forth between the two.
- The computed values of CSS properties live in structs, namely nsStyleStructs, and CSS properties that tend to be set together live in the same struct. For example — when a user sets the font-family property they are likely to also set the font-size, font-weight, etc, so storing the computed values of these properties together makes sense.
- Single instances of nsStyleStructs, i.e. one set of related properties, are shared across many different DOM nodes as DOM nodes are more likely to share one set of CSS properties then they are to have different sets of properties. For example, the vast majority of times that a page sets font properties the entire page will share these properties. This cuts down on memory usage. These structs are immutable and when the same CSS property needs to be set differently on another set of DOM nodes a new struct will be created for the new set of properties.
- Each nsStyleStruct has a CalcDifference() function that gives hints to Gecko about when it needs to update the rendering of the page based on the CSS properties changing, i.e. if it needs to reconstruct the frame or the text needs to be reflowed, etc.
In the future I’m hoping to help with more layout things. Hopefully, I can do the same kind of bug for the “text-orientation” CSS property, which is also needed for vertical text. I haven’t started as of yet as it’s still kind of up in the air in the spec. Right now I’ve also started work on another easy bug to reorganize the reftests in layout/reftest/form. Hopefully, I can get that done sometime next week. We’ll need reftests for WEBVTT as well so this will be a good bug where I can learn about them.
So I’m back home now. The flight home was a bit shorter, about two hours, thankfully. Interestingly enough I feel more jet lagged now that I’m home then I was in Taiwan. I’ve heard that going east is worse then going west, so that might be the case for me right now. The rest of the rendering week, after I blogged last Tuesday, was just as awesome as the first part.
How Gecko Does X
I got to sit in on a lot more talks about “How Gecko does X” — like how the graphics engine works, how the layout system works, and my favourite in particular, how cycle collection works. Kyle Huey did an excellent job explaining how the Gecko cycle collector works. He gave us this paper as forward reading about it (it’s a little dense, but definitely worth reading). I’ll try to do a blog post in the future on what I learned/will learn about it.
David Baron and Adam Roach also gave an awesome talk on how the W3C and IETF work. WEBVTT is my first major exposure to open specifications that I’ve had and so I’ve been interested in all the hows/whats/whys of open specifications and the politics behind them.
Initial WEBVTT Support
It wasn’t all fun and games over in Taiwan though. We were also doing a lot of work. We finally got bug 833385 landed near to the end of the week. This means that we have support for all the new DOM elements that WEBVTT introduces such as: HTMLTrackElement, TextTrack, TextTrackList, TextTrackCue, and TextTrackCueList. We ran into a random inexplicable bug when we were doing full tries on the code, just before landing it. Ralph and I went to work debugging it (had to use an ASAN build) and we ended up discovering that it happened in a very rare situation where the cycle collector nulls out the HTMLMediaElement’s TextTrackList member while the HTMLMediaElement is still alive. This results in a situation where HTMLMediaElement::FireTimeUpdate() is called just before it is about to be deleted and since we weren’t doing null checks on the call to TextTrackList::Update() we would crash. After we got that fixed we were all green.
That leaves just bug 833382 left before we get initial support for WEBVTT. It was going well last week and we got an r+ from Chris Pearce. Now we just need an r+ from Boris and we should be good to go. It might take a few more rounds of review before that happens, but I’m optimistic we will be able to get this landed within a week or so.
One of the major problems I was having in Taiwan was trying to get a clean diff for 833382. The problem centered around the fact that up until now, mainly in my previous open-source class, we decided to use a git branch as a main point of ‘integration’. We’ve all been working off this branch for a while. The history of this branch has been so ridiculous and the code necessary for 833382 depends on so many other parts of the code that have been touched by, well everyone, that it was pretty much impossible for me to do a clean diff or even get a good rebase. To rebase this beast I would have had to go through 150+ commits, each having merge conflicts… So I ended up just making a clean branch off master and manually moving all the code that I needed over to the new branch and doing a diff like that. I’ll probably be staying away from these kinds of ‘integration’ branches in the future in order to ensure that my repo history can be more linear. Easier to get diffs that way.
The other thing I’ve been dealing with in the last few days is some code we landed back in February that was spotted to not be up to par by Boris. The issue is with some of the CSS selectors that we are using to style WEBVTT text — namely, we are using slow CSS selectors which is bad. This is the first that I’ve heard about some CSS selectors being slower than others, although that’s not suprising as I’m not super-super knowledgeable about CSS. Mozilla even has a page devoted to this that you can check out here. Ralph and I put together a patch yesterday to deal with this which will land today most likely. I’ll have to update 833382 to reflect those changes today as well.
CSS Parser Hacking
I also sat in on the vertical-text layout meeting as it is of particular interest to WEBVTT. WEBVTT requires the ability to have vertical-text and so far in Gecko we don’t have this. Apparently vertical-text has been kind of a thorn in the side of the layout team for a long time as it’s been particularly hard to implement. However, there is a major push now to get it done, so that’s great. In accordance with this Daniel Holbert asked me if I wanted to do some stuff for vertical-text in the CSS parser, I accepted and got my first layout bug! So I’ll be hacking around in the CSS parser and layout section of Gecko more in the future, hopefully.
Dave also told me the other day that he’s figured out an area of WebMaker that I can start contributing to, so I’m excited about that. I’ll be starting in about two weeks on this. I’ll most likely be splitting my time like 70/30 or something like that for WEBVTT and WebMaker. We talked briefly about it and so my understanding isn’t 100%, but what I got from our talk is that I will be implementing a kind of wrapper around an HTML5 video element that will allow Popcorn Maker to be able to work with it. From my understanding Popcorn Maker works with many different video formats/sources and so it needs a uniform interface to work with all these different videos. That’s where the wrapper comes in. It allows Popcorn Maker to work with many different video formats and sources without worrying about the particulars. However, all this might be completely wrong as I might have misunderstood some things from our brief conversation… So don’t take my word for it! At any rate I’ll do another blog post about it when I get more information.
Until next time.
A month back I found out that Caitlin and I were being invited to Mozilla’s Web Rendering Week in Taiwan. Needless to say I was shocked. Ralph Giles, who has been working with us closely on WEBVTT and was/is our main contact with Mozilla for WEBVTT, kindly invited us out — thanks Ralph!!
Therefore, a lot of this week has been spent getting ready for the trip — getting ready to travel and trying to push on some final bugs to get initial support for WEBVTT into Nightly before we leave.
Getting that initial support entailed a few things, all of which haven’t happened yet… Don’t worry though! If we don’t get it finished before Taiwan we will definitely get it finished while we’re in Taiwan. We need(ed) too:
- Get the WEBVTT library to a state where we felt comfortable tagging 0.5 and landing it in Gecko.
- Land the DOM implementation of HTMLTrackElement and the TextTrack objects.
- Land the parser and Gecko integration bug.
Getting WEBVTT to 0.5 was pretty easy. The only extra thing we wanted to add before 0.5 was support for the new <lang> tag. We needed a new string list pop function for that as well. I implemented that this week, which also exposed a bug that we hadn’t found yet because we didn’t have a proper test for it. The bug happened when we had an <rt> tag that was not enclosed within a <ruby> tag. In these situations the <rt> tag should not be processed. We weren’t handling it correctly and it was crashing the parser. That got landed as well yesterday. We tagged 0.5 and Caitlin is working on the bug to land it in Gecko now. WEBVTT 0.5 contains a lot of good things like sec-critical bug fixes and library usability fixes.
Ralph is very close to landing the DOM implementation. That should be happening within the next few days. The parser integration is getting close as well. I’ve been working on changing the code that converts the C node tree to a DOM tree from a recursive algorithm to an iterative one. It’s been a little tricky as we need to keep track of where we are in the C tree as we iterate over it, as well as where we are in the tree of DOM nodes that we are creating. We also need to keep track of the ‘last branching parent’ so that we’re able to tell when we need to go back up the tree. I have a solution that I put together that will hopefully be able to get an r+. I’ll have it up for review tonight on the bug.
The other major thing I did this week was give a presentation to CDOT about WEBVTT. You can check out the presentation over on my GitHub page. It was recorded so maybe you all can see it at some point when it gets posted! I’ll link at that time.
Until next time.
For the last two weeks we’ve been working steadily on the WEBVTT parser. Most of the work being done now is related to getting the parser integrated into Firefox. We’re building on top of the original bug filed on bugzilla by Ralph and are now using it as a co-ordination bug for the five other bugs we’ve split it up into. The “bug sections” that we’ve split it up into are:
- Integrating the webvtt build system into the Firefox build system.
- Adding a captions div to an nsVideoFrame so the captions can be rendered on screen.
- Creating a “Text Track Decoder” that will function as the entry point for Gecko into the webvtt parser.
- Creating new DOM bindings i.e. TextTrack, TextTrackCue, TextTrackCueList.
- Creating DOM tests using Mochitest for the new DOM bindings.
You can check out a more in depth break down of our bug plan here.
The other major thing that a few of us in the class have been engaged in is the review of the 0.4 parser code. The review is still in it’s early to mid stages, so we have a lot more to do on that. I’ve been participating there by filing and commenting on issues and fixing a few of the bugs that have surfaced.
We’ve also moved the parser code over to the mozilla webvtt repository on GitHub (yay!) and have landed the 0.4 parser code there in a dev branch. After the review is done it will be landed on the master branch.
I’ve been working on the Text Track Decoder for the parser integration into Firefox. This part of the integration functions as an entry point into our parser for Gecko.
How It Works
The short version of how the Text Track Decoder works is that when an HtmlTrackElement receives new data from a vtt byte stream it passes it off to it’s WebVTTLoadListener (Text Track Decoder) which then calls our webvtt parser to parse the chunk of the byte stream it just reveived. The WebVTTLoadListener also provides call back functions to the parser for passing back cues when the parser has finished them or for reporting errors when the parser encounters them. The final function that the WebVTTLoadListener facilitates is converting the cues that have been passed back in the call back function to the various DOM elements that represent a webvtt_cue and then attaching those to either the HtmlTrackElement’s track, in the case of the cue settings, or the HtmlTrackElement’s MediaElement’s video div caption overlay (phew), in the case of the parsed webvtt cue text node tree.
What We’ve Done
The first order of business that we took care of in getting this done was to ask Chris Pearce, who works very closely with Firefox’s media stuff, to give us a high level overview of what we would need to accomplish in order to get this working. That was sent in the form of an email which my Professor, Dave Humphrey, then kindly posted on our bug (I forgot to do so!).
We then quickly went about implementing Chris’s initial steps that he talked about. We’ve done steps 1 – 4 so far:
- The HtmlTrackElement::LoadListener has been moved to it’s own file and renamed WebVTTLoadListener.
- The HtmlTrackElement now has a WebVTTLoadListener reference which is initialized in LoadResource.
- WebVTTLoadListener now manages a single webvtt parser which is created and destroyed along with it.
- WebVTTLoadListener now provides call back functions to the parser for returning finished cues and reporting errors.
We’ve also added three convenience functions to turn webvtt cue stuff into the DOM bindings. These are:
- cCueToDomCue – Transforms a webvtt cue’s settings into a TextTrackCue (almost done).
- cNodeToHtmlElement – Transforms a webvtt node into an HtmlElement; recursively adds HtmlElements to it’s children if it is converting an internal webvtt node (not done at all!).
- cNodeListToDocumentFragment – Transforms the head node’s children into HtmlElements and adds them to a DocumentFragment (pretty much done).
The call back function for returning cues now:
- Calls the cCueToDomCue function and adds the resulting HtmlTextTrackCue to it’s owning HtmlTrackElements cue list.
- Calls the cNodeListToDocumentFragment and adds the resulting DocumentFragment to the caption overlay.
Right now we’ve run into some problems in figuring out how to work with the Firefox code. I’ve listed those in my recent WIP update on the bug. Other then implementing those steps I’ve just been getting acquainted with the Firefox code that we have to touch and figuring out the basics of how it’s all going to fit in. I think we’ve gotten a big chunk of it done so far, mostly the overall frame of how it’s going to work as well as turning a webvtt cue’s settings into a TextTrackCue. I’ve also met the deadlines and goals that I set for myself at the beginning of this semester, so I’m fairly happy. Going forward I think I know enough now to ask intelligent questions about how to solve the problems that I listed in the WIP, so that’s what I will be doing in the coming weeks when I get stuck.
As always, I’m ever confident that we will finish the project!
So after a vacation that wasn’t long enough it is a new semester and that means, you guessed it, more WEBVTT development! We had our first class and have roughly got an idea of what we need to do and the direction that we need to head in.
This semester I will still be contributing to the WEBVTT parser on and off, but the bulk of my focus will be turned towards getting it to work with Gecko which is the layer in Firefox that will be utilizing it. I don’t know much about Gecko or the Firefox code base yet so learning that is a major hurtle I’m going to face this term.
Our class is working within two week development time frames and we will need to be doing three demos of what we have done to the class. I’ve signed up for my three demo time slots and these are roughly the gaols that I’d like to meet for those demos:
- January 28: I will have the refactored changes done to the parser that I discussed in my previous blog post; I’ll have compiled Firefox and gotten a basic understanding of the direction I need to be headed in for Gecko.
- March 4: A good understanding of Gecko and the Firefox code base; I’ll have written the first one or two Gecko integration iterations; I’ll have communicated any design changes that I think the parser needs to make in order to work with Gecko easily to the team.
- April 1: I’ll have gotten the Gecko integration to a good place to ship in 1.0
Some of the problems that I think I will have are mainly with getting started on the integration with Gecko. I’ve never looked at Gecko before so I’m coming into this with no idea of how the Firefox code fits together. I predict that I will probably have to ask questions of people in the know who have worked on it before, so that means that I am going to need to go out and talk to people in the community. After I get rolling on the code and have a better understanding it’ll probably be more easy going.
The biggest amount of work that I see us doing this semester in terms of incorporating the parser into Gecko is changing the parser based on our new knowledge of how it is going to fit into Gecko. Once it’s in there we will be able to see more clearly how the end architecture of the parser should be. This means that we will probably have a few changes to the public API and architecture of the parser.
There are a few areas to tackle in getting this parser done and we’ve all decided to own one part of it or the other. Even though we’ve done this I suspect that we will eventually contribute to all aspects of the code in small ways. We will have to work as a close knit team and help each other out in order to get the WEBVTT parser to 1.0 by the end of the term.