Tagged: software

Connected Wellness: Build problems..

For the last week I’ve been trying to fix some Xcode build problems that we’ve been having. After I upgraded the project to 2.3 Carl tried to run it on the simulator and it didn’t work. After doing a lot of digging I figured out that I only provided an armv7 static library of Cordova. What this meant was that we couldn’t run it on the simulator because it runs on the Mac which uses an x86 processor. What I ended up having to do was add two versions of the library to the project, one compiled for x86, and one compiled for armv7, and then make the project conditionally link them in based on if we were building for x86 or armv7. This was kind of tricky because I didn’t know at first how Xcode manages this stuff. I finally discovered that you can add a build condition underneath other linker flags in the target’s properties that will allow you to do this.

After this was all done I had to take care of adding in the 2.3 version of the AppDelegate and ViewController because these weren’t in the project when I switched over to 2.3. They had been taken out, by me I believe, I forget why. Once that was done I began to get some problems with the linker not being able to resolve some of the symbols in the Cordova library that had to do with extending classes. Fixing this required me to add “-force_load” to the linker flags. This specified to Xcode to load the entire library, whereas if you don’t put that, its default mode is to not load extensions… strange.

There’s still one last problem with the build that Carl has informed me of. The problem isn’t currently surfacing on my own machine so we’re going to have to do some more digging.

Final, thing that I did was attempt to simplify the Command hierarchy by making ReturnableCommand just a Returnable and then having BluetoothCommand implement the Command protocol. This simplifies it a little because ReturnableCommand no longer has to be an abstract class. It also makes sense because an object that is returnable to the Cordova library isn’t necessarily a command. This way in the future it will be possibly to have many different types of objects be able to return a plugin result back to Cordova.

Advertisements

Connected Wellness: Upgrading Cordova 2.0 to 2.3

We’ve gotten quite a lot of work done this week in the Connected Wellness project. However, we’re getting close to a point where we might not be able to do any work until we get Apple Developer licenses. Most of the preliminary development has been done and we now need to start testing cod; to do that we need those licenses! I heard from Carl that it’s being worked on and we should have them soon, so I’m hopeful.

Other than that we had a big review on Catilin’s packet parser implementation. I probably spent a few hours just reviewing that alone. I also went around cleaning up a few of the issues such as removing commands that we aren’t able to implement yet due to not being able to fully control Bluetooth on Apple Devices and implementing the last few commands.

The biggest thing that I’ve done this week is to include the Cordova library into the mercurial repo and get it set up so that it is incorporated into the Xcode project out of the box. This way we don’t have to set it up manually every time we clone a fresh repo (that was getting tiring). It took a little bit of figuring out how to do this, but what ended up working was to include the library file along with the header files of the library (Apple requires header files for statically linked libraries to be somewhere in the project). After I did this I needed to add the folder that holds the library and header files to the required frameworks list of the project, modify the header search paths to include the directory where the library’s headers are, and set the linker up to link with the library by giving it the path to the .a file. All these options can be found in the “build settings” tab of the build target in Xcode i.e. the project file.

After I got the library incorporated into the repo I went about moving our current code to support Cordova 2.3 (we were on 2.0 before). This proved to be very easy. The only things that changed were the include paths of the Cordova headers when #import-ing them, the function signature of the method that Cordova calls in the plugin, and the way that Plugin Results are passed back to Cordova. It was actually easier moving to 2.3 than it was to code 2.0 because they’ve made it a lot more easier and intuitive to work with.

You can check out that code in the pull request here.

After that pull request lands there’s only going to be one last thing to code – the stopListeningCommand. That will probably not be hard. I’m hoping that we will be able to get those Apple Developer licenses early next week so we can continue powering through this. It is due by the end of February…

WEBVTT Update: Parser Review, Integration into Firefox

For the last two weeks we’ve been working steadily on the WEBVTT parser. Most of the work being done now is related to getting the parser integrated into Firefox. We’re building on top of the original bug filed on bugzilla by Ralph and are now using it as a co-ordination bug for the five other bugs we’ve split it up into. The “bug sections” that we’ve split it up into are:

  • Integrating the webvtt build system into the Firefox build system.
  • Adding a captions div to an nsVideoFrame so the captions can be rendered on screen.
  • Creating a “Text Track Decoder” that will function as the entry point for Gecko into the webvtt parser.
  • Creating new DOM bindings i.e. TextTrack, TextTrackCue, TextTrackCueList.
  • Creating DOM tests using Mochitest for the new DOM bindings.

You can check out a more in depth break down of our bug plan here.

The other major thing that a few of us in the class have been engaged in is the review of the 0.4 parser code. The review is still in it’s early to mid stages, so we have a lot more to do on that. I’ve been participating there by filing and commenting on issues and fixing a few of the bugs that have surfaced.

We’ve also moved the parser code over to the mozilla webvtt repository on GitHub (yay!) and have landed the 0.4 parser code there in a dev branch. After the review is done it will be landed on the master branch.

Firefox Integration

I’ve been working on the Text Track Decoder for the parser integration into Firefox. This part of the integration functions as an entry point into our parser for Gecko.

How It Works

The short version of how the Text Track Decoder works is that when an HtmlTrackElement receives new data from a vtt byte stream it passes it off to it’s WebVTTLoadListener (Text Track Decoder) which then calls our webvtt parser to parse the chunk of the byte stream it just reveived. The WebVTTLoadListener also provides call back functions to the parser for passing back cues when the parser has finished them or for reporting errors when the parser encounters them. The final function that the WebVTTLoadListener facilitates is converting the cues that have been passed back in the call back function to the various DOM elements that represent a webvtt_cue and then attaching those to either the HtmlTrackElement’s track, in the case of the cue settings, or the HtmlTrackElement’s MediaElement’s video div caption overlay (phew), in the case of the parsed webvtt cue text node tree.

What We’ve Done

The first order of business that we took care of in getting this done was to ask Chris Pearce, who works very closely with Firefox’s media stuff, to give us a high level overview of what we would need to accomplish in order to get this working. That was sent in the form of an email which my Professor, Dave Humphrey, then kindly posted on our bug (I forgot to do so!).

We then quickly went about implementing Chris’s initial steps that he talked about. We’ve done steps 1 – 4 so far:

  • The HtmlTrackElement::LoadListener has been moved to it’s own file and renamed WebVTTLoadListener.
  • The HtmlTrackElement now has a WebVTTLoadListener reference which is initialized in LoadResource.
  • WebVTTLoadListener now manages a single webvtt parser which is created and destroyed along with it.
  • WebVTTLoadListener now provides call back functions to the parser for returning finished cues and reporting errors.

We’ve also added three convenience functions to turn webvtt cue stuff into the DOM bindings. These are:

  • cCueToDomCue – Transforms a webvtt cue’s settings into a TextTrackCue (almost done).
  • cNodeToHtmlElement –  Transforms a webvtt node into an HtmlElement; recursively adds HtmlElements to it’s children if it is converting an internal webvtt node (not done at all!).
  • cNodeListToDocumentFragment – Transforms the head node’s children into HtmlElements and adds them to a DocumentFragment (pretty much done).

The call back function for returning cues now:

  • Calls the cCueToDomCue function and adds the resulting HtmlTextTrackCue to it’s owning HtmlTrackElements cue list.
  • Calls the cNodeListToDocumentFragment and adds the resulting DocumentFragment to the caption overlay.

Right now we’ve run into some problems in figuring out how to work with the Firefox code. I’ve listed those in my recent WIP update on the bug. Other then implementing those steps I’ve just been getting acquainted with the Firefox code that we have to touch and figuring out the basics of how it’s all going to fit in. I think we’ve gotten a big chunk of it done so far, mostly the overall frame of how it’s going to work as well as turning a webvtt cue’s settings into a TextTrackCue. I’ve also met the deadlines and goals that I set for myself at the beginning of this semester, so I’m fairly happy. Going forward I think I know enough now to ask intelligent questions about how to solve the problems that I listed in the WIP, so that’s what I will be doing in the coming weeks when I get stuck.

As always, I’m ever confident that we will finish the project!

Connected Wellness: Threads, Cordova Lib, and Java to iOS

So it’s been another eventful week for the Connected Wellness iOS team at CDOT. We’ve made a lot of progress towards having the project completed, but there are still some major outstanding issues that we need to tackle. The PacketParser has yet to be completed (one of the most complex portions of the plugin), we’ve still yet to hear back from A&D about low energy BT devices, which we need in order to use the CoreBluetooth framework provided by Apple, and we’re still running into some problems regarding the amount of control that the CoreBT framework gives us. The CoreBT framework does not expose a way to start, stop, or enable Bluetooth in the device. These are all things that the Android plugin is able to do. Although it is not completely necessary to implement these (the user can start it manually) it would be nice to provide it to the user as a convenience.

Threads

To continue my discussion from last week in regards to the Invoker and dispatching Commands asynchronously on threads – it has been ridiculously easy to implement this. Instead of going through and creating our own WorkQueue and thread management classes I’ve implemented NSOperationQueue and NSInvocationOperation inside of the Invoker class. NSOperationQueue is provided by Apple and has all the functions to properly manage thread execution on a work queue and NSInvocationOperation allows us to specify the target and selector that the threads started from the work queue will invoke. Check out how easy it was to do here.

Cordova Lib

During this last week we ran into a big problem of everyone checking code into the repo that wasn’t compiling or causing problems. Yesterday I went through and fixed all the compilation problems. We’re planning to have a more strict review process in the future. One of the major problems that I encountered was the Cordova Lib not being properly compiled and linked to the project. This was caused because for some reason Cordova 2.0 doesn’t support armv6, so I had to remove that from the Cordova project as well as the iOS port project. The other big issue is that currently the Cordova library is not included in the repository. This means that when a fresh copy is cloned you have to go through the process of manually downloading PhoneGap and incorporating it into the project. So in the next day or so I’m going to set up the repo to include the compiled archive and point the project to it by default.

Java to iOS

The other major problem that I ran into this week was specific to the port from Java to iOS. In the Java version the concrete commands reside as inner classes in the DevicePlugin class. Due to this they all have access to the BluetoothServer stored within the DevicePlugin class. Moving over to iOS we cannot use inner classes and so the commands need to be provided with access to the server in some other way. To compensate this I’ve introduced a new command, BluetoothCommand, which sits between the concrete commands and the ReturnableCommand. BluetoothCommand exposes the base plugin in the ReturnableCommand as a MedicalDevicePlugin as well as exposing a property on it to access the BluetoothServer on the MedicalDevicePlugin. You can check out that here. Now that I’m thinking about it a BluetoothCommand doesn’t necessarily need to work with a MedicalDevicePlugin… what we’d need to do then in order to abstract it more is to create a base class like BluetoothPlugin that will expose a BluetoothServer property. I don’t really know if we should do this though as I’ve read a lot recently about the dangers of over designing code. I’ll post back next week about what comes from this.

Next Week

By the end of next week I’m hoping that we have most of the pieces in place to start testing. The PacketParser will be done and the few other commands that are left to be implemented will be done. Personally I will be working on some of the outstanding commands such as the listen and isSupported() commands.

Connected Wellness Dev Update

We’ve gotten a lot done in the last week for the iOS port of the Bluetooth Plugin. You can check out my work here.

  • I’ve finished the MedicalDevicePlugin class.
  • Wrote the Command Protocol.
  • Wrote a template for the ReturnableCommand class.
  • I started on the WorkQueue class, but Carl told me to shift from that to the Invoker class because WorkQueue will be removed sometime in the near future.
  • I also wrote the Invoker class but left out the bulk of the implementation as it uses the WorkQueue, so there is no point to code it for now as I don’t know what kind of design we will be using.
  • I’ve also pushed a lot of bug fixes to the repo.

Objective-c vs. Java

There were a lot of changes going to iOS from Android. The biggest one was that a lot of the architecture and functions of the Cordova Library in iOS are different than in Android. They still follow the same idea, but are different in implementation. Most of the functions in Android have an iOS equivalent. One of the ones that do not is onActivityResult() in the DevicePlugin implementation. I’ve talked with Carl about this though and it seems that it won’t be that big of a deal since we are planning to remove that function in the future. It seems to me that the Cordova Library for Objective-c is a lot more immature then the Java one. You can see this in the fact that going from version 2.0 (the one that we are using) to 2.3, most of the Objective-c code changes, but almost nothing from the Java code does. This indicates that there is still a lot of work to be done on the Objective-c library. I’m not complaining, but the Java Cordova Library just looks so much nicer.

Another thing that is a lot different are language specific things. For example in Objective-c you cannot specify the visibility level (i.e. public, protected, private) on member functions, and you can’t specify static member variables in exactly the same way as Java. This isn’t a big deal, but it is one of the many things that must change from going to Java to iOS. Another, rather concerning, thing that is different is that Objective-c offers no way to synchronize entire functions. This is a problem because this design paradigm is used a lot in the Java plugin. We will have to find a way around it in Objective-c.

Other Work

  • Caitlin has currently been tackling the BluetoothServer class as well as doing the full implementation of the ReturnableCommand class.
  • Vince has been working on the PacketParser implementation. I haven’t heard much from him this week, but I hope he has been making good progress.

Moving Forward

After Caitlin gets the ReturnableCommand landed we will be able to start implementing the concrete Command objects that inherit from ReturnableCommand and begin to tie the commands into the Invoker as well as the MedicalDevicePlugin class. I’ve taken on writing the isSupported(), isEnabled(), and isDisabled() commands. Everything is looking good so far baring the problem we may have with the Bluetooth Protocols…

Edit:

Recently heard from Carl that I should be implementing threads in Objective-c in the Invoker class. It’s up to me how we get that done. So I will be working on that before continuing the Command implementations.

WEBVTT: Refactoring the Parser

So I’ve been working on refactoring the parser a bit over the last week. I talked about this in the one of my previous posts. You can check out the work I’ve done so far on my GitHub.

Squashing Code is Nice

So I have successfully squashed the node and cuetext token data code and it’s worked out nicely. This reduces the overall amount of code that needs to be written and the complexity of the code that needs to be written to access those structs. Heres what they look like now:

Token:

struct
webvtt_cuetext_token_t {
  webvtt_cuetext_token_type token_type;
  webvtt_string tag_name; // Only used for start token and end token types.
  union {
    webvtt_string text;
    webvtt_timestamp time_stamp;
    webvtt_cuetext_start_token_data *start_token_data;
  };
};

/**
* Represents a start tag in the cue text.
* These take the form of <[TAG_NAME].[CLASSES] [POSSIBLE_ANNOTATION]> in the cue text.
*/
struct
webvtt_cuetext_start_token_data_t {
  webvtt_string_list *css_classes;
  webvtt_string annotations;
};

Node:

struct
webvtt_node_t {
  /**
  * The specification asks for uni directional linked list, but we have added a parent
  * node in order to facilitate an iterative cue text parsing solution.
  */
  webvtt_node *parent;
  webvtt_node_kind kind;

  union {
    webvtt_string text;
    webvtt_timestamp time_stamp;
    webvtt_internal_node_data *internal_data;
  };
};

struct
webvtt_internal_node_data_t {
  webvtt_string annotation;
  webvtt_string_list *css_classes;

  webvtt_uint alloc;
  webvtt_uint length;
  webvtt_node **children;
};

Here is one example how it simplifies the code:

Before:

if( webvtt_create_node_from_token( token_ptr, &temp_node_ptr, current_node_ptr ) != WEBVTT_SUCCESS )
  /* Do something here? */
  continue;
  else
  {
  webvtt_attach_internal_node( (webvtt_internal_node_ptr)current_node_ptr->concrete_node, temp_node_ptr );
  if( WEBVTT_IS_VALID_INTERNAL_NODE( temp_node_ptr->kind ) )
    current_node_ptr = temp_node_ptr;
  }
}

After:

if( webvtt_create_node_from_token( token, &temp_node, current_node ) != WEBVTT_SUCCESS )
  /* Do something here? */
  { continue; }
  else {
    webvtt_attach_internal_node( current_node, temp_node );
     if( WEBVTT_IS_VALID_INTERNAL_NODE( temp_node->kind ) )
       { current_node = temp_node; }
      }
  }
}

Tokens vs Nodes

In my last blog post about refactoring the parser I talked about removing the token stuff and instead just using webvtt_node straight away. I haven’t done that yet because I have discovered a problem with it. The problem is that tokens have a way of representing an end tag
whereas nodes do not. This is important because when the tokenizer returns an end tag token a new node is not instantiated, instead the current node will change to the current nodes parent. Therefore, since a node has no idea of what an end tag is it cannot replace a token… yet.
I’m still not convinced that we can’t do this in some way.

Those Pesky Pointer Typedefs

I’ve also begun to remove all the pointer typedefs I’ve been using throughout my code. I liked it at first, but now that I have been thinking about it just using straight pointer notation is probably better.

  1. It’s more clear as to what is actually going on.
  2. I was typedef-ing pointers as a [object]_ptr. So if I instead just use pointer notation it will decrease the character count everywhere. Instead of [object]_ptr everywhere we will just have *[object].

Memory Leaks in C++ Bindings

The last thing I have done for now is  changed the C++ bindings over to instantiate the entire tree of nodes when the cue is created. This way the cue knows about the nodes and will be able to release them at the end if it’s lifespan. In this way we can avoid memory leaks.

UTF16 to UTF8

The final thing that I need to do is get the parser using UTF8 instead of UTF16. Caitlin is currently doing a major overhaul on the webvtt_string stuff to change it from using UTF16 to instead use our webvtt_byte which is straight UTF8. That way we can have all those nice string functions that we created for UTF16 with our UTF8 stuff.

Forward to Review

I’m looking to get this done as well as debugging the unit tests by Friday. Then I believe the parser will be ready for review.

Getting Started on the Connected Wellness Android to iOS Port

So I’ve started a new job here at CDOT (Seneca’s Centre for Development of Open Technology) and the first project that I’ve been assigned to work on is a project called Connected Wellness that is being done in collaboration with NexJ and other colleges. I’m on a team of three who will be porting the current android Bluetooth plugin over to iOS. I will be blogging a lot about this in the next two months since the deadline for it to be done is the end of February.

What I’ve been doing so far is setting up my development environment and getting acquainted with Mercurial which I have never used before. I’ve started work on the MedicalDevicePlugin class which will be the entry point for Apache Cordova to the plugin. I’ve also ported over the basic outline for the Command interface and inheritance structure in order for me to be able to reference them in the MedicalDevicePlugin class. Everything for my part looks like smooth sailing so far. The only thing I’m worried about is getting the Cordova library to work properly in iOS.

My teammate, Caitlin, has recently blogged about troubles with the current libraries we are using for the Bluetooth protocol not supporting the specific devices we want to interface with. Hopefully, we will be able to work around this. My first thought is to use an open source Bluetooth library for iOS, that is if Apple even lets outside frameworks access the Bluetooth hardware directly in some way.