Right now (as of September 2009)  the testing process for EBIF applications is confusing at best - so here's some insight on the current state of testing an EBIF application.  

While EBIF is a widely accepted standard for developing ITV applications, there is not a standard method for getting an application tested and deployed across millions of digital set-top boxes within the US cable MSOs (multiple system operators).  Application developers will need to discover the current certification method on an MSO by MSO basis prior to deployment.

Hopefully, in the near future - CableLabs and the MSO's will settle on a single point of entry for testing and certifying EBIF Applications which should go a long way to alleviate this pain for developers, but until that happens -  getting an app tested will take some effort.  Be sure in your planning process to account for significant time going through MSO QA, anywhere from one to four months is not unreasonable.  While the actual testing time may only be 2-6 weeks, scheduling time with the various QA labs is likely to add additional time to the process.

The Basic Process

Testing your application will take a few phases of user experience testing. For the purposes of this article we will only discuss testing of the EBIF application itself.  Out of scope is testing of two-way application servers or validating input data, most back-end systems can be tested with standard web-based test suites.

1. 1. Determine Test Scenarios

2. 2. Test application in your development environment
   

3. 3. Developer QA on a Set-Top-Box

4. 4. Initial Application QA

5. 5. MSO QA.
   

Determine Test Scenarios

The first step in this process is to determine what the valid test scenarios are for your particular application.  Key questions to ask.

1. 1. Which MSO's are we deploying with?

2. 2. What ETV (EBIF) User Agent are we deploying against?

3. 3. What version of the User Agent are we deploying against?

4. 4. What Set-Top Boxes will we be deploying on?

5. 5. How will we provide test or production data to the testing labs?

6. 6. What is likely to be different about testing on a set-top box as opposed to testing in a simulator?

7. 7. How many users are likely to use your application simultaneously?

Testing in a development environment

Initial testing will generally be done in a development PC-based simulator environment. Testing in these environments is the best way to get initial testing done.  This environment won't exactly replicate a production environment, how well the simulator does this depends on your development environment.  The TVWorks XDK for example is based on the actual TVWorks User Agent code and actually does a very good job of simulating what you will see on the real set-top boxes.  *(The free TVWorks XDK can be downloaded at developer.tvworks.com)

There are a few things that can be tested well in this environment.  Some key things to test are:

1. 1. Navigation focus is not lost in through the application
   

2. 2. Rendering display elements is correct
   

3. 3. Data is being handled properly
   

4. 4. Application logic is sound
   

5. 5. Error conditions are captured and handled properly
   

While much of your application’s logic and UI can be tested well in the simulator environment, there are a few things that need to be tested on an actual set-top box.

Developer Set-Top Box QA

There are really two ways to do initial testing on an actual set-top box. One is to set up your own development lab; the second is to use a 3rd party development lab.  Setting up your own lab is not trivial (in terms of both cost and effort)  - though a few manufacturers are developing ways to alleviate this pain with a less-expensive "headend in a box" type of device.   Some application developers may also be able to piece together their own QA lab using off-the-shelf hardware and some of the free tools provided by TVWorks (such as the ETV Streamer).

Without your own lab you can also test on a few available labs. Companies like Itaas and EnableTV provide testing for the TVWorks User Agent via their ETV developer programs.

There are some key things to test in this environment that are difficult to replicate in a simulated environment.

• 1. Page rendering on low-end set-top boxes.  On some low-end set-tops (Moto DCT2Ks in particular) the redraw of a view can be slow and this may be detrimental to the user experience.
  

• 2. Channel Tuning into and out of a bound app. For unbound applications making sure the application resumes, or doesn't resume correctly after channel changes (the correct resume behavior on channel changes depends on application requirements).
  

• 3. VOD playback
  

• 4. DVR and reminder functionality
  

• 5. Remote Control differences if capturing remote keys.
  

• 6. Any changes in rendering based on user agent or set-top-box differences.
  

• 7. General application performance issues. Performance to key presses. Two-way form submission performance.
  

• 8. Interactions with the guide and other interactive applications. Other applications (EBIF or other) may cause the application to be suspended at any time.  Does the application recover correctly?
  

• 9. Synchronization with real time events - important with applications that are synchronized to video, such as a quiz or polling/voting campaign.
  

• 10.   Application is in a screen safe area.
  

• 11.   Application colors are TV friendly.

• 12.   Alignment to background video and in-show graphics.

Initial Application QA

Different MSO's may collapse the testing process into a single phase, but generally (at this time) the actual application QA will be a two step process.  The first is a general application QA certification by an application QA group –  this can be either a 3rd party like itaas, EnableTV or another test group within the MSO organization. This group will generally be the User Agent experts and will be the most comprehensive phase of testing. This is then generally followed by a separate MSO test phase that is more focused on making sure the application will do no harm to an existing MSO's user experience.

At this point your EBIF application and the supporting infrastructure should be as close as possible to the actual production versions of your applications.  Here are some of the key tests for that may occur in this phase that are difficult to replicate in a lab scenario:

• 1. Automated longevity testing.  Many of the QA labs have automated tests that will run through your application without a STB reboot for a long (days or weeks) amount of time. This can help discover some user agent issues that may need to be addressed in the application.
  

• 2. Automated Two-way stress testing.  Making sure your infrastructure can support the estimated number of users.
  

• 3. Full test against all available Set-Top-Box types.
  

• 4. Full Application Use Case QA.
  

• 5. Application Set Interaction.  Making sure your application will not negatively affect other EBIF applications running on the set top box.
  

• 6. Testing with other guide implementations.
  

Final Application Certification

While the final phase of Application QA will generally be focused on making sure that your EBIF Application will do no harm on actual deployment scenarios, it also has an opportunity to find any application bugs that have slipped through any cracks during the application QA processes.  In this phase the following elements will be tested.

• 1. Any interactions between the application and various other code modules or applications.  Up until this point there may not have been opportunities to test against the different software configurations that your application will be running with.  Differences in guides, VOD Clients, assorted 3rd party modules will be tested at this point.

• 2. Out Of Band Bandwidth Issues.   Make sure your application won't cause OOB network congestion (in the form of excess QPSK traffic)  - causing an interruption in user service
  

• 3. InBand Bandwidth Issues (Bound Apps Primarily).  Make sure that insertion of your application will not cause degradation of the user's video service.
  

The Final Word

Testing an EBIF Application can be complicated but hopefully help is on the way.  Right now, the key to moving forward is working as closely as possible with your MSO partners throughout the process and making sure you have open lines of communication throughout the various cycles.

About the author:  Mike Ryan's company A Different Engine provides consulting, design and development work for interactive television applications. He has spent the last 15 years developing interactive applications, and apps for TV specifically since 2000. Prior to starting A Different Engine, he was at TVWorks, (a joint venture between Comcast and Cox communications) developing their ITV infrastructure. While at TVWorks, he led the Client Applications group’s development of Comcast’s first generation of ETV apps.

Special thanks to Amy Gonzales of TV Works who contributed to this article.