Wednesday, 10 October 2012

WebVTT Test Bugs

First, from the work I just did I've made two correction to my WebVTT specification post.

I've gone over all the fail tests for WebVTT that we have made in our open source course DPS909. I've found many problems between the WebVTT syntax and parsing sections. I expect changes will have to be made to both parts.

Test errors

I only checked if the parser will pass or fail, not if things work as intended (they probably don’t). Most of the time malformed expressions are simply skipped.

webvtt / test / spec / bad / tc2003-no_bank_line_before_cue.test
Test will pass. Second cue will be treated as part of payload for first cue.

webvtt / test / spec / bad / tc2004-cue_id_cannot_be_standalone.test
Test will pass. The blank line after “1” will cause parser to simply skip the “1” line.

webvtt / test / spec / bad / tc3050-missing_spaces.test
webvtt / test / spec / bad / tc3051-missing_space_left.test
webvtt / test / spec / bad / tc3052-missing_space_right.test
Test result unknown. The syntax states one space character is required. The “skip whitespace” step allows for zero whitespace.

webvtt / test / spec / bad / tc3054-arrows_wrong_direction.test
Test will pass. “<--” line will be treated as additional header information. When parser hits end of line, it will end without error.

webvtt / test / spec / bad / tc3203-missing_time_cue_symbol.test
Test will pass. “00:00.000 00:00.001” line will be treated as additional header information. When parser hits end of line, it will end without error.

webvtt / test / spec / bad / tc3210-invalid_cue_spacing.test
Test result unknown. The syntax states one space character is required. The “skip whitespace” step allows for zero whitespace.

webvtt / test / spec / bad / tc3211-random_symbols_in_cue.test
webvtt / test / spec / bad / tc3053-nonnums_in_timestamp.test
Tests same things. Should have tests for “00:00:00.00a”, “00:00:0a.000”, “00:0a:00.000”, “0a:00:00.000”, “00:00.00a”, “00:0a.000”, “0a:00.000”

webvtt / test / spec / bad / tc4002-cue_settings_vertical_02_bad.test
Test will pass. “vertical:lr” is valid.

webvtt / test / spec / bad / tc4003-cue_settings_vertical_03_bad.test
webvtt / test / spec / bad / tc4005-cue_settings_vertical_05_bad.test
webvtt / test / spec / bad / tc4006-cue_settings_vertical_06_bad.test
webvtt / test / spec / bad / tc4007-cue_settings_vertical_07_bad.test
Test will pass. Parser skips settings without “:” or where “:” is first or last character in setting.

webvtt / test / spec / bad / tc4004-cue_settings_vertical_04_bad.test
Test will pass. Parser will ignore settings that do not match valid options.

webvtt / test / spec / bad / tc4008-cue_settings_line_01_bad.test
webvtt / test / spec / bad / tc4010-cue_settings_line_03_bad.test
Test will pass. Parser will ignore settings that do not match valid options.

webvtt / test / spec / bad / tc4009-cue_settings_line_02_bad.test
webvtt / test / spec / bad / tc4012-cue_settings_line_05_bad.test
webvtt / test / spec / bad / tc4013-cue_settings_line_06_bad.test
webvtt / test / spec / bad / tc4014-cue_settings_line_07_bad.test
webvtt / test / spec / bad / tc4015-cue_settings_line_08_bad.test
Test will pass. Parser skips settings without “:” or where “:” is first or last character in setting.

webvtt / test / spec / bad / tc4016-cue_settings_multi_01_bad.test
Test will pass. Parser does not check for duplicate settings. It will use the last setting, earlier ones get overridden.

webvtt / test / spec / bad / tc4017-cue_settings_multi_02_bad.test
Test will pass. Parser splits only on the first colon of each setting. The second half, which contains a colon, will be treated as the value, and since it isn’t a valid value the setting will be skipped.

webvtt / test / spec / bad / tc4019-cue_settings_delimiter_bad_02.test
Test will pass. Every charcter after the end time offset timestamp is used for settings, and the settings first splits on spaces, so not space after the timestamp has no effect. “^line” is not a valid option, so the parser just skips it.

webvtt / test / spec / bad / tc4503-cue_settings_align_bad_value.test
Test will pass. Parser will ignore settings that do not match valid options.

webvtt / test / spec / bad / tc4504-cue_settings_align_no_colon.test
webvtt / test / spec / bad / tc4505-cue_settings_align_no_setting.test
webvtt / test / spec / bad / tc4506-cue_settings_align_no_value.test
webvtt / test / spec / bad / tc4507-cue_settings_align_wrong_colon.test
webvtt / test / spec / bad / tc4603-cue_settings_position_no_colon.test
webvtt / test / spec / bad / tc4605-cue_settings_position_size_no_setting.test
webvtt / test / spec / bad / tc4606-cue_settings_position_wrong_colon.test
Test will pass. Parser skips settings without “:” or where “:” is first or last character in setting.

webvtt / test / spec / bad / tc4604-cue_settings_position_no_percent.test
Test will pass. Parser will ignore settings that do not match valid options.

webvtt / test / spec / bad / tc4703-cue_settings_size_no_colon.test
webvtt / test / spec / bad / tc4705-cue_settings_size_wrong_colon.test
Test will pass. Parser skips settings without “:” or where “:” is first or last character in setting.

webvtt / test / spec / bad / tc4704-cue_settings_size_no_percent.test
Test will pass. Parser will ignore settings that do not match valid options.

webvtt / test / spec / bad / tc5028-cue_text_format.test
Test will pass. Parser will consider “<i are</i>” a single tag with name “i” and annotation “are</i”. The annotation will be ignored and the remainder of the text in the payload will in the WebVTT Italic Objects (it will be italicized).

webvtt / test / spec / bad / tc5029-cue_text_format.test
Test will pass. Parser ignores tags that do not match valid names.

webvtt / test / spec / bad / tc5030-cue_text_format.test
Test will pass. Parser will return tag name at either “>” or end of input (they are treated the same). Parser ignores tags that do not match valid names. Note that in end tags there are no annotation so “i in New York City” will be considered the tag name.

webvtt / test / spec / bad / tc5031-cue_text_format.test
Test will pass. Parser does not require end tags. Parser creates a node object after tag and make current object it’s parent. Parser return root object list when it reaches the end of payload text.

webvtt / test / spec / bad / tc5032-cue_text_format.test
Test will pass. “>” outside of tag is treated normally (same as &gt;).

webvtt / test / spec / bad / tc5033-cue_text_format.test
Test will pass. Parser will return empty string for tag name if start tage reaches end of payload input before “>”. Parser will ignore tags with names that don’t match valid names.

webvtt / test / spec / bad / tc5034-cue_text_format.test
Test will pass. Parser will ignores end tags where previous tag is not the matching start tag. “</b>” will end and the rest will retain italicized.

webvtt / test / spec / bad / tc5035-disallow_annotation_italic.test
webvtt / test / spec / bad / tc5036-disallow_annotation_underline.test
webvtt / test / spec / bad / tc5037-disallow_annotation_bold.test
webvtt / test / spec / bad / tc5038-disallow_annotation_class.test
webvtt / test / spec / bad / tc5039-disallow_annotation_ruby.test
Test will pass. Parser allows annotations on all tag but will ignore annotations except those on voice tags (“<v> tags”).

webvtt / test / spec / bad / tc5040-disallow_annotation_time_stamp.test
Test will pass. Parser ignores timestamp tags with characters following the timestamp within the tag.

webvtt / test / spec / bad / tc5043-incorrect_cue_class.test
Test will pass. Parser ignores tags that do not match valid names.

webvtt / test / spec / bad / tc5057-incorrect_space_character_escape_nbp.test
webvtt / test / spec / bad / tc5058-incorrect_space_character_escape_nsp.test
webvtt / test / spec / bad / tc5059-incorrect_space_character_escape_bsp.test
webvtt / test / spec / bad / tc5060-incorrect_space_character_escape_bp.test
webvtt / test / spec / bad / tc5061-incorrect_space_character_escape_b.test
webvtt / test / spec / bad / tc5062-incorrect_space_character_escape_s.test
webvtt / test / spec / bad / tc5063-incorrect_space_character_escape_ns.test
webvtt / test / spec / bad / tc5064-incorrect_space_character_escape_np.test
webvtt / test / spec / bad / tc5065-incorrect_space_character_escape_sp.test
webvtt / test / spec / bad / tc5067-incorrect_space_character_escape_p.test
webvtt / test / spec / bad / tc5068-incorrect_space_character_escape_bs.test
webvtt / test / spec / bad / tc5069-incorrect_left_to_right_character_escape_lr.test
webvtt / test / spec / bad / tc5070-incorrect_left_to_right_character_escape_l.test
webvtt / test / spec / bad / tc5071-incorrect_left_to_right_character_escape_lm.test
webvtt / test / spec / bad / tc5072-incorrect_left_to_right_character_escape_rm.test
webvtt / test / spec / bad / tc5073-incorrect_left_to_right_character_escape_m.test
webvtt / test / spec / bad / tc5074-incorrect_left_to_right_character_escape_r.test
webvtt / test / spec / bad / tc5075-incorrect_right_to_left_character_escape_lm.test
webvtt / test / spec / bad / tc5077-incorrect_left_to_right_character_escape_rl.test
webvtt / test / spec / bad / tc5078-incorrect_right_to_left_character_escape_rl.test
webvtt / test / spec / bad / tc5079-incorrect_ampersand_without_escape.test
webvtt / test / spec / bad / tc5080-incorrect_ampersand_escape_a.test
webvtt / test / spec / bad / tc5081-incorrect_ampersand_escape_am.test
webvtt / test / spec / bad / tc5082-incorrect_ampersand_escape_mp.test
webvtt / test / spec / bad / tc5083-incorrect_ampersand_escape_p.test
webvtt / test / spec / bad / tc5084-incorrect_ampersand_escape_ap.test
webvtt / test / spec / bad / tc5085-incorrect_less_than_escape_l.test
webvtt / test / spec / bad / tc5086-incorrect_less_than_escape_t.test
webvtt / test / spec / bad / tc5087-incorrect_greater_than_escape_g.test
webvtt / test / spec / bad / tc5088-incorrect_space_character_escape_nbs.test
webvtt / test / spec / bad / tc5089-incorrect_space_character_escape_nb.test
webvtt / test / spec / bad / tc5090-incorrect_space_character_escape_n.test
Test will pass. Parser will treat invalid escape sequence as normal text.
webvtt / test / spec / bad / tc5115-multi_component_multi_line_bad_crlf.test
webvtt / test / spec / bad / tc5116-multi_component_multi_line_bad_cr.test
webvtt / test / spec / bad / tc5117-multi_component_multi_line_bad_lf.test
webvtt / test / spec / bad / tc5118-multi_escape_multi_line_bad_crlf.test
webvtt / test / spec / bad / tc5119-multi_escape_multi_line_bad_cr.test
webvtt / test / spec / bad / tc5120-multi_escape_multi_line_bad_lf.test
webvtt / test / spec / bad / tc5121-component_and_escape_multi_line_bad_crlf.test
webvtt / test / spec / bad / tc5122-component_and_escape_multi_line_bad_cr.test
webvtt / test / spec / bad / tc5123-component_and_escape_multi_line_bad_lf.test
Test will pass. Parser discards malformed cues.

webvtt / test / spec / bad / tc5116-multi_component_multi_line_bad_cr.test
webvtt / test / spec / bad / tc5119-multi_escape_multi_line_bad_cr.test
webvtt / test / spec / bad / tc5122-component_and_escape_multi_line_bad_cr.test
webvtt / test / spec / bad / tc5123-component_and_escape_multi_line_bad_lf.test
Python comment is badly written. Github fails to display it properly.

webvtt / test / spec / bad / tc5301-voice_cue_no_annotation.test
Test will pass. Parser will use an empty string if there is no annotation.

webvtt / test / spec / bad / tc5306-ruby_missing_open_tag.test
Test will pass. Parser will skip rt tags not contained by a ruby tag. Parser will skip end tags where the previous tag was not the matching start tag.
webvtt / test / spec / bad / tc5307-ruby_missing_close_tag.test
Test will pass. Parser does not require end tags. Parser creates a node object after tag and make current object it’s parent. Parser return root object list when it reaches the end of payload text.

webvtt / test / spec / bad / tc5324-ruby_multiple_rt_optional_line_terminator.test
Test will pass. Cue will end before “</ruby>” and “</ruby>” will be considered a new cue, which will be discarded.

webvtt / test / spec / bad / tc5325-timestamp_less_than_previous.test
webvtt / test / spec / bad / tc5326-timestamp_less_than_start_timestamp.test
webvtt / test / spec / bad / tc5327-timestamp_greater_than_end_timestamp.test
Test is unknown. Parser does not check if timestamps in timestamp tags or cue timings are in the order specified in the syntax section of the spec. If the parser is not changed, then according to my reading of the HTML5 media spec, either the cues and timestamps are are sorted into the correct order, or bad ones are discarded.

webvtt / test / spec / bad / tc_1006_header-no-new-line.test
Test will pass. Parser allows newline in the header. A blank line (two newlines in a row) will end the header and start the cues. However, if the line contains the string “-->”, the parser will take that to be the timing line of the first cue.

webvtt / test / spec / bad / tc_1010_new-linestart.test
Test will pass. Parser does not require a newline, or anything, after “WEBVTT”.

webvtt / test / spec / bad / tc_1016_missing_new_line.test
Test will pass. Text payload can be over more than one line, but not blank lines.

webvtt / test / spec / bad / tc_1017_line_break.test
webvtt / test / spec / bad / tc_1018_missing_cue_identifier.test
Test will pass. Malformed cues are discarded.

webvtt / test / spec / bad / tc_1019_missing_req_blank_line.test
Test will pass. Second line has “-->” string and will be taken as start of a cue, with that line being the timing line.

No comments:

Post a Comment