VAST 2011 Challenge Reviews of submission #116: "ODU-VisualInvestigator-MC3" ------------------------ Submission 116, Review 1 ------------------------ Reviewer: external Overall rating: 17 (scale is 0..50; 50 is best) Clarity of Explanation Average Comments on Clarity of Explanation The explanation was reasonably clear, but that seems to be because there was reasonably little to explain. There were three points I would have liked to see explained further. First, I see a 'keep' button next to the articles, and no explanation of what that does (a separate view that displayed the accumulated "interesting" documents would presumably be a feature). Second, the submission included graphs of the solution indicating the perceived threat and how the connections were made. Since it was not included as a screenshot, it seems possible that the submitters do not consider it to be "part of the answer", but clearly they felt it useful enough to create and include in the video. Given the paucity of visualizations involved in the submission, this seems worth a mention (and a commodity graphing tool is no less a tool than phpMyAdmin). Third, the search interface seems under discussed. The interface seems to suggest that it can do simultaneous searches and suggest which terms are more useful, but this is not discussed at all. Finally, given the limited available space, I thought that the inclusion of the failed visualization was not necessary (though I can understand the motivation). Threat Accuracy Inaccurate Comments on Threat Accuracy This submission touched on some of the basic themes of the threat, but has focused on the wrong group and the wrong vector for the threat. Threat Detail Little Detail Comments on Threat Detail The submission does link together a couple of important documents but other than identifying the basic threat of bioterrorism and (incorrectly) the group behind it, there are sparse details about the actual threat. We are told that airborne microbes will be distributed by airplane sometime in May, which is a somewhat broad prediction, and there is little support for the date or attack vector (released on planes). Supporting Documents A Few Correct Documents Comments on Supporting Documents The solution appears to have been based on five correct documents, one isolated document, and three noise documents. Visualizations Marginal Comments on Visualizations The only visualization described in this submission is described as not contributing. As I mentioned earlier, the chart shown in the video at least seems to have been a useful marshaling tool, but it does not seem to be considered part of the approach by the submitters. Interactions Marginal Comments on Interactions In fairness, the main tool seems unfinished. The need for two clicks to get at the actual results of a search is unfortunate. There is certainly no reason to display the TF-IDF score to the user, and the table format only makes sense if more than one word was going to come up in response to the query, which was never demonstrated. I suppose the theory was to enter a collection of terms and then pick the one that scored better. However, this would seem to be an obvious candidate for a simple visualization (and, as I mentioned earlier, worth discussing). The restriction to single word search terms also seems problematic (especially has it necessitated leaving the tool and issues raw SQL searches). I liked the popup view of the documents when they were moused over, even if the submitter did not think they were interesting enough to rate a mention. I also, theoretically, like the idea of a keep button to collect together the documents of interest, but again, this seems to have not been interesting to the submitter, as it was never mentioned. Novelty Marginal Comments on Novelty As presented, the tool provides a slightly broken search functionality (only single word search terms). There are potentially some other capabilities that I've guessed at above, but they seem to have played no actual role in developing the solution. Overall Rating Marginal Comments on Overall Rating The submission did succeed in latching on to part of the solution, which is indeed a challenge for this particular dataset. However, the solution was fairly inaccurate and it is hard to see where the "Visual" in "Visual Investigator" might be (though the broken visualization hints at greater ambitions). The writeup was nominally reasonable in that it traced the thought process, but did little to sell the deficiencies of the tool. It is hard not to believe that the submitters could have reached the same conclusions even faster just using the search facility built into the Windows file browser. ------------------------ Submission 116, Review 2 ------------------------ Reviewer: external Overall rating: 27 (scale is 0..50; 50 is best) Clarity of Explanation Good Comments on Clarity of Explanation Explanations are clear so is supporting material like video and figures. Threat Accuracy Inaccurate Comments on Threat Accuracy The authors identify the right threat but unfortunately reach wrong conclusions on everything else: who, how and when. Threat Detail Detailed Comments on Threat Detail The submitter answer was very detailed, so was the description of the analytical process followed to reach the provided answer. Supporting Documents Majority Correct Documents Comments on Supporting Documents The authors found 6 correct documents out of 13. Visualizations Good Comments on Visualizations The visual-analytic tool and its output are critical in the analytical process performed by the authors. Interactions Average Comments on Interactions The authors developed their own tool which due to time-constrain is a bit rough. It appears easy to use and with a bit more work its utility could definitely improve. Novelty Moderate Comments on Novelty I appreciate the fact that the authors developed their own tool and that it actually looks quite good. There is however no significant novelty. Overall Rating Average Comments on Overall Rating The submission is well done, clear and detailed descriptions and sound supporting material. Both visualization and analytical tool are not particularly novel, though I appreciate the effort put into developing a proprietary tool. ------------------------ Submission 116, Review 3 ------------------------ Reviewer: external Overall rating: 25 (scale is 0..50; 50 is best) Clarity of Explanation Good Comments on Clarity of Explanation The submission clearly explained the iterative process that the team of investigators followed. Although the approach was less sophisticated than other submissions reviewed, it was nonetheless effective for manually identifying articles of interest, expanding to related articles, and selecting articles identified as relevant. Threat Accuracy Accurate Comments on Threat Accuracy The primary threat identified by the team was correctly identified as a threat of bioterrorism using a genetically-modified pathogen. However, many of the details of the threat were slightly inaccurate, such as not identifying the primary terrorist group and not accurately identifying the intended means of distribution (contaminated food supply). Threat Detail Detailed Comments on Threat Detail The team did describe the threat in a respectable amount of detail, including who, what, how, and when. However, in the opinion of the reviewer, not all of the conclusions that the team drew were well supported by the articles provided as evidence. Supporting Documents Majority Correct Documents Comments on Supporting Documents Six out of nine documents that were cited as evidence to support the identified threat were relevant (i.e. high precision), but the recall could have been better, as only six of 13 actual true positives were retrieved. The F measure (54%) combining precision and recall (for the primary threat of interest) was comparable to the other submissions this reviewer observed. Visualizations Marginal Comments on Visualizations The team created modest but somewhat effective tools for exploring the document corpus. The most effective interactive tool permitted keyword search and review of the resulting documents. This was not very visual, as it simply permitted users to review a lists of documents sorted by keyword prominence and perform one of three actions (read, keep article, or search for similar articles). The more innovative document similarity visualization was described as ineffective because the similarity metric needed improvement. Interactions Average Comments on Interactions The modest tools created for this investigation appeared easy to use but limited in functionality. I did appreciate the straightforward design of the custom investigation tool, as well as the similarity view's mouseover labels showing document titles. I question whether mouse-over labels showing document text is appropriate for the primary investigation tool. No additional interactions were described beyond those described above and the stock phpMyAdmin query tool. Novelty Marginal Comments on Novelty There was not a substantial level of novelty evident in the solution developed for this exercise. I appreciated the document similarity view the most although it may require extensions to allow tuning of similarity metrics in order to be of greater use in future investigations. Overall Rating Average Comments on Overall Rating I've rated this submission as average overall. Although the tools developed were not very visually oriented, the solution was nominally effective for exploring the document collection. As a result, many of the documents relevant to this task were discovered. The threat analysis itself left a bit to be desired, as some conclusions about the nature of the threat were not well supported and did not perfectly match the exercise's threat scenario.