Use of Graphics Rendering and Artificial Intelligence for Improved Mobile Search Capabilities
GRAISearch is an Industry-Academia Partnerships and Pathways project (IAPP) aiming at transferring knowledge from Academia to Industries. It is also a support for training and career development of researchers (Marie Curie).
The principal aim of this project is to develop and apply revolutionary graphics rendering and artificial intelligence (AI) methods to an existing social media search engine platform thereby creating ground breaking mobile search capabilities with significant online commercial potential.
Technologies from research in two academic institutions (INSA de Lyon (LIRIS UMR CNRS 5205) and Trinity College Dublin (TCD)), will be transfered into the products of Tapastreet LTD, the Dublin based Web startup.
Julie Connelly
Project Manager
graisearch@scss.tcd.ie
Rozenn Dahyot
Project Coordinator
Project start: 01/02/2014
Project end: 31/01/2018
Funding agency: EU Marie Curie Actions
Project Websites: graisearch.eu and Cordis website
Workpackage Leader: TCD
This package aims at developing video summarization algorithms (vsa) for amateur social media video to display local event highlights as they occur anywhere in the world. The first brute force strategy to create a gif summary by down-sampling videos, will look at the different resolutions that could be proposed to the user with the requirement that the information displayed is of good quality for a quick understanding of the content of the input video. software 1.0 will be used to process a database of videos and create short summaries at different resolutions. The analysis of the low level content (e.g. motion, colour) of the video can be performed with metrics defined in information theory. We propose to create a smart software prototype for creating summaries that will use these metrics for measuring automatically the content of videos. These metrics will be used to automatically select what are the images in the video that should be retained to be part of the summary, and also this will help in selecting the best spatial resolution automatically. Some artifacts (hand shake, blur, and occlusion) that often occur in amateur videos will be dealt with to improve the quality of the summaries.
Workpackage Leader: TCD
WP2 aims at developing automatic local 3d scene rendering algorithms (sra) leveraging public geo-located social media photos of a particular location. merging several images or videos to create an augmented image can be a step further towards creating a good quality summary. this work package led by tcd will look at merging several images and/or videos recorded at the same place at the same time for creating a 3d rendering of a scene. using location information embedded with the input images and videos, and potentially using additional 3d content available (e.g. google maps in 3d), this work package will look at computing local descriptors in the images, suitable for image stitching and 3d reconstruction, but also for image classification (wp5). we propose to use a modeling based on the generalised relaxed radon transform (gr2t) to estimate a probability density function of the 3d location and colour. an animated gif will then be created by moving a virtual camera into the scene and a perceptual testing using questionnaires and eye tracking technique will be used to assess the quality of the rendering. the path of the virtual camera will be automatically chosen such that the summary is both informative, and visually pleasing. metrics from information theory will be used to assess the information content of the summary.
Workpackage Leader: INSA
Mathematical methods and prototypes for mining and predicting local communities workflows through contextualized trajectory pattern mining applied to social network data from Tapastreet. For that, the following sub-tasks are considered: (i) community detection, (ii) places of interest (POI) characterization, and finally (iii) constrained-based mining of contextualized trajectories. The goal is to provide a valuable input for WP5 which entails making recommendations to social media end users w.r.t. their social trajectories and context, the recommendation could be a breaking news events (whose detection is handled in WP4) and is triggered by a fix on the persons location.
Workpackage Leader: INSA
Develop a computing solution for (i) event detection and (ii) sources of trust identification in geolocalized social media data streams. It is a centre piece between WP3 and WP5: Geo-localized breaking events are detected and recommended to a user entering (or predicted to enter in) the corresponding location. For task (i), we design/use data-mining techniques to detect/predict unexpected events/patterns in data streams in presence of concept drift. We propose to model task (ii) as an original problem of temporal dependency discovery between some topics from different social media and bring an algorithmic solution through a graph-mining algorithm.
Workpackage Leader: INSA
Identification of a recommendation strategy and the design of a recommender prototype for geo-located social media users in a geo-local context. Thanks to the results of WP3 and WP4, we make use of user trajectories stratified by demography, characterized points of interest, and trusted breaking news. Here we close the loop on WP3&4 who's learnings are applied and built into this recommender system prototype. The strategy will be based on real data supplied by Tapastreet, expertise from Tapastreet's machine learning department and knowledge and expertise from INSA de Lyon.
Workpackage Leader: Tapastreet
WP6 is to develop strategies and methods for implementation of video summarisation and automated 3d scene rendering into tapastreets social media search engine platform. Research effort in wp6 is led by tapastreet and will be looking at what strategy (e.g. cloud computing, parallel processing, gpu processing, etc.) can be used for a fast reliable implementation of wp1 & wp2 into the tapastreet platform. video summarization will need to be processed rapidly, and therefore some solutions may be better adapted than others. a hierarchical approach can be considered where about simple summaries are proposed and then replaced when the smarter ones becomes available. beside the strategy for processing the information in a timely fashion, this workpackage will look at storage of the summaries, and easy access via mobile platform.