Media Campaign Prototype
Contents
System Architecture
This page summarises the architecture and main functionalities of the first integrated MediaCampaign prototype. It focuses mainly on the Press/TV workflows, for which most of the targeted system functionalities have been already met. The first prototype has been tested against a corpus of NMR test data representing a year of Press and TV Spots divided for the two domains tackled by the project (ICT and car spots).
The Media Campaign system can be seen as a collection of integrated components that share the same communication mechanisms. Each component provides tools for component configuration and administration. The overall system architecture is depicted in the picture below.
The main parts of the prototype system are:
- Media Acquisition
- Data Management Store
- Media Analysis
- Knowledge Fusion and Campaign Discovery
- Delivery System
Media Acquisition
 |
The Acquisition (ACQ) component acquires essences from different media sources and can be seen as a modularised producer-consumer system. It centralises the acquisition of different media essences (images and streams), which is carried on by specialised grabbers. The individual grabbers are autonomous (and potentially remote) components. The interaction between the different grabbers and the ACQ component is file system based, although the transfer library can decide which protocol is best suited for the specific deployment case. Interaction with the EMS component (which can be seen as a consumer) is via web services. The ACQ component has been implemented in Perl 5 (5.8.7), so it can be installed on Windows MS, Linux and all platforms where Perl has been ported. It stores data on flat files (although the use of a DBMS can be decided and configured at runtime), so no DBMS is needed for a standard installation. Being an event-based component, hardware resources (CPU/RAM) consumption is rather small. Disk consumption is related with the size of the ACQ circular buffer. |
Data Management Store
EMS Knowledge Store
The Essence Management Store (EMS) component is the essence storage for the Media Campaign system. Its main duties are essence acquisition (upload) from the ACQ component and essence distribution (download) to the rest of the system. The key module of the component is the essence data store, composed by a DBMS engine and different file system partitions: an online partition, and an offline partition where essences are packed and compressed (can be moved to online if requested).
The first prototype of the EMS component is build entirely in Perl 5 (5.8.7) and can be installed under Windows, Linux or any environment where Perl has been ported. Different DBMS products can be used as data storage, MySQL 5 has been used to test the EMS first prototype, but ORACLE or PostgreSQL could be used as well (no specific triggers/stored procedures are needed and DB model is represented in standard SQL92 format/data types). For test purposes a file system based data storage can be configured as well. Data source type and connection parameters are specified in the EMS configuration file, no changes to the code are needed shifting from one data source type to another.
MMS Knowledge Store
Main purposes of Metadata Management Store (MMS) are to serve as exchange point between several analysis subsystems and its function as a static metadata repository. Generally there are two different parts of MMS component that can be distinguished: MMS Data Repository (metadata storage) and MMS Web Service (API). MMS Data Repository stores metadata output of analysis workflow and contains basic information on “creatives” and “spots”. It is implemented in classical manner as a relational database. MMS Web Service provides basic set of functions acting as interfaces that allow other system components to retrieve and manipulate data from storage.
OMS Knowledge Store
The Ontology Metadata Store (OMS) serves as a semantic repository providing storage and inference capabilities over formal knowledge expressed in RDF or OWL. OMS is based on the OWLIM semantic repository engine by Ontotext. The MEPCO and PROTON ontologies are being loaded in OMS and serve as schema basis for the modelling of spots, creatives, campaigns and related entities (organizations, brands, dates, etc.). The OMS is being used by the knowledge fusion and campaign discovery module (KFCD), offering a repository for spots, creatives and the resulting campaigns. The Delivery System is another component basing on the OMS, where from it derives information about campaigns and their relationship to creatives.
Media Analysis
Collection Processing Manager
The Collection Processing Manager (CPM) manages the entire workflow for the analysis modules. The main tasks of the CPM are to receive new essence messages from the EMS, organise them into jobs and choose the proper workflow for each one depending on the type of the essence. The CPM provides SOAP web services for the communication with the analysis modules and the EMS.
Press Workflow
 |
Optical Character Recognition (OCR) is used to recognise text in advertisements. Text is presented as artificial overlays in these images/videos and on real world objects. There are two different OCR modules, one for videos in the TV workflow and one for single images in the press and internet workflow. These two modules mainly differ in the fact, that text recognition is performed on different media types but the same technology and algorithms are used for both tasks. The Text Analysis module analyses text transcripts coming from the OCR and ASR modules and identifies instances of entities in the MEPCO ontology, in particular Advertiser and Product. The initial prototype works for English documents only. The Image Fingerprinting component tries to find existing creatives, which are identical or similar to the investigated spot. This analysis is done with the help of visual features, such as the colour similarity, and is divided in two steps: similarity search and exact matching. The similarity search is a fast comparison of the input spot with the whole creative database and it returns an ordered list of the most similar image. The exact matching step uses the results of the similarity search to find out which images from the creative database are identical to the input spot. The Creative Detector is used to combine the results of different analysis modules in the press workflow. Therefore, the creative detector extracts relevant information from all available analysis modules belonging to an investigated spot. This information is than weighted according to the quality (precision/recall) of the module before it is combined to a merged analysis result. The AdComparer is used for manual ad comparing in the press workflow using the results of the creative detector as input. Ambiguous results can be clarified by the user using an auto suggestion mode or automatic classifications can be verified or overruled. The output of the components is a revised version of the output of the creative detector. |
TV Workflow
 |
The Audio Segmentation module creates a temporal segmentation of an audio file into the classes “speech”, “music”, “other” (sounds), “silence”, and all possible mixtures of “speech”, “music” and “other”. The result is a valid MPEG-7 document containing timecode and classification terms for each segment. The Jingle Recognition module tries to recognise previously learnt reference jingles in the audio data of a TV spot. If a jingle is found, a segment is added to the MPEG-7 description of the Audio Segmentation input containing timecode and the associated brand metadata. The reference jingles are learnt by extracting a fingerprint from jingle audio files. These fingerprints are stored in a jingle database. The metadata is supplied by MPEG-7 descriptions of the jingles. The Word Spotting modules produce location information about when a word or concept is found plus a confidence measure within speech parts of television commercials (speech in isolation and in music). The Speech to Text modules produce full text transcripts with word timing information of those parts in television commercials that contain speech (speech in isolation and in music). Currently speech-to-text for Dutch and English is available. Logo Recognition is performed to find out to which product or company an advertisement belongs. The logo recognition module works with a number of previously learnt logos (one or several images per logo) and tries to recognise these logos in a video. It is possible to recognise learnt logos everywhere in an image at different sizes, rotated, with minor colour changes, under different illumination conditions and perspective distortions. Video Fingerprinting tries to find existing creatives, which are identical or similar to the investigated spot. For this task the input spot is first fragmented into shots with a simple hardcut shot detection. From each shot the first frame, unless it is a blackframe, is used as keyframe. Each keyframe of the input spot is than matched against all the keyframes of the known creatives using the similarity search of the image fingerprinting module. Then the results of all the input keyframes are merged and the shot structure of the input spot is matched against the shot structure of the best matching creatives from the database to find out if they are identical or similar to the input spot. The OCR, Text Analysis and Creative Detector components are similar to the ones used in the press workflow, only adapted for video input. |
Knowledge Fusion and Campaign Discovery
The Knowledge Fusion module gathers information about new Creatives generated by the Creative Detector and converts that into a RDF representation related to the MEPCO ontology. This RDF representation is sent to the Campaign Detection module which stores it in the OMS by creating a new instance of the class Creative. The Campaign Detection module is available as a web service and determines whether a new Creative is part of an existing Campaign or the beginning of a new one. The relation between Creatives and Campaigns is stored in the OMS.
The Knowledge Fusion module connects to the MMS and checks whether new Creatives have been discovered. If this is the case, these Creatives are sent to the Campaign Detection module, which is available as a web service and provides an interface to the OMS.
Delivery System
 |
The Delivery component is a web portal, used to browse and analyse multimedia related data. The portal lets the operator browse the content of the three project data storages EMS, MMS and OMS, providing also useful links among them. The portal can be seen as a collection of services:
|