Exploitation of Social Media for Emergency Relief and Preparedness (SMERP)

The motivation of the data challenge is to promote development of IR methodologies that can be used to extract important information from social media during emergency events, and to arrange for comparative evaluation of the methodologies. There will be two sub-tracks / challenges:

• Text Retrieval
• Text Summarization

Participants can choose to participate in either or both of the sub-tracks.

Dataset

Tweet collection: A large dataset of microblogs (tweets) posted on Twitter during the earthquake in Italy in August 2016, will be provided. Since the Twitter terms do not allow public sharing of tweets, only the tweetids of the tweets will be provided, along with a Python script that can be used to download the tweets using the Twitter API.
Topics: A set of topics (information needs) will be provided. Each topic will identify a broad information need during a disaster, such as – what resources are needed in the disaster­ affected area, what resources are available, what damages are being reported, etc. Specifically, each topic will be in TREC format, and will contain a title, a brief description, and a more detailed narrative on what type of tweets will be considered relevant to the topic.

Text Retrieval sub-track

In this sub-track, the participants are required to develop methodologies for extracting tweets that are relevant to each topic, with high precision (i.e., ideally, only the relevant tweets should be identified) as well as high recall (i.e., ideally, all relevant tweets should be identified).
This sub-track will have two levels:

Level 1

In this level, the tweets collected during the first day (24 hours) after the earthquake will be provided, and the participants will be asked to extract tweets relevant to each specified topic. The extracted tweets should be submitted to us, and we will arrange for evaluation of some of the top-ranked results of each submission.

Level 2

In this level, the tweets collected during the second day (24 hours) after the earthquake will be provided. Additionally, some of the tweets that are actually relevant to each topic, from among the tweets used in Level 1, will be provided. The participants will again be expected to retrieve tweets relevant to each topic from among the tweets posted during the second day.
The purpose of the second level is to provide the participants some idea about which tweets are considered relevant, so that they can apply this knowledge to improve retrieval on an unseen set of tweets. Thus, this exercise will present a platform to evaluate the dynamic reusability and adaptability of the IR system in a practical scenario.

Evaluation

The retrieval results will be evaluated by Mean Average Precision (MAP) and Recall.

Text Summarization sub-track

In this sub-track, the participants are required to develop methodologies for summarizing the information that is relevant to each topic. Specifically, summaries of length at most 300 words should be generated for each topic. Both extractive and abstractive summarization methods are welcome.
This sub-track will also have two levels:

Level 1

In this level, the tweets collected during the first day (24 hours) after the earthquake will be provided, and the participants will be required to submit a summary (of maximum 300 words) for each topic. We will arrange for evaluation of the summaries.

Level 2

In this level, the tweets collected during the second day (24 hours) after the earthquake will be provided. Additionally, we will indicate which parts of the summaries produced from the tweets of the first day, are important for the second day. The participants will again be required to submit summaries produced from the tweets of the second day. It will be expected that the parts that were indicated to be important after the first day, will be given more importance while generating the summaries for the second day.
The purpose of the second level is to provide the participants some idea of which types of information are important for each topic after the first day, so that they can tune the algorithms while generating the summaries on the tweets of the second day. Thus, this exercise will present a platform to evaluate the dynamic adaptability of the summarization system in a practical scenario.

Evaluation

The Text Summarization results will be evaluated in two ways:

1. Using automatic measures like ROUGE score, BLEU score, etc.
2. Using manual assessment of the quality of the summaries

Note

• A participating team should contain between 1 and 5 members.
• For both the sub-tracks, a team can participate in Level 1 only, or in both the levels. However, a team can not participate in Level 2 only (i.e., to participate in Level 2, participation in Level 1 is mandatory).
• The two teams with the best performances in each sub-track will be invited to submit papers based on their methodologies, which will be included in the workshop proceedings (through a process of shepherding).
• Workshop proceedings will be archived online in the CEUR workshop proceedings publication service (http://ceur-ws.org/, ISSN 1613-0073).

Communication

Interested participants should send email to smerp2017 [at] gmail [dot] com, stating the names and affiliations of the team members, and which sub-track(s) they wish to participate in (Text Retrieval or Text Summarisation or both). The organizers will send the dataset by replying to the mail. For both sub-tracks, there should be separate data requests for Levels 1 and 2. All further correspondences, viz., result submission, evaluation result declaration and request for paper submission, will be done via the same mail id.

Important dates for Data Challenge Track

Results of Data Challenge Track : Task 1 (Extraction) and Task 2 (Summarization)