The following is an update from a member of the development team on the project's progress:
Identifying Trends
The central idea we're building upon is to identify Twitter and Facebook trends across a pre-defined set of keywords and keyword groups. While this tool will never tell you where exactly what is going on, it allows the team to identify certain trends across different geo-boundaries. This will help to identify anomalies which could be indicators that something is going on.
Building Out the Data Pipeline
I've been working on the data pipeline: basically from ingesting data from Twitter, de-duping the messages, perform the natural language processing and the inference of keywords and locations. We made some great progress across all these components and further developed the ability to fuzzy match keywords and locations.
The More Data, the More the Machine Can Learn
That said, not understanding Arabic is a major challenge when it comes to NLP and inference. While we worked with translators and the on-the-ground team, we decided to constrain our work to English data streams only for now. it just reminded me once more how difficult it is to build solutions for a language one is not capable of speaking and writing.
Having more data will help us to refine our approach, especially in the area of machine learning and inference. This will be ongoing. The UN team needs to run the tool as a service and apply dev-ops practices. It needs to constantly refine and retrain the models.
.svg)
