Cygnus is a connector in charge of persisting certain sources of data in certain configured third-party storages, creating a historical view of such data.
Internally, Cygnus is based on Apache Flume, a technology addressing the design and execution of data collection and persistence agents. An agent is basically composed of a listener or source in charge of receiving the data, a channel where the source puts the data once it has been transformed into a Flume event, and a sink, which takes Flume events from the channel in order to persist the data within its body into a third-party storage.
Cygnus is designed to run a specific Flume agent per source of data.
Current stable release is able to persist the following sources of data in the following third-party storages:
- NGSI-like context data in:
- HDFS, the Hadoop distributed file system.
- MySQL, the well-known relational database manager.
- CKAN, an Open Data platform.
- MongoDB, the NoSQL document-oriented database.
- STH Comet, a Short-Term Historic database built on top of MongoDB.
- Kafka, the publish-subscribe messaging broker.
- DynamoDB, a cloud-based NoSQL database by Amazon Web Services.
- PostgreSQL, the well-known relational database manager.
- Carto, the database specialized in geolocated data.
- Twitter data in:
Documentation and API