-
A system to collect real-time location coordinates data from an android app and send them to a server via heterogeneous network channels viz Bluetooth, Wire, and Wi-fi
-
Data is then pushed to the public broker of Hive-MQ by the publisher server which is later collected by subscribers for storing in a Gluster parallel file system consisting of 6 storage targets.
-
Note : This Repo also contains assignment code for course Data Storage Technologies and Networks
-
Note : The storage system is migrated from GlusterFS to BeeGFS File System and might have some issues.
Geofencing is a location-based technology that uses GPS, RFID, or other technologies to create a virtual boundary around a specific geographic area. This virtual boundary can be used to trigger certain actions or events when a device enters or leaves the defined area. For example, a geofence could be used to send a notification to a user's smartphone when they arrive at a specific location or to control access to a building or other secure area.
Geofencing has several benefits. One of the main benefits is that it allows for location-based automation, which can save time and improve efficiency. For example, a geofence could be used to automatically open the doors to a building when a user arrives within a certain distance of the entrance. Additionally, geofencing can be used to improve security by triggering alerts or other actions when a device enters or leaves a defined area. For example, a geofence could be used to send an alert to security personnel if a device enters a restricted area. Geofencing can also be used for marketing and advertising, as it allows businesses to send targeted messages to users based on their location.
We build an application for Geofencing which sends the live coordinates of a device to a parallel file-system. The system communicates with the file system using the MQTT protocol (because of its efficiency) and uses a parallel file-system (because of its scalability) for storage of the data.
The client (which is a publisher for the MQTT), the broker, and the subscriber which is responsible for sending the data to the file system.
Fetch and store API for the data sent by Group 1. Storing the data on parallel file system BeeGFS with scheduled compression algorithms.
To send the data from the client to the distributed file system for storage, we use a MQTT (lightweight open messaging protocol that provides resource-constrained network clients with a simple way to distribute telemetry information in low-bandwidth environments) broker, specifically HiveMQ. Initially, we considered using Kafka and ZeroMQ for the communication between the clients and the file system. Kafka being highly scalable was the first choice; however, we didn't have the hardware infrastructure to leverage the scalability. With ZeroMQ, it provided a lot of APIs for lower-level implementation, but given the time constraint, we decided against using it. We went with HiveMQ because of the HiveMQ Cloud Platform, which allows easier communication without requiring physical connections between devices or poses the constraint of having them on a single device.
The client is an Android App (using Android SDK) with core modules written in Java, whereas the UI is developed in Kotlin. This is because of the familiarity of developers with the respective languages. The App uses the Android location API to get the live coordinates of the device.
The client app subscribes to the MQTT broker and listens for messages from the device. When the device's location changes, the app sends the updated coordinates to the broker using the MQTT publish method. The MQTT broker then forwards the message to the subscriber, which is responsible for storing the data on the parallel file system.
One of the main challenges we faced was ensuring reliable and efficient communication between the client and the MQTT broker. We had to handle cases where the client lost internet connectivity or where the broker was unavailable. We implemented retry mechanisms to handle such scenarios and ensure that no data was lost.
Another challenge was handling the high volume of data generated by the devices. We had to ensure that the file system could handle the incoming data without any performance issues. We used BeeGFS, a parallel file system, for storing the data, which allowed for efficient and scalable storage.
Our implementation successfully sends the live coordinates of the device to the parallel file system using the MQTT protocol. We tested the system with multiple devices and observed that the data was accurately stored on the file system. We also measured the performance of the system and found that it could handle a large volume of data without any issues.
Group 2 worked on developing the fetch and store API for the data sent by Group 1. We used Python Flask, a micro web framework, to develop the API. Flask provides a simple and easy-to-use interface for building web applications. We chose Flask because of its simplicity and the availability of libraries and tools for building APIs.
We developed two APIs: one for fetching the data from the file system and another for storing new data. The fetch API accepts parameters like the device ID and time range and retrieves the corresponding data from the file system. The store API accepts the data sent by the Group 1 client and stores it on the file system.
To store the data on the file system, we used BeeGFS, a parallel file system known for its high performance and scalability. BeeGFS allowed us to efficiently store and retrieve the large volume of data generated by the devices.
We also implemented scheduled compression algorithms to reduce the storage space required by the data. This helped in optimizing the storage and reducing costs.
One of the main challenges we faced was handling concurrent requests to the APIs. As the system was expected to handle a large number of devices and multiple requests simultaneously, we had to ensure that the APIs could handle concurrent requests without any issues. We implemented thread-safe mechanisms and optimized the code to handle concurrent requests efficiently.
Another challenge was optimizing the storage and reducing costs. The high volume of data generated by the devices required efficient storage mechanisms. We implemented compression algorithms to reduce the storage space required by the data, thereby reducing costs.
Our implementation of the fetch and store APIs successfully handled concurrent requests and efficiently stored and retrieved the data from the parallel file system. We tested the APIs with multiple requests and observed that they could handle the load without any issues. The scheduled compression algorithms also effectively reduced the storage space required by the data.
In this project, we developed an application for Geofencing that sends live coordinates of a device to a parallel file system. We used the MQTT protocol for efficient communication between the client and the file system and BeeGFS, a parallel file system, for scalable storage of the data. Our implementation successfully handled the high volume of data generated by the devices and provided reliable and efficient storage and retrieval of the data. Geofencing has numerous applications in various industries, and our system can be further extended and customized