Tag Archives: pubsub

DynPubSub: A Peer To Peer Overlay For Topic-Based Pub/Sub Systems Deployed at the Edge

Bouallegue, C., Gascon-Samson, J. (2020) DynPubSub: A Peer To Peer Overlay For Topic-Based Pub/Sub Systems Deployed at the Edge. Proceedings of the 21st International Middleware Conference Demos and Posters (Middleware 2020 Demos and Posters)
[Preprints] [Poster]

Abstract: There are more and more IoT devices that produce and consume and ever increasing amount of data. Publish-subscribe (Pub/Sub) is a well known paradigm that simplifies the task of exchanging messages, as it decouples the communication between the entities that emit and consume messages. While traditionally deployed in a centralized cloud-based manner, the different components of a pub/sub system can be deployed directly onto the edge devices, in a peer-to-peer manner, to achieve the required low latency for most IoT applications. In this poster, we propose DynPubSub, a new peer-to-peer network overlay for topic based pub/sub systems deployed at the edge. DynPubSub provides fault tolerance and scalability, and aims at minimizing the latency while respecting the constraints of the edge devices and networks.

Scalable Edge Computing for Low Latency Data Dissemination in Topic-Based Publish/Subscribe

Khare, S., Sun, H., Zhang, K., Gascon-Samson, J., Gokhale, A., Koutsoukos, K., Abdelaziz H. (2018) Scalable Edge Computing for Low Latency Data Dissemination in Topic-Based Publish/Subscribe, 2018 IEEE/ACM Symposium on Edge Computing (SEC 2018), Seattle, WA, USA
[Preprint] [Presentation Slides]

Abstract: Advances in Internet of Things (IoT) give rise to a variety of latency-sensitive, closed-loop applications that reside at the edge. These applications often involve a large number of sensors that generate volumes of data, which must be processed and disseminated in real-time to potentially a large number of entities for actuation, thereby forming a closed-loop, publish-process-subscribe system. To meet the response time requirements of such applications, this paper presents techniques to realize a scalable, fog/edge-based broker architecture that balances data publication and processing loads for topic-based, publish-process-subscribe systems operating at the edge, and assures the Quality-of-Service (QoS), specified as the 90th percentile latency, on a per-topic basis. The key contributions include: (a) a sensitivity analysis to understand the impact of features such as publishing rate, number of subscribers, per-sample processing interval and background load on a topic’s performance; (b) a latency prediction model for a set of co-located topics, which is then used for the latency-aware placement of topics on brokers; and (c) an optimization problem formulation for k-topic co-location to minimize the number of brokers while meeting each topic’s QoS requirement. Here, k denotes the maximum number of topics that can be placed on a broker. We show that the problem is NP-hard for k >=3 and present three load balancing heuristics. Empirical results are presented to validate the latency prediction model and to evaluate the performance of the proposed heuristics.

Poster Abstract: Ensuring Low-Latency and Scalable Data Dissemination for Smart-City Applications

Khare, S., Sun, H., Zhang, K., Gascon-Samson, J., Gokhale, A., Koutsoukos, K. (2018) Poster Abstract: Ensuring Low-Latency and Scalable Data Dissemination for Smart-City Applications, IoTDI 2018, Orlando, USA

Abstract: Low latency and scalable data dissemination is a critical requirement for many IoT applications, e.g., smart city applications, which are often built over a publish/subscribe communication paradigm. Ensuring low latency requires effective load balancing of the publish/subscribe topics across the different publishers and subscribers. To that end we present ongoing work on a data-driven approach to learning a latency-aware model of IoT broker loads, and in turn using it to determine broker replication, and balancing topics across them.

CacheDOCS: A Dynamic Key-Value Object Caching Service

Gascon-Samson, J., Coppinger, M., Jin, F., Kienzle, J., Kemme, B. (2017) CacheDOCS: A Dynamic Key-Value Object Caching Service, ICDCS-PED2017, Atlanta, USA
[Preprint] [Presentation Slides]

Abstract: Caching plays an important role in many domains, as it can lead to important performance improvements. A key-value based caching system typically stores the results of popular queries in efficient storage locations. While caching enjoys widespread usage in the context of dynamic web applications, most mainstream caching systems store static binary items, which makes them impractical for many real-world applications that would benefit from storing dynamic items. In this paper, we propose CacheDOCS, a dynamic key-value object caching service that allows for caching arbitrary objects. As part of our model, CacheDOCS provides an API that supports the execution of operations against cached objects, and allows for clients to seamlessly subscribe to keep their local copies in sync with cached remote objects. CacheDOCS supports multiple update dissemination strategies in order to optimize performance, and proposes a versioning mechanism to ensure consistency. We implemented a full version of CacheDOCS and we ran several performance-related experiments under three use-case scenarios.

MultiPub: Latency and Cost-Aware Global-Scale Cloud Publish/Subscribe

Gascon-Samson, J., Kemme, B., Kienzle, J. (2017) MultiPub: Latency and Cost-Aware Global-Scale Cloud Publish/Subscribe, ICDCS 2017, Atlanta, USA [Preprint] [Presentation Slides]

Abstract: Topic-based pub/sub is a widely used communication mechanism in distributed systems for targeted information dissemination between loosely coupled entities. To scale dynamically depending on the current communication demands, pub/services can be conveniently deployed in the cloud. To provide fast dissemination, the service can be distributed across multiple cloud regions. The architectural design and run-time deployment of such a middleware is tricky, though, as it can have a significant effect on communication latency and cloud-based cost. In this paper, we propose MultiPub, a flexible pub/sub middleware for latency-constrained, world-wide distributed applications that dynamically reconfigures the communication layer to ensure a predefined maximum latency for publication dissemination while minimizing cloud-based costs. This is achieved by routing publications either through a single or across multiple cloud regions. We demonstrate the effectiveness of MultiPub by presenting a set of experiments that report on the achieved communication latency and cost savings compared to traditional approaches, as well as a performance evaluation.

DynFilter: Limiting Bandwidth of Online Games using Adaptive Pub/Sub Message Filtering

Gascon-Samson, J., Kienzle, J., Kemme, B. DynFilter: Limiting Bandwidth of Online Games using Adaptive Pub/Sub Message Filtering, NetGames 2015, Zagreb, Croatia
[Preprint] [Presentation Slides]

Abstract: Multiplayer online games can generate a lot of server-related outgoing bandwidth, due to many factors such as highly variable amounts of players or the gathering of many players towards the same in-game locations. Predicting the exact amount of required bandwidth to support varying conditions can be costly, and players can experience game-wide failures if bandwidth is insufficiently provisioned. We present DynFilter, a game-oriented message processing middleware designed to adaptively filter state update messages for in-game entities located apart, in order to reduce bandwidth needs and stay within predefined quotas. We ran experiments on Amazon EC2 over a prototype game mimicking a FPS and a MMOG. Our results show that DynFilter is properly able to maintain bandwidth use within the pre-established quotas while still maintaining adequate delivery of relevant state update messages.

Dynamoth: A Scalable Pub/Sub Middleware for Latency-Constrained Applications in the Cloud

Gascon-Samson, J., Garcia, F.-P., Kemme, B., Kienzle, J. (2015) Dynamoth: A Scalable Pub/Sub Middleware for Latency-Constrained Applications in the Cloud, ICDCS 2015, Columbus, USA
> Acceptance ratio: 12.8% [Preprint] [Presentation Slides]

Abstract: This paper presents Dynamoth, a dynamic, scalable, channel-based pub/sub middleware targeted at large scale, distributed and latency constrained systems. Our approach provides a software layer that balances the load generated by a high number of publishers, subscribers and messages across multiple, standard pub/sub servers that can be deployed in the Cloud. In order to optimize Cloud infrastructure usage, pub/sub servers can be added or removed as needed. Balancing takes into account the live characteristics of each channel and is done in an hierarchical manner across channels (macro) as well as within individual channels (micro) to maintain acceptable performance and low latencies despite highly varying conditions. Load monitoring is performed in an unintrusive way, and rebalancing employs a lazy approach in order to minimize its temporal impact on performance while ensuring successful and timely delivery of all messages. Extensive real-world experiments that illustrate the practicality of the approach within a massively multiplayer game setting are presented. Results indicate that with a given number of servers, Dynamoth was able to handle 60% more simultaneous clients than the consistent hashing approach, and that it was properly able to deal with highly varying conditions in the context of large workloads.