Amazon’s early principles of distributed system design to meet Amazon S3 requirements:
- Decentralisation: Use fully decentralised techniques to remove scaling bottlenecks and SPOF
- Asynchrony: The system makes progress under all circumstances.
- Autonomy: The system is designed such that individual components can make decisions based on local information.
- Local Responsibility: Each individual component is responsible for achieving its consistency; this is never the burden of its peers.
- Controlled Concurrency: Operations are designed such that no or limited. concurrency control is required.
- Failure Tolerant: The system considers the failure of components to be a normal mode of operation and continues operation with…
In the current connected world — Websites, Mobile Apps, IoT Devices collect a large volume of users’ personally identifiable activity data. These collected data is used for varied purposes of analytics, marketing, personalisation of services, etc. Data is assimilated through site cookies, tracking device IDs, embedded JavaScripts, Pixels, etc. to name a few. Many of these tracking and usage of collected data happens behind the scene and is not apparent to an average user. Consequently, many Countries and Regions have formulated legislations (e.g. …
This is a quick note on the key considerations that I have made over the years while designing and developing large scale distributed real time data processing systems. These principles follow evolving usage patterns and use cases.
- Processing Semantics
- “Exactly Once” processing semantics or “at-least once”. Choose your semantics based on your use case needs. It is possible to configure different semantics for each use case.
- Consistency (strict — eventual mix) over Availability. Refer CAP Theorem, and the fact that you must account for network partitioning.
- In-memory Data Grid (IMDGs) or Caching layer that support Transactions — preferably ACID complaint…
Subhadip Mitra
Distributed Systems, Artificial Intelligence, Blockchain, Theoretical Physics, Open Source
