Chapter 1: Big Data Analysis and Extraction Techniques
This chapter introduces Big Data, its sources, analysis tools, analytics techniques, and the preprocessing steps needed before modelling or decision-making.
1.1 Big Data
Big Data describes extremely large datasets that are too voluminous, complex, or fast-moving for traditional data-processing tools to handle effectively. Managing it requires specialized technologies for storage, processing, analysis, and retrieval.
The guide links Big Data directly to efficient storage and analysis: organizations must manage volume, variety, velocity, and veracity so they can extract value.
1.2 Characteristics of Big Data: The Five Vs
| V |
Meaning |
Revision cue |
| Volume |
Huge amounts of data, often growing into terabytes, petabytes, exabytes, or more. |
Storage scale and cloud/distributed systems. |
| Variety |
Different forms of data: structured, semi-structured, and unstructured text, images, audio, and video. |
Choose flexible tools such as NoSQL or data lakes. |
| Velocity |
The speed of data generation and the speed at which it must be analysed or acted on. |
Streaming, real-time analytics, Kafka, Flink. |
| Veracity |
The accuracy, reliability, uncertainty, and trustworthiness of data from multiple sources. |
Cleaning, validation, governance. |
| Value |
The usefulness gained from analysing data and turning it into knowledge or action. |
Analytics must support decisions. |
Sources of Big Data
Sources include business transactions, social media interactions, IoT sensor streams, digital activity, healthcare records, scientific experiments, and government/public-service datasets.
1.3 Common Tools and Frameworks
| Tool |
Primary purpose |
Best fit from the guide |
| Hadoop |
Distributed storage and batch processing using HDFS and MapReduce. |
Large structured, semi-structured, or unstructured datasets with fault tolerance. |
| Apache Spark |
In-memory big data analytics for batch, streaming, and machine learning workloads. |
Faster processing and mixed batch/stream use cases. |
| Apache Kafka |
Distributed event streaming and high-throughput real-time data flows. |
High-velocity data streams and durable event messaging. |
| MongoDB |
NoSQL document database for semi-structured and unstructured data. |
Flexible, schema-light data storage. |
| Tableau / Power BI |
Visualization dashboards, charts, and insight sharing. |
Communicating findings from large datasets. |
| AWS / Azure-style cloud platforms |
Cloud processing, storage, analytics, and scalable infrastructure. |
Elastic and cost-managed big data workflows. |
| Apache Flink |
Low-latency stream and batch processing. |
Event-driven and real-time applications. |
Choosing a tool: match the tool to data type, processing need, scalability, and affordability.
1.4 Big Data Analysis Techniques
Descriptive analysis
Summarizes historical data to answer what happened, often through reports, dashboards, and infographics.
Predictive analytics
Uses historical data and machine learning models to forecast future events such as demand, defaults, or market trends.
Machine learning and deep learning
Allows systems to learn from data, improve over time, classify records, cluster groups, and model complex patterns.
Real-time analytics
Processes data as it is generated so decisions can happen immediately, such as fraud detection or IoT maintenance alerts.
Sentiment analysis
Uses natural language processing to classify text as positive, negative, or neutral.
1.5 Data Cleaning, Normalization, and Transformation
| Step |
Purpose |
Examples |
| Cleaning |
Find and correct errors, inconsistencies, duplicates, missing values, and outliers. |
Imputation, duplicate removal, typo correction, date/number format standardization. |
| Normalization |
Standardize or scale data to make it easier to compare and process. |
Min-max scaling, z-score standardization, log transformation, label or one-hot encoding. |
| Transformation |
Convert raw data into formats suitable for analysis or model construction. |
Aggregation, encoding, wrangling, feature engineering, type conversion. |
Case study route: for the RetailMax pipeline, mention each source, clean missing/duplicate records, normalize customer/product fields, transform JSON/CSV/MongoDB records into a unified customer view, then segment by purchase behavior, lifetime value, and preferences.
1.1.1 Data forms and the extraction problem
The guide’s Big Data discussion assumes that useful data arrives in different forms and at different speeds. Structured data fits a defined schema such as transaction rows. Semi-structured data carries tags or flexible fields, such as JSON logs. Unstructured data includes posts, images, audio, video, documents, and other content without a fixed table structure.
| Form | Example in the guide | Extraction implication |
|---|
| Structured | Banking, sales, and transaction records. | Can be queried after schema checks and type validation. |
| Semi-structured | JSON mobile-app logs and sensor messages. | Fields must be parsed, aligned, and converted to common names and types. |
| Unstructured | Social posts, comments, images, and healthcare documents. | Needs text, media, or document extraction before analysis. |
Exam trap: Big Data is not defined only by size. Variety, velocity, veracity, and the value that can be extracted are equally important in the guide.
1.2.1 Sources of Big Data
Different sources explain why Big Data is both large and varied. A strong answer names the source, the form of data it produces, and the type of insight it can support.
| Source | Typical data | Possible insight |
|---|
| Business transactions | Purchases, banking, finance, orders. | Sales performance, customer value, and financial trends. |
| Social media | Posts, comments, likes, shares. | Human interaction, sentiment, and emerging trends. |
| IoT devices | Sensor readings, wearables, industrial equipment. | Real-time monitoring, automation, and predictive maintenance. |
| Web and digital activity | Searches, clicks, browsing, adverts, e-commerce behaviour. | Preferences, journeys, campaign response, and personalization. |
| Healthcare | Electronic records, diagnostic images, wearable health data. | Personalized care, treatment outcomes, and research. |
| Scientific and public sources | Experiments, observations, government and public-service records. | Research patterns, planning, and service improvement. |
1.3.1 Tool-selection worked example
Suppose a retailer receives live checkout events, stores customer documents, and needs a management dashboard. The source guide’s tools fit together as a pipeline rather than as competing answers.
CollectKafka receives high-velocity event streams.
StoreMongoDB keeps flexible customer documents; HDFS keeps distributed files.
ProcessSpark analyses batch and streaming data in memory.
ModelMLlib or another analytics layer finds patterns and predictions.
CommunicateTableau or Power BI turns results into dashboards.
Why this works: Hadoop is emphasized for distributed storage and MapReduce processing; Spark for faster in-memory batch/stream analytics; Kafka for durable high-throughput events; MongoDB for flexible document data; cloud platforms such as AWS for scalable services; Flink for low-latency stream and batch processing.
1.4.1 From description to action
The analysis techniques in the guide answer different questions. They can be combined: descriptive analysis shows a sales drop, diagnostic analysis investigates complaints or operational issues, predictive analytics forecasts demand, and prescriptive analysis can recommend inventory or marketing action.
| Technique | Question | Guide-based application |
|---|
| Descriptive analysis | What happened? | Reports, dashboards, and infographics summarize historical business, healthcare, marketing, or finance performance. |
| Predictive analytics | What is likely to happen? | Historical data and models forecast demand, risk, behaviour, or market trends. |
| Machine/deep learning | Can a system learn patterns? | Classification, clustering, and complex pattern detection support automated decisions. |
| Real-time analytics | What is happening now? | Streaming data supports immediate fraud alerts, maintenance warnings, or operational responses. |
| Sentiment analysis | How do people feel? | Natural-language processing classifies text as positive, negative, or neutral. |
1.5.1 RetailMax preprocessing walkthrough
RetailMax has an Oracle CRM, PostgreSQL e-commerce data, JSON app logs, CSV social-media exports, and MongoDB service tickets. The workflow should be explained in this order:
- Profile and ingest: record the source, timestamp, owner, format, and expected fields for every input.
- Clean: flag or impute missing values, remove duplicate customers, correct typos, standardize dates/numbers/categories, and inspect outliers.
- Normalize: scale numeric features where comparison is needed, encode categories, and use a consistent customer/product key.
- Transform: parse JSON, flatten nested tickets, aggregate purchases, calculate counts/totals, and engineer features such as frequency and lifetime value.
- Validate and load: check row counts, types, referential links, and quality rules before loading an analysis-ready customer view.
Impact of skipping preprocessing: duplicate customers can inflate segment sizes; inconsistent dates can distort trends; missing values can bias models; and extreme values can make averages or predictions misleading.
Useful formulas: min-max = (x - min) / (max - min); z-score = (x - mean) / standard deviation. Use the first to map values to a range such as 0–1 and the second to center values around 0.
Chapter 1 Glossary: Big Data and preprocessing terms
Big Data- Large, complex, fast, or varied datasets that require specialized storage, processing, and analysis.
Volume- The amount of data, ranging in the guide from terabytes and petabytes toward larger scales.
Variety- The mix of structured, semi-structured, and unstructured data forms.
Velocity- The speed at which data is generated, moved, processed, or acted on.
Veracity- The accuracy, reliability, consistency, and trustworthiness of data.
Value- The useful insight or action obtained from data analysis.
HDFS- Hadoop Distributed File System; distributed storage that splits files into blocks and replicates them across nodes.
MapReduce- Hadoop’s parallel processing model using map tasks and reduce tasks.
Kafka- A distributed event-streaming platform for collecting, storing, and moving high-throughput data flows.
Cleaning- Finding and fixing missing values, duplicates, errors, inconsistent formats, and problematic outliers.
Normalization- Standardizing scale or structure so values can be compared or processed efficiently.
Transformation- Converting raw data into a format suitable for analysis or model construction.
Min-max scaling(x - min) / (max - min); rescales a feature to a bounded range such as 0–1.Z-score(x - mean) / standard deviation; standardizes a value relative to the dataset mean.Imputation- Replacing a missing value using a rule such as the mean, median, or a model-based estimate.
Feature engineering- Creating useful derived attributes, such as purchase frequency or lifetime value.
Back to Exam Focus Areas
Chapter 2: IoT Architectures and Applications
This chapter explains the IoT ecosystem, device-to-cloud data flow, layered architecture, security challenges, and major IoT application domains.
2.1 Core IoT Concepts and Principles
IoT is a network of physical objects, sensors, software, and technologies that communicate and share information online. It supports real-time data collection, sharing, analysis, automation, and better decisions.
Devices and sensors: collect environmental data or perform physical actions.
Connectivity: Wi-Fi, Bluetooth, Zigbee, Ethernet, 4G/5G, and other links.
Data processing: edge computing near the device or cloud computing in centralized systems.
Platforms: middleware for device management, data analysis, and application development.
Applications: user interfaces and dashboards for monitoring and control.
Security frameworks: encryption, authentication, and access control.
2.2 How IoT Works
IoT devices communicate over the internet or local networks. Data is captured by sensors, sent through an IoT gateway, processed locally at the edge or centrally in the cloud, and then used to trigger analysis, alerts, dashboards, or actuator commands.
Sense Devices capture data such as temperature, motion, location, or health metrics.
Connect Networks move data through gateways or directly to platforms.
Process Edge, fog, or cloud systems clean and analyse the stream.
Decide Applications detect patterns, predictions, or thresholds.
Act Users or actuators respond with alerts, controls, or automation.
2.3 Layer Architecture of IoT
| Layer |
Role |
Example function |
| Perception |
Collects physical-world data using sensors and actuators. |
Temperature, moisture, sound, or intrusion detection. |
| Network |
Transfers data from perception to middleware using communication technologies. |
Wi-Fi, infrared, 3G, 4G, UTMS-style links. |
| Middleware |
Stores, processes, computes, and selects relevant information. |
Device lookup by name/address and decision logic. |
| Application |
Controls end-user operations and services. |
Alarms, smart agriculture, security systems, wearable apps. |
| Business |
Analyses service performance and presents value to users or managers. |
Graphs, flowcharts, optimization, and business decisions. |
2.4 IoT Architectures and Building Blocks
The guide describes smart devices as objects with sensors and actuators that collect data and initiate physical action. IoT is enabled by tagging, sensing, thinking, and shrinking.
- RFID tagging: identifies and tracks objects.
- Sensing: gathers contextual or environmental data.
- Thinking/smart technology: supports automation and intelligent decisions.
- Nanotechnology/shrinking: makes components smaller for smoother integration.
System building blocks
Gateways connect devices to cloud systems, filter data, support protocol compatibility, and transmit commands. IoT data may enter a data lake in raw form, move into a warehouse for structured analysis, and feed machine-learning models or control applications.
2.5 Evaluating IoT Security Challenges
| Area |
Challenge |
Resolution techniques |
| Encryption |
Resource-limited devices, complex end-to-end encryption, difficult key management. |
Low-power encryption such as ECC, PKI key management, transport-level encryption such as TLS. |
| Authentication |
Default credentials, huge device scale, device impersonation, limited support for advanced methods. |
Remove defaults, strong passwords, MFA where possible, secure boot, certificate-based authentication. |
| Privacy |
Sensitive behavioral, personal, and environmental data; compliance across regions. |
Privacy-by-design, transparency, opt-out controls, anonymisation, encryption, regular compliance checks. |
2.6 IoT Applications
Smart homes
Thermostats, lighting, cameras, and locks improve energy use, convenience, and security.
Healthcare / IoMT
Wearables, smart pills, inhalers, and beds support remote monitoring and personalized care.
Industrial IoT
Sensors and connected machines enable predictive maintenance, smart factories, and supply-chain optimization.
Agriculture
Soil sensors, livestock monitoring, and autonomous drones improve yield and resource management.
Smart cities
Traffic systems, waste management, and street lighting improve city services and energy use.
Transport and logistics
Fleet management, connected cars, and cargo sensors improve safety and efficiency.
Retail
Smart shelves, beacons, and automated checkout improve inventory and customer experience.
Energy and environment
Smart grids, meters, pollution sensors, water quality systems, and wildlife tracking improve sustainability.
2.2.1 The IoT data-to-action loop
The guide describes an IoT gateway as a central link between devices and the wider system. It can receive sensor data, filter or preprocess it locally, translate between protocols, send useful data to the cloud, and return commands to actuators.
DeviceSensor observes temperature, moisture, motion, health, or machine condition.
GatewayCollects messages, filters noise, and provides protocol compatibility.
PlatformStores raw data in a lake and structured data in a warehouse.
AnalyticsFinds patterns, predicts a condition, or checks a threshold.
ControlApplication sends an alert or an actuator command back through the gateway.
Worked example: dry-soil readings are filtered at the gateway; the platform compares them with weather data; the control application starts irrigation; the user app reports the action. This is the same sense–connect–process–decide–act logic used throughout the guide.
2.4.1 Building blocks: from data lake to user application
| Block | What it does | Revision example |
|---|
| Gateway | Preprocesses and filters data, translates protocols, and transmits commands. | Combines a low-power field sensor with a cloud service. |
| Data lake | Stores large raw streams in their unprocessed form. | Retains historical sensor readings for later modelling. |
| Data warehouse | Stores selected, cleaned, organized data for efficient analysis. | Provides tables used to identify trends and patterns. |
| Machine learning | Uses past warehouse data to improve predictions and control algorithms. | Predicts a machine pre-failure state or future irrigation need. |
| User application | Connects people to devices through web or mobile interfaces. | Shows readings, changes settings, and initiates actions. |
Architecture distinction: the gateway is not merely storage. Its local filtering, protocol compatibility, and command transmission roles reduce cloud traffic and make field devices work together.
2.5.1 Security scenario: securing a connected health monitor
A connected health monitor has limited memory and battery power, sends sensitive readings, and must be accessible only to authorized people and services. Apply the guide’s three security areas as follows:
- Encryption: use an efficient method such as ECC on the device, protect transport with TLS, and manage keys through PKI.
- Authentication: remove default credentials, use strong passwords and MFA where practical, authenticate devices with certificates, and use secure boot so only verified firmware runs.
- Privacy: collect only what is needed, explain collection and use, apply anonymisation and encryption, and review privacy-law compliance across deployment regions.
Failure chain: a default password can permit unauthorized access; weak encryption can expose readings; poor privacy controls can cause breaches, loss of trust, regulatory penalties, and reputational harm.
2.6.1 Application map
| Domain | Devices or data | Outcome described in the guide |
|---|
| Smart homes | Thermostats, lights, cameras, locks. | Automated tasks, comfort, energy savings, and security. |
| IoMT healthcare | Wearables, glucose monitors, inhalers, smart pills, beds. | Continuous monitoring, medication adherence, fewer visits, personalized care. |
| IIoT | Machines, factory sensors, tracked goods. | Predictive maintenance, safety, productivity, and supply-chain optimization. |
| Smart farming | Soil sensors, livestock tags, drones. | Better irrigation/fertilization, animal wellbeing, yield, and less waste. |
| Smart cities | Traffic sensors, bins, streetlights. | Improved traffic flow, waste routes, energy use, and public safety. |
| Transport and logistics | Fleet trackers, connected cars, cargo sensors. | Route planning, vehicle diagnostics, safety, and condition monitoring. |
| Retail | Smart shelves, beacons, automated checkout. | Stock visibility, personalized offers, and faster shopping. |
| Energy/environment/wearables | Smart meters, pollution and water sensors, wildlife trackers, activity devices. | Sustainability, public health, conservation, and personal health insight. |
Emerging applications in the source include smart classrooms, connected financial services such as ATMs and transaction monitoring, and connected entertainment such as smart TVs and gaming consoles.
Chapter 2 Glossary: IoT architecture and application terms
IoT- A network of physical objects, sensors, software, and technologies that communicate and share information.
Sensor- A device that captures a condition such as temperature, moisture, motion, sound, or health data.
Actuator- A component that performs a physical action, such as opening a valve, moving an arm, or changing a motor speed.
Gateway- A link between field devices and cloud/platform systems that can filter data, translate protocols, and transmit commands.
Perception layer- The layer that senses the physical environment and sends collected information onward.
Network layer- The communication layer that transfers data between perception and middleware.
Middleware layer- The layer for storage, computation, processing, device lookup, and decision logic.
Application layer- The service layer that controls operations such as alarms, smart agriculture, and wearable applications.
Data lake- A repository for large raw streams in their original or unprocessed form.
Data warehouse- A structured store of selected and organized data optimized for analysis.
IoMT- Internet of Medical Things; connected healthcare devices and systems.
IIoT- Industrial Internet of Things; connected industrial machines, sensors, and systems.
ECC- Elliptic-curve cryptography; an encryption approach the guide identifies as suitable for resource-limited devices.
PKI- Public Key Infrastructure for managing certificates, identities, and encryption keys.
Secure boot- A startup control that allows a device to execute only verified firmware.
Back to Exam Focus Areas
Chapter 3: IoT Technologies and Standards
This chapter focuses on the technologies, protocols, and standards that allow IoT devices to sense, connect, process, secure, store, and exchange data.
3.1 IoT Technologies: Overview
| Technology area |
Purpose |
Examples from the guide |
| Sensing and actuation |
Capture environmental data and perform physical action. |
Proximity sensors, environmental sensors, valves, robotic arms. |
| Connectivity |
Allow devices to communicate across short-range, long-range, and high-speed links. |
Bluetooth, Zigbee, LoRaWAN, NB-IoT, 5G. |
| Edge, fog, cloud |
Process data close to devices, between edge and cloud, or centrally in the cloud. |
Low-latency edge decisions, fog distribution, cloud analytics. |
| Storage and analytics |
Store raw or structured IoT data and extract insight. |
Data lakes, data warehouses, Hadoop, Spark, AI, machine learning. |
| Identification |
Track or identify objects securely. |
RFID and NFC. |
| Security and integration |
Secure communications and connect platforms/devices. |
TLS, DTLS, authentication, blockchain, middleware platforms, API gateways. |
| Power and nanotechnology |
Extend device life and enable miniaturized sensors. |
Energy harvesting, low-power chips, nanosensors. |
3.2 IoT Communication Protocols
| Protocol |
Type |
Best use |
Key point |
| MQTT |
Application |
Lightweight sensor messaging with publish/subscribe topics. |
QoS levels range from no guarantee to exactly-once delivery. |
| HTTP |
Application |
Web-style request/response communication. |
Often too heavy for battery-powered IoT. |
| CoAP |
Application |
Constrained networks and low-power devices. |
REST-like GET, POST, PUT, DELETE over lower overhead transport. |
| DDS |
Application |
High-demand, real-time systems without brokers. |
Devices share a Global Data Space. |
| AMQP |
Application |
Reliable messaging with delivery guarantees. |
More overhead than MQTT. |
| OPC UA |
Application / industrial |
Industrial interoperability. |
Transport-agnostic and vendor-neutral. |
| 6LoWPAN |
Network |
IPv6 for low-power wireless personal-area networks. |
Useful with smart meters and environmental sensors. |
| BLE |
Network |
Short-range low-power wearables and medical monitors. |
Best for periodic communication. |
| LoRaWAN / NB-IoT |
Network |
Long-range low-power systems. |
Useful in agriculture, utilities, smart cities, and remote monitoring. |
| Zigbee / Wi-Fi / 5G |
Network |
Mesh smart-home networks, high-speed local networks, and ultra-low latency applications. |
Match choice to range, speed, latency, and power. |
3.3 IoT Standards
Standards ensure interoperability, security, privacy, and reliable communication across varied devices and vendors.
- Connectivity standards: IPv6, 6LoWPAN, IEEE 802.15.4, LoRaWAN, and NB-IoT.
- Application-level standards: MQTT, CoAP, OneM2M, and OASIS-style integration frameworks.
- Security standards: TLS/SSL, DTLS, AES, IEEE 802.1X, PKI, OAuth 2.0, IPsec, ETSI EN 303 645, IoTSF guidance.
- Privacy and data protection: GDPR and ISO/IEC 27001.
- Industrial IoT: OPC UA and ISA/IEC 62443 for secure industrial automation and control systems.
3.4 Data Standards in IoT
Data formats
JSON is lightweight and readable, XML is older and more verbose, and Protocol Buffers are compact for low-bandwidth serialization.
Security
TLS/SSL secures transmission, X.509 certificates authenticate devices, and OAuth2/OpenID Connect manage access.
Storage
SQL and NoSQL databases store structured and unstructured data; time-series databases handle timestamped IoT streams.
Interoperability
OneM2M, ODF, and AllJoyn help devices and applications work together across manufacturers.
Processing
Edge computing reduces latency, while Hadoop and Spark process high-volume IoT datasets.
Organizations
ISO/IEC, IEEE, IETF, and GSMA contribute standards for architecture, networking, protocols, and mobile IoT.
3.5 Practical Application of IoT Technologies and Standards
Practical IoT design combines technologies and standards to fit a domain. For example, a smart agriculture system might use soil sensors, LoRaWAN, edge computing, CoAP, OneM2M, and TLS. A hospital wearable may use BLE, MQTT, IEEE medical-device communication standards, and encrypted transmission.
Design-answer pattern: identify the environment, choose sensors, choose connectivity, decide edge/fog/cloud processing, choose protocol/standards, then explain security.
3.1.1 Technology choices by constraint
Technology selection is a constraint-matching exercise. The source guide repeatedly connects a choice to range, power, data rate, latency, processing location, and device size.
| Constraint | Guide-aligned choice | Reason |
|---|
| Very small, local exchange | Bluetooth / BLE or NFC | Short-range communication for wearables, medical monitors, contactless payment, or access control. |
| Low-power home mesh | Zigbee | Low-power, low-data-rate networking for smart-home devices. |
| Remote low-data sensing | LoRaWAN or NB-IoT | Long-range, low-power communication for agriculture, utilities, and remote monitoring. |
| High speed or very low latency | 5G | Supports real-time applications such as industrial automation and connected vehicles. |
| Immediate local response | Edge computing | Processes data at or near the device instead of waiting for a central cloud. |
| Large-scale analysis | Cloud, Hadoop, Spark | Provides scalable storage, distributed processing, AI, and machine learning. |
Remember the three computing positions: edge is closest to the device and fastest for local action; fog distributes processing between edge and cloud; cloud centralizes large-scale storage and analytics.
3.2.1 Protocol comparison and message delivery
| Protocol | Communication model | Important exam point |
|---|
MQTT | Publish/subscribe through a broker. | QoS 0 gives no delivery guarantee; QoS 1 gives delivery but may duplicate; QoS 2 aims for delivery without duplicates. |
CoAP | Lightweight client/server interaction. | Uses REST-like GET, POST, PUT, and DELETE for constrained networks. |
DDS | Brokerless publish/subscribe in a Global Data Space. | Good for dynamic, high-demand real-time environments. |
HTTP / WebSocket | Web request/response or persistent full-duplex TCP. | Useful for web integration, but overhead can be high for constrained devices. |
AMQP | Reliable publish/subscribe and request/response. | Supports at-most-once, at-least-once, and exactly-once delivery, with more overhead than MQTT. |
OPC UA | Industrial request/response or publish/subscribe. | Transport-agnostic and designed for vendor/platform interoperability. |
Exam trap: MQTT and DDS are both publish/subscribe, but MQTT normally uses a broker while DDS connects devices through a Global Data Space without a broker.
3.3.1 Standards map: what each family solves
| Standards family | Examples | Problem addressed |
|---|
| Connectivity/network | IPv6, 6LoWPAN, IEEE 802.15.4, LoRaWAN, NB-IoT. | Device addressing and communication over constrained or wide-area networks. |
| Application | MQTT, CoAP, OneM2M. | Efficient messaging, resource interaction, and cross-domain service interoperability. |
| Security | TLS/SSL, DTLS, AES, IEEE 802.1X, PKI, OAuth 2.0, IPsec. | Confidentiality, integrity, authentication, network access, and delegated authorization. |
| Consumer and lifecycle security | ETSI EN 303 645 and IoTSF guidance. | Baseline requirements, secure updates, authentication, communication, and device lifecycle practice. |
| Privacy and management | GDPR and ISO/IEC 27001. | Responsible personal-data handling and an information-security management framework. |
| Industrial | OPC UA and ISA/IEC 62443. | Secure, interoperable industrial automation and control systems. |
The organizations behind the standards are also examinable: ISO/IEC develops international standards such as IoT architecture; IEEE works on networking and data communication; IETF develops Internet protocols such as IPv6; and GSMA focuses on mobile IoT, including NB-IoT and 5G.
3.4.1 Data standards and interoperability walkthrough
FormatJSON is readable; XML is more verbose; Protobuf is compact for serialization.
TransmitMQTT, CoAP, HTTP/HTTPS, Zigbee, Bluetooth, or LoRaWAN carry data.
StoreSQL/NoSQL systems and time-series databases keep timestamped readings.
InteroperateOneM2M, ODF, and AllJoyn help devices from different manufacturers work together.
AnalyseEdge processing, Hadoop, Spark, AI, and ML convert streams into decisions.
Why standards matter: without common formats, interfaces, security rules, and interoperability practices, devices become isolated ecosystems and data cannot move reliably between manufacturers or platforms.
3.5.1 Case study: precision farming design
A source-grounded precision-farming answer can be built as a chain:
- Use soil-moisture sensors, weather stations, and drones to collect field data.
- Use
LoRaWAN for long-range, low-power field communication and CoAP for constrained device interaction.
- Process urgent readings at the edge so irrigation decisions do not wait for the cloud.
- Use
OneM2M for interoperability and store historical data for analysis.
- Use
TLS to protect transmission; the irrigation actuator changes water delivery when the soil reading and weather context justify it.
The same pattern transfers to other case studies: IIoT uses sensors, edge, LoRaWAN/NB-IoT, OPC UA, and X.509 certificates; smart healthcare uses wearables, BLE, Wi-Fi/cellular, IEEE 11073, and HL7; smart cities use traffic/parking/waste sensors, 5G, ISO 37120, OneM2M, and TLS/SSL; logistics uses RFID/GPS, condition sensors, LoRaWAN/cellular/Wi-Fi, GS1/GTIN, and ISO/IEC 18000.
Chapter 3 Glossary: IoT technologies, protocols, and standards
Edge computing- Processing at or near the device or gateway to reduce latency and cloud traffic.
Fog computing- Processing distributed between edge devices and the cloud.
Cloud computing- Centralized, scalable storage and processing for large IoT datasets and analytics.
RFID- Radio Frequency Identification for automatic object tracking and identification.
NFC- Near Field Communication for short-range transfer such as contactless payments and access control.
MQTT- Lightweight publish/subscribe application protocol using a broker and topic subscriptions.
QoS- Quality of Service; MQTT delivery levels 0, 1, and 2 express increasing delivery assurance.
CoAP- Constrained Application Protocol with low overhead and REST-like resource methods.
DDS- Data Distribution Service; brokerless publish/subscribe for high-demand real-time systems.
6LoWPAN- IPv6 adaptation for Low-Power Wireless Personal Area Networks.
LoRaWAN- Long-range, low-power wide-area communication for low-data-rate remote sensing.
NB-IoT- Narrowband IoT; low-power wide-area connectivity for reliable low-data-rate transmission.
JSON- Lightweight, readable data format often used for IoT exchange.
Protocol Buffers- Compact binary serialization format useful when bandwidth is limited.
Interoperability- The ability of devices and systems from different manufacturers or platforms to work together.
OPC UA- Industrial standard for secure, vendor-neutral data exchange.
OneM2M- Standard for communication and interoperability across IoT industries and devices.
ISO/IEC 27001- Information security management system standard applicable to protecting IoT data.
Back to Exam Focus Areas
Chapter 4: Big Data Storage and Security
This chapter covers storage fundamentals, Hadoop and HDFS, storage architectures, management techniques, and security frameworks for big data environments.
4.1 Fundamentals of Big Data Storage
Big data storage is infrastructure designed to store, manage, retrieve, sort, access, and process large datasets for analytics. It usually scales into terabyte or petabyte ranges and often relies on clusters of commodity servers attached to high-capacity storage.
The guide emphasizes flexibility, scalability, and parallel processing so analytic software can process data from diverse sources.
4.2-4.3 Apache Hadoop and Hadoop Architecture
Apache Hadoop is an open-source framework that distributes storage and processing across many cluster nodes. Its architecture is built around HDFS for storage and MapReduce for processing.
| Component |
Role |
Why it matters |
| HDFS |
Splits files into blocks and stores them across cluster machines. |
Supports large-scale storage, replication, and fault tolerance. |
| NameNode |
Maintains the file-system namespace and block-to-DataNode mapping. |
Coordinates metadata and client access. |
| DataNode |
Stores blocks and serves read/write requests. |
Provides distributed data storage across worker machines. |
| MapReduce |
Splits processing into map and reduce phases. |
Processes large datasets in parallel. |
| YARN |
Manages and schedules cluster resources. |
Optimizes task execution across the cluster. |
4.4 Big Data Storage Solutions
Distributed file systems
HDFS stores file blocks across multiple machines with replication for fault tolerance.
NoSQL databases
MongoDB, Cassandra, and Couchbase handle unstructured or semi-structured data with horizontal scalability.
Cloud storage
Cloud platforms offer on-demand scale, geographic redundancy, built-in security, and lower infrastructure burden.
Data lakes
Store raw structured, semi-structured, and unstructured data until it is needed for analysis.
4.5 Big Data Storage Architectures
Big data storage architectures are designed for enormous scale, variety, and speed. They rely on horizontal scalability, fault tolerance, distributed processing, and high throughput.
| Architecture |
Core idea |
Advantage |
| Distributed file system |
Split data into blocks across nodes. |
Scalable, fault-tolerant batch storage. |
| NoSQL database |
Use document, column-family, key-value, or graph models. |
Flexible, fast, horizontally scalable storage. |
| Data lake |
Store raw data first, structure later. |
Flexible for analytics and machine learning. |
| Cloud-based storage |
Use elastic object storage and cloud analytics integration. |
Pay-as-you-go scale and remote access. |
| Hybrid / multi-cloud |
Combine on-premises and one or more cloud providers. |
Balance sensitivity, cost, performance, and geographic needs. |
4.6 Data Management Techniques
- Data storage solutions: distributed storage and cloud systems handle large scale and redundancy.
- Data integration: ETL or ELT combines structured, semi-structured, and unstructured data from many sources.
- Data governance: policies manage accuracy, access, compliance, and data lifecycle.
- Data quality management: cleansing, validation, and enrichment keep analysis reliable.
- Analytics and processing: machine learning, AI, real-time analytics, Spark, Hadoop, and data mining extract insight.
- Scalability and flexibility: systems must grow as data volume grows.
- Real-time processing: Kafka and Flink support continuous ingestion and event-driven decisions.
4.7 Big Data Security
Big data security protects privacy, prevents cyberattacks and theft, supports regulatory compliance, and preserves data integrity for decision-making.
| Technique / framework |
Use |
| Encryption |
Protect data at rest and in transit. |
| Access control, IAM, RBAC, ABAC, MFA |
Restrict access to authorized users, roles, and attributes. |
| Data masking, tokenization, segmentation, sharding |
Limit exposure of sensitive information and reduce breach impact. |
| Anomaly detection, IDS, SIEM, monitoring, audits |
Detect unusual behavior, vulnerabilities, and policy violations. |
| Backup and recovery |
Restore operations after data loss or incidents. |
| NIST, CSA, ISO/IEC 27001, GDPR |
Provide structured approaches to cybersecurity, cloud security, ISMS, and privacy compliance. |
4.4.1 Storage solution comparison
| Solution | How it stores data | Best revision point |
|---|
| HDFS / distributed file system | Splits large files into blocks across DataNodes and replicates them. | High throughput, scalability, and fault tolerance for large datasets. |
| MongoDB | Document-oriented, JSON-like flexible records. | Good for variable semi-structured data and schema flexibility. |
| Cassandra | Column-family distributed storage. | High availability and fault tolerance across distributed systems. |
| Couchbase | Key-value and document storage. | High-performance, real-time data processing. |
| Cloud object storage | Elastic remote objects such as S3, Google Cloud Storage, or Azure Blob Storage. | On-demand scale, geographic redundancy, access from anywhere, and pay-as-you-go cost. |
| Data lake | Raw structured, semi-structured, and unstructured data in native form. | Structure later for flexible future analytics and machine learning. |
Selection rule: choose based on data form, scale, read/write pattern, latency, fault tolerance, governance, and cost—not simply on the largest storage capacity.
4.5.1 Architecture characteristics and HDFS flow
ClientRequests a file operation through the HDFS namespace.
NameNodeKeeps metadata, namespace, and block-to-DataNode mappings.
BlocksLarge files are divided into blocks, commonly 128 MB or 256 MB in the guide.
DataNodesStore blocks and serve read/write requests; blocks are replicated, typically three times.
RecoveryIf a node fails, another replica remains available; more nodes can be added as data grows.
Big Data architectures are designed around horizontal scalability (add machines), fault tolerance (continue after node failure), distributed processing (work in parallel), and high throughput (move large amounts of data efficiently). Hadoop adds MapReduce for processing, YARN for resource management, and Hadoop Common utilities.
Exam trap: the NameNode stores metadata and coordinates the namespace; it does not carry user data blocks. DataNodes store the blocks and serve client reads/writes.
4.6.1 Data management workflow
- Store: use distributed or cloud systems that split, replicate, and retrieve data in parallel.
- Integrate: combine databases, JSON, sensors, social media, and other sources using
ETL or ELT.
- Govern: define policies for quality, access, security, privacy, lifecycle, and regulations such as GDPR or HIPAA.
- Assure quality: cleanse, validate, and enrich data so decisions are based on accurate information.
- Analyse: apply Spark, Hadoop, machine learning, AI, data mining, or real-time analytics.
- Scale: add servers or use elastic cloud resources as data volume and demand grow.
- Stream: use Kafka and Flink when data must be ingested and processed continuously.
4.7.1 Security controls by risk
| Risk | Control from the guide | What it protects |
|---|
| Data is read in transit or at rest | Encryption and key management. | Confidentiality of stored and moving data. |
| Unauthorized user access | IAM, RBAC/ABAC, MFA, strong authentication. | Limits users to the data and actions they need. |
| Personal data exposed during testing | Masking and tokenization. | Allows analysis without revealing real identifiers. |
| Data altered in a distributed system | Hashing, digital signatures, integrity checks. | Authenticity and integrity across nodes. |
| Misconfiguration or suspicious activity | CSPM, IDS, SIEM, anomaly detection, monitoring, audits. | Finds vulnerabilities, attacks, and policy violations. |
| Loss or breach | Replication, backup, recovery, incident response, patching, staff training. | Availability and reduced damage/downtime. |
For distributed systems, the guide also identifies sharding/data segmentation, blockchain or distributed ledgers for traceability, and homomorphic encryption for selected computations on encrypted data.
Chapter 4 Glossary: storage, management, and security terms
Big data storage- Infrastructure for storing, managing, retrieving, and processing very large datasets, often at TB or PB scale.
Hadoop- Open-source framework that distributes storage and processing over cluster nodes.
HDFS- Hadoop Distributed File System; block-based, replicated distributed storage.
NameNode- HDFS master that maintains the namespace and block metadata.
DataNode- HDFS worker that stores blocks and serves read/write requests.
YARN- Yet Another Resource Negotiator; manages and schedules cluster resources.
Replication- Keeping copies of a data block on multiple nodes to preserve availability after failure.
Horizontal scalability- Growing capacity by adding more machines or nodes rather than only upgrading one machine.
NoSQL- Flexible database family for large unstructured or semi-structured data, including document, column-family, key-value, and graph models.
Data lake- Central repository that keeps raw data in native format until it is needed.
ETL / ELT- Extract–Transform–Load or Extract–Load–Transform data integration approaches.
Data governance- Policies and procedures controlling quality, access, security, compliance, and lifecycle.
IAM- Identity and Access Management for controlling users, services, roles, and resources.
RBAC / ABAC- Role-based or attribute-based access control.
Masking- Replacing sensitive values with safe but structurally similar values for analysis or testing.
Tokenization- Replacing sensitive data with a unique token that does not reveal the original.
NIST Cybersecurity Framework- Security lifecycle of identify, protect, detect, respond, and recover.
Back to Exam Focus Areas
Chapter 5: Strategy Development and Big Data Analytics
This chapter connects Big Data to strategic decision-making, analytics types, visualization, business integration challenges, ethics, and continuous monitoring.
5.1 Role of Big Data in Strategic Decision-Making
Big Data helps organizations make evidence-based decisions by identifying patterns, forecasting trends, optimizing operations, understanding customers, and reducing reliance on intuition alone.
Strategy value: big data is useful when it improves anticipation, efficiency, personalization, and competitive decision-making.
5.2 Key Components of a Big Data Strategy
Collection and integration: gather data from internal, external, sensor, and platform sources.
Storage: choose scalable systems for large-volume retrieval and processing.
Processing: clean, structure, and enrich data with tools such as Hadoop or Spark.
Analytics and insights: use analytics, machine learning, and AI to find patterns.
Governance and security: manage privacy, compliance, access, and risk.
Talent and tools: rely on data scientists, analysts, engineers, cloud platforms, lakes, and visualization tools.
Scalability and flexibility: design for future data growth.
Business alignment: connect data work to organizational goals.
5.3 Data-Driven Strategy Formulation
A data-driven strategy starts with clear goals, gathers relevant data, analyses it for insights, develops a plan, implements it, and then measures and optimizes performance.
- Define measurable objectives.
- Identify and collect relevant data.
- Analyze and interpret the data.
- Build a plan from the insights.
- Implement the strategy and monitor progress.
- Measure, optimize, and repeat.
Retail example: the guide uses customer retention, online sales, segmentation, inventory optimization, delivery improvement, A/B testing, CRM, and analytics platforms to show how data becomes strategy.
5.4 Types of Big Data Analytics
| Type |
Question answered |
Example use |
| Descriptive |
What happened? |
Summarize last year's sales and product performance. |
| Diagnostic |
Why did it happen? |
Find why sales dropped by checking complaints, market changes, or operational issues. |
| Predictive |
What could happen? |
Forecast demand, churn, risk, revenue, or market shifts. |
| Prescriptive |
What should we do? |
Recommend routes, schedules, promotions, pricing, or resource allocation. |
5.5 Data Visualization for Strategy Communication
Visualization turns complex data into charts, graphs, dashboards, and visual stories so stakeholders can understand performance, track KPIs, spot trends, collaborate, and make faster decisions.
- Line graphs can show customer sign-up growth.
- Pie charts can show acquisition sources such as ads, referrals, or organic search.
- Bar charts can compare campaign conversion rates.
- Maps can show customer acquisition by region.
5.6 Challenges of Integrating Big Data with Business Strategy
Data quality
Incomplete, inconsistent, or outdated data can lead to weak decisions.
Data overload
Large volumes can hide relevant insight if filtering and prioritization are poor.
Legacy integration
Existing systems may not support big data scale and complexity.
Talent gaps
Data scientists, analysts, and engineers are needed to convert data into action.
Privacy and security
Sensitive customer data creates compliance and trust risks.
Cultural resistance
Teams may resist evidence-based decision-making if they rely on intuition.
Real-time processing
Fast-moving industries need infrastructure for up-to-date insights.
5.7 Ethical and Legal Considerations
- Privacy and data protection: collect, store, and process personal data responsibly and securely.
- Transparency and accountability: explain what data is collected and how it is used.
- Bias and fairness: prevent algorithms from producing discriminatory decisions.
- Data ownership: clarify who owns data and who may use it.
- Security and breaches: protect sensitive information and respond properly when incidents occur.
- Unintended use: do not repurpose data in ways users were not told about.
5.8 Monitor and Adjust Big Data Strategies
Big Data strategies should be monitored through goals, KPIs, real-time dashboards, processing performance, governance reviews, stakeholder feedback, infrastructure optimization, model evaluation, security checks, communication, and continuous improvement.
Answer pattern: set metrics, monitor systems and outcomes, review compliance and data quality, adjust technology or models, then communicate progress to stakeholders.
5.3.1 Data-driven strategy loop
1. ObjectiveSet a measurable goal tied to the business, such as retention or online sales.
2. DataCollect relevant customer, sales, market, operational, or sentiment data.
3. InsightUse segmentation, prediction, and sentiment analysis to identify patterns.
4. ActionChange marketing, inventory, delivery, product, or resource decisions.
5. FeedbackMeasure results, run tests, learn, and update the strategy.
RetailMax-style worked example: target a 20% retention increase and 15% online-sales increase; combine purchase, browsing, demographic, product, competitor, review, and sentiment data; segment frequent buyers, first-time shoppers, and discount hunters; personalize offers and recommendations; optimize inventory; run A/B tests using click-through and conversion rates; then revise the strategy using the results.
5.4.1 The four analytics questions
| Type | Question | Strategy use | Example |
|---|
| Descriptive | What happened? | Summarize historical performance, trends, and behaviours. | Which products performed best last year? |
| Diagnostic | Why did it happen? | Search for causes, relationships, complaints, market changes, or inefficiencies. | Why did sales decline last quarter? |
| Predictive | What could happen? | Forecast demand, churn, risk, revenue, market shifts, or customer behaviour. | Which product will be needed during the holiday period? |
| Prescriptive | What should we do? | Use optimization, simulation, or decision models to recommend action. | Which price, route, promotion, or inventory allocation is best? |
Predictive analytics tells the strategy team what is likely; prescriptive analytics turns that forecast into a recommended response. For example, predicted demand can lead to a prescription to scale production, adjust supply, or allocate marketing resources.
Exam trap: a forecast is not a recommendation. “Demand will rise” is predictive; “increase stock by the recommended amount and shift promotion budget” is prescriptive.
5.5.1 Choosing visuals for strategy communication
| Visual | Best question | Guide example |
|---|
| Line chart | How is a measure changing over time? | Sales growth or seasonal fluctuations. |
| Bar chart | How do categories compare? | Campaign conversion rates or product performance. |
| Pie chart | How is a whole divided? | Customer acquisition sources such as ads, referrals, or organic search. |
| Gauge / KPI card | Are we near a target? | Customer retention or operational efficiency. |
| Scatter plot | Do two measures relate, and are there outliers? | Marketing ROI and campaign performance. |
| Heat map / map | Where are concentrations or regional differences? | High-sales regions and resource allocation. |
Good visualization makes complex data understandable, tracks KPIs, supports storytelling, speeds decisions, encourages collaboration, reveals trends and outliers, and aligns teams with strategic goals. An interactive dashboard lets stakeholders explore the same evidence and discuss actions.
5.6.1 Integration challenges and responses
| Challenge | Why it harms strategy | Response to discuss |
|---|
| Data quality and accuracy | Incomplete, inconsistent, or outdated data produces poor outcomes. | Clean, validate, enrich, and prioritize trusted sources. |
| Data overload | Relevant signals are hidden in excessive information. | Define objectives, filter for decision relevance, and use dashboards. |
| Legacy integration | Older systems may not handle big-data scale or complexity. | Use staged integration, ETL/ELT, APIs, and scalable platforms. |
| Talent shortage | Without analysts, scientists, engineers, and IT specialists, insight cannot become action. | Build skills, collaborate across teams, and invest in data literacy. |
| Privacy and security | Sensitive data creates breach, compliance, and trust risks. | Governance, access control, encryption, privacy-by-design, and audits. |
| Cultural resistance | Teams may prefer intuition or past practice over evidence. | Train staff, communicate benefits, and foster a data-driven culture. |
| Real-time processing | Fast industries need current data but streaming infrastructure can be costly. | Use cloud, streaming platforms, and monitor throughput and latency. |
5.7.1 Ethics and legal decision checklist
- Privacy and protection: obtain consent where required, secure sensitive data, and respect access/deletion rights.
- Transparency and accountability: explain what is collected, why it is used, and how decisions are made; provide opt-out control where practical.
- Bias and fairness: test whether historical data or model design disadvantages groups in hiring, lending, insurance, or other decisions.
- Ownership: define rights over data created by users, services, or third parties before sharing or selling it.
- Breaches: use encryption, secure storage, audits, incident response, and required notifications.
- Purpose limitation: do not repurpose data for marketing or another use that was not communicated or agreed.
Exam trap: anonymising data does not automatically remove privacy risk; the source notes that advanced analytics can sometimes re-identify people.
5.8.1 Monitoring and continuous improvement plan
| Monitor | Evidence or KPI | Adjustment |
|---|
| Business outcomes | Retention, sales, customer satisfaction, operational efficiency. | Change objectives, actions, or resource allocation. |
| Pipeline performance | Throughput, latency, failures, processing time. | Scale infrastructure or remove bottlenecks. |
| Data quality | Missingness, duplicates, validity, freshness, consistency. | Improve source rules, cleaning, validation, and enrichment. |
| Governance and compliance | Audit results, access reviews, privacy and security checks. | Update policies, permissions, controls, and training. |
| Models | Accuracy, relevance, drift, A/B-test outcomes. | Retrain, recalibrate, compare approaches, or replace a model. |
| Stakeholder adoption | Feedback, reporting cadence, data-literacy participation. | Improve dashboards, communication, collaboration, and skills. |
A complete cycle is: set goals and metrics, use real-time monitoring and alerts, assess performance, review governance, stay flexible, gather feedback, optimize storage/infrastructure, evaluate models, ensure security, communicate progress, and foster continuous improvement.
Chapter 5 Glossary: strategy, analytics, ethics, and monitoring
Data-driven strategy- A strategy formulated and refined using relevant data, analysis, measurable outcomes, and feedback.
KPI- Key Performance Indicator; a measurable value used to track progress toward an objective.
Descriptive analytics- Summarizes historical data to answer “what happened?”
Diagnostic analytics- Investigates causes to answer “why did it happen?”
Predictive analytics- Uses historical data, statistical algorithms, and machine learning to estimate what could happen.
Prescriptive analytics- Uses decision models, optimization, or simulation to recommend what should be done.
Segmentation- Dividing customers or records into groups with similar behaviours or preferences.
A/B testing- Comparing two approaches and using responses such as click-through or conversion rates to refine strategy.
Data governance- Rules and responsibilities for quality, access, privacy, security, compliance, and lifecycle.
Privacy-by-design- Building privacy controls into the system from the start rather than adding them after deployment.
Bias- A systematic distortion that can cause an analytical model or decision process to disadvantage a group.
Purpose limitation- Using collected data only for communicated or authorized purposes.
Throughput- The amount of data a pipeline can process in a given period.
Latency- The delay between data generation or request and processing or response.
Continuous improvement- An iterative cycle of measuring, learning, adjusting, and retraining or redesigning as conditions change.
Back to Exam Focus Areas