Big Data & IoT 600 Study Hub

The guide’s Big Data discussion assumes that useful data arrives in different forms and at different speeds. Structured data fits a defined schema such as transaction rows. Semi-structured data carries tags or flexible fields, such as JSON logs. Unstructured data includes posts, images, audio, video, documents, and other content without a fixed table structure.

Different sources explain why Big Data is both large and varied. A strong answer names the source, the form of data it produces, and the type of insight it can support.

Suppose a retailer receives live checkout events, stores customer documents, and needs a management dashboard. The source guide’s tools fit together as a pipeline rather than as competing answers.

The analysis techniques in the guide answer different questions. They can be combined: descriptive analysis shows a sales drop, diagnostic analysis investigates complaints or operational issues, predictive analytics forecasts demand, and prescriptive analysis can recommend inventory or marketing action.

RetailMax has an Oracle CRM, PostgreSQL e-commerce data, JSON app logs, CSV social-media exports, and MongoDB service tickets. The workflow should be explained in this order:

1. Profile and ingest: record the source, timestamp, owner, format, and expected fields for every input.
2. Clean: flag or impute missing values, remove duplicate customers, correct typos, standardize dates/numbers/categories, and inspect outliers.
3. Normalize: scale numeric features where comparison is needed, encode categories, and use a consistent customer/product key.
4. Transform: parse JSON, flatten nested tickets, aggregate purchases, calculate counts/totals, and engineer features such as frequency and lifetime value.
5. Validate and load: check row counts, types, referential links, and quality rules before loading an analysis-ready customer view.

Useful formulas: min-max = (x - min) / (max - min); z-score = (x - mean) / standard deviation. Use the first to map values to a range such as 0–1 and the second to center values around 0.

The guide describes an IoT gateway as a central link between devices and the wider system. It can receive sensor data, filter or preprocess it locally, translate between protocols, send useful data to the cloud, and return commands to actuators.

A connected health monitor has limited memory and battery power, sends sensitive readings, and must be accessible only to authorized people and services. Apply the guide’s three security areas as follows:

1. Encryption: use an efficient method such as ECC on the device, protect transport with TLS, and manage keys through PKI.
2. Authentication: remove default credentials, use strong passwords and MFA where practical, authenticate devices with certificates, and use secure boot so only verified firmware runs.
3. Privacy: collect only what is needed, explain collection and use, apply anonymisation and encryption, and review privacy-law compliance across deployment regions.

Emerging applications in the source include smart classrooms, connected financial services such as ATMs and transaction monitoring, and connected entertainment such as smart TVs and gaming consoles.

Technology selection is a constraint-matching exercise. The source guide repeatedly connects a choice to range, power, data rate, latency, processing location, and device size.

The organizations behind the standards are also examinable: ISO/IEC develops international standards such as IoT architecture; IEEE works on networking and data communication; IETF develops Internet protocols such as IPv6; and GSMA focuses on mobile IoT, including NB-IoT and 5G.

A source-grounded precision-farming answer can be built as a chain:

1. Use soil-moisture sensors, weather stations, and drones to collect field data.
2. Use LoRaWAN for long-range, low-power field communication and CoAP for constrained device interaction.
3. Process urgent readings at the edge so irrigation decisions do not wait for the cloud.
4. Use OneM2M for interoperability and store historical data for analysis.
5. Use TLS to protect transmission; the irrigation actuator changes water delivery when the soil reading and weather context justify it.

The same pattern transfers to other case studies: IIoT uses sensors, edge, LoRaWAN/NB-IoT, OPC UA, and X.509 certificates; smart healthcare uses wearables, BLE, Wi-Fi/cellular, IEEE 11073, and HL7; smart cities use traffic/parking/waste sensors, 5G, ISO 37120, OneM2M, and TLS/SSL; logistics uses RFID/GPS, condition sensors, LoRaWAN/cellular/Wi-Fi, GS1/GTIN, and ISO/IEC 18000.

Big Data architectures are designed around horizontal scalability (add machines), fault tolerance (continue after node failure), distributed processing (work in parallel), and high throughput (move large amounts of data efficiently). Hadoop adds MapReduce for processing, YARN for resource management, and Hadoop Common utilities.

1. Store: use distributed or cloud systems that split, replicate, and retrieve data in parallel.
2. Integrate: combine databases, JSON, sensors, social media, and other sources using ETL or ELT.
3. Govern: define policies for quality, access, security, privacy, lifecycle, and regulations such as GDPR or HIPAA.
4. Assure quality: cleanse, validate, and enrich data so decisions are based on accurate information.
5. Analyse: apply Spark, Hadoop, machine learning, AI, data mining, or real-time analytics.
6. Scale: add servers or use elastic cloud resources as data volume and demand grow.
7. Stream: use Kafka and Flink when data must be ingested and processed continuously.

For distributed systems, the guide also identifies sharding/data segmentation, blockchain or distributed ledgers for traceability, and homomorphic encryption for selected computations on encrypted data.

Predictive analytics tells the strategy team what is likely; prescriptive analytics turns that forecast into a recommended response. For example, predicted demand can lead to a prescription to scale production, adjust supply, or allocate marketing resources.

Good visualization makes complex data understandable, tracks KPIs, supports storytelling, speeds decisions, encourages collaboration, reveals trends and outliers, and aligns teams with strategic goals. An interactive dashboard lets stakeholders explore the same evidence and discuss actions.

• Privacy and protection: obtain consent where required, secure sensitive data, and respect access/deletion rights.
• Transparency and accountability: explain what is collected, why it is used, and how decisions are made; provide opt-out control where practical.
• Bias and fairness: test whether historical data or model design disadvantages groups in hiring, lending, insurance, or other decisions.
• Ownership: define rights over data created by users, services, or third parties before sharing or selling it.
• Breaches: use encryption, secure storage, audits, incident response, and required notifications.
• Purpose limitation: do not repurpose data for marketing or another use that was not communicated or agreed.

Applied example: before using purchase history to personalize offers, state the purpose to customers, check that use and access are authorized, test whether the recommendations disadvantage a group, and keep a record of who can change the model. If an unrelated partner later requests the same data, review ownership, consent, and purpose before sharing it.

A complete cycle is: set goals and metrics, use real-time monitoring and alerts, assess performance, review governance, stay flexible, gather feedback, optimize storage/infrastructure, evaluate models, ensure security, communicate progress, and foster continuous improvement.

Worked follow-through: if a retention campaign misses its target, compare the KPI with its baseline, check data freshness and pipeline latency, inspect model accuracy by customer segment, review access and consent, then test a revised offer. Record the result on the dashboard, discuss it with stakeholders, and repeat the cycle.