Data Engineering Test

Your company’s data pipeline refreshes a daily orders analytics dataset using daily files from the source system. Sometimes, after the original daily file has already been processed, a later file arrives containing corrected or updated rows for previous dates.

Currently, the pipeline does not always apply late arriving rows correctly.

What changes should you make to resolve the issue?

As a data engineer at an electronic components factory, you have been tasked with improving the performance of the factory's information database. The database is experiencing slow read operations, and the factory plans to add several new servers to help with the problem.

The database includes several critical tables, including the ones mentioned below along with their related issues:

• Orders: The table is read-heavy and it's where recent orders are accessed most frequently, and has become a bottleneck due to its size.
• QualityControl: The table has numerous columns, many of which are rarely queried causing unnecessary data processing during reads.

What can be done to improve the mentioned issues while taking advantage of new servers?

A platform that schedules and records video game matches has an on-premise database of several terabytes in size. The database contains match results, players, recordings of best plays, and scheduled matches.

Analytic queries are taking too long to execute, so the CTO wants to change the data architecture of the platform and has asked you to provide suggestions.

Considering that:

• Platform users scheduling matches generate a high volume of INSERT operations.
• Analytic reports don't need real time data, and they can tolerate significant latency.
• Videos are generated automatically for each match, and are stored as BLOBs in the database.

Where should data be stored to satisfy the points above while having good performance for user and analytics operations?