Streaming Log Analytics ( only Open Source Tools to be used ) Reference ( https://d1

Subject:Computer SciencePrice:0 Bought3

Share With

Streaming Log Analytics ( only Open Source Tools to be used )

Reference ( https://d1.awsstatic.com/Projects/P4113850/aws-projects_build-log-analyticssolution-on-aws.pdf )

The assignment solution should achieve the following objectives:

1) Log records are continuously forwarded from the web server ( can use https://github.com/

kiritbasu/Fake-Apache-Log-Generator to generate fake logs ) to the Log Analytics system.

2) System writes each log record to a persistent storage in an efficient manner.

3) Log records are not lost during the transit and made available to the processor whenever required.

4) Continuous monitoring of the streaming input data is happening.

5) An aggregated data is generated every minute and outputs that data through appropriate mechanism.

6) A view of the streaming data using Dashboards/Visualizations is possible.

Submission requirements:

1. Short note on describing (in WORD / PDF document)

2. the streaming data pipeline architecture

3. components, tools used and purposes of the same

4. data flows

5. business logic used along with commands and queries

6. dashboards / visualizations screenshots

Python programs implementing

1. Relevant data flows

2. Data monitoring and processing logic

3. Commands / Queries used with their outputs