~/work/aws-log-monitoring
AWS Log Monitoring & Archival System
A small scheduled pipeline that pulls error lines out of an EC2 application log and archives them to S3 every hour, authenticated through an IAM instance role instead of stored keys.
Python · Docker · AWS EC2 · AWS S3 · AWS IAM · cron · GitHub Actions
Problem
Finding error signals in application logs on an EC2 instance usually means someone SSHing in and grepping through a file by hand. That's manual, easy to forget, and the log disappears if the instance is replaced. This project automates that one task: pull out error lines and archive them somewhere durable, on a schedule, without a person doing it.
Architecture
A single Python script reads the local application log, keeps only lines containing ERROR, and uploads them to S3 as a timestamped object so nothing gets overwritten. It runs inside a Docker container for environment parity between a laptop and the EC2 host, and cron triggers it hourly. A GitHub Actions workflow validates that the Docker image still builds on every push.
Architecture diagram with 5 layers. Host: EC2 instance. This connects to Source: app.log (local file). This connects to Filter: monitor.py (keeps lines containing ERROR). This connects to Upload: boto3 put_object() (credentials from IAM instance role). This connects to Archive: S3 bucket (errors_<timestamp>.log). Trigger: cron, hourly. Build check: GitHub Actions, on every push.
Decisions
IAM instance role instead of static credentials
- The EC2 instance carries an IAM role scoped to s3:PutObject on exactly one bucket. boto3 picks up temporary credentials automatically from the instance metadata service.
- There is no AWS access key in the code, the environment, or a credentials file. It would have been easy to hardcode a key on a project this small, but setting up the instance role was the same amount of work.
Deployment
A Docker image built from the repo's Dockerfile, run on the EC2 host and scheduled hourly via a cron entry. GitHub Actions validates the Docker build on every push, independent of the runtime schedule.
Result
Runs the intended loop successfully: filters error lines, uploads a timestamped archive to S3, on schedule, with no credential ever touching the code or the instance's filesystem. It has not been extended past that one loop — one log file, one bucket, substring matching rather than structured log parsing, and no retry or alerting if an individual upload fails.
Lessons
The habit worth keeping: default to an IAM role instead of a static key, even on something this small. It didn't take any extra effort here.