Building an automated moderation workflow with AWS Rekognition to reduce risk and protect both users and moderators.
User-generated content is valuable for engagement, but it also introduces moderation risk at scale. In this project, manual review was no longer practical, and the platform needed a safer way to handle explicit media.
The moderation challenge
As upload volume increased, the team was spending too much time manually screening images for harmful categories such as nudity, weapons, and other policy violations. That created two clear issues: inconsistent moderation speed and unnecessary exposure of administrators to harmful content.
There was also a people risk that often gets ignored. Content does not only enter through public feeds. It can come through support forms, enquiry attachments, and other back-office workflows that staff open during a normal working day. Asking someone to open unknown uploads in the middle of an office creates avoidable risk and pressure. We wanted a process that protects staff first, not one that depends on someone seeing harmful material before action is taken.
Solution approach
We implemented a hybrid moderation workflow that combined automated detection with human oversight where needed.
A media lifecycle status was introduced so every image entered a processing state on upload. While in this state, content was not visible to end users. AWS Rekognition then analysed each image and returned category-level signals used for moderation decisions.
If no violation was detected, the image moved to a published state. If a violation signal crossed threshold, the image moved to a blocked state and stayed hidden from public views.
Human review where it matters
Automation handled most of the workload, but moderation remained controllable. We added an admin override layer so teams could correct false positives and manually remove content that should not remain live.
This kept moderation quality high without forcing every image through manual review.
Outcome
The platform gained a safer and more scalable moderation process. Harmful media exposure was reduced, moderation throughput improved, and operational pressure on administrators dropped significantly.
The broader lesson was that moderation automation works best when it is paired with clear lifecycle logic and human controls, not treated as an all-or-nothing replacement.