For any content platform, compliance isn't a "bonus feature" — it's the bottom line.
When an ad network rejects your site, an app store delists it, or even a search engine de-ranks it, the root cause is often not a technical failure but non-compliant content — someone posted something they shouldn't have, and your platform didn't stop it. Ad networks (like AdSense) are especially sensitive to content compliance; one piece of low-quality or violating content can trigger a whole-site rejection.
This article breaks down how this project manages "content from entry to disposition" end to end: sensitive-word filtering (blocked at entry), review status (marked pending), review tasks (a daily checklist emailed to the admin), reporting (letting users be sentinels), and abuse control (Redis-Lua scoring + a six-level punishment ladder). You'll find content compliance isn't "one review endpoint" — it's a defense line woven from the moment content walks in the door.
Content compliance's first defense isn't in the review task — it's at the moment content enters. SensitiveUtil is a sensitive-word filter that replaces sensitive words before content is written:
public class SensitiveUtil {
// the sensitive-word file sits in resources
private static final String SENSITIVE_WORD = "sensitive-words/sensitive-words.txt";
private static final String REPLACEMENT = "***"; // the replacement marker
// root node (Trie / DFA)
private static final ... root = new ...();
public void init() {
// at startup, load words from the file and build a tree
String keyword;
while ((keyword = reader.readLine()) != null) {
addWord(keyword.trim());
}
}
public static String filter(String text) {
// scan the text; replace hits with ***
// return the filtered text
}
}
Sensitive-word matching uses a Trie (prefix tree / DFA): at startup, each word in the library is built character-by-character into a tree, and filtering scans the text once, matching all words simultaneously. Why not a simple string contains? Because contains is "scan once per word" — a library of hundreds or thousands of words means hundreds or thousands of scans; a Trie is "scan the text once, walking the tree as you go," O(text length × word length), an order of magnitude faster. Sensitive-word filtering is a high-frequency operation (every piece of content passes through it), so the matching algorithm must be efficient.
The Trie also supports partial matching — say the word is "gambling," and the text has "gamb ling" (with a space inserted) or "gambling site": the Trie scan follows tree paths, and as long as the character sequence can walk down the tree, it's a hit. That's far more flexible than string contains — sensitive words appearing in disguised forms still get recognized. Of course it's not omnipotent; deliberately broken forms like "gam—bling" it can't catch, which is where the review fallback comes in. Sensitive-word filtering blocks "most," not "all" — it keeps the obvious violations at the door, and hands the ones that slip through to human review.
The loading method also matters: the word library lives in a txt file under resources (sensitive-words/sensitive-words.txt), one word per line, read into memory to build the tree at startup. The library is externalized to a file, not hard-coded in code — because the library updates frequently (new words, new rules), an external file lets ops update it without touching code. That's a small but important engineering decision: anyone who hard-codes a sensitive-word library into code is planting a landmine for their future self.
Recall a18's article — it's already being called at upload confirm:
if (spaceId == null) { // sensitive-word filter for public images
picName = SensitiveUtil.filter(picName);
if (picName == null) picName = "Untitled Image";
}
String tags = req.getTagName();
if (spaceId == null && StrUtil.isNotBlank(tags)) {
tags = SensitiveUtil.filter(tags);
}
Note what's filtered and the scope: public space image names and tags are filtered, but content in a user's own space isn't forced through. This trade-off is interesting — public content affects everyone (it appears in the plaza, search, recommendations), so it's mandatorily filtered; private-space content only the user sees, so the platform "reminds but doesn't intercept." The intensity of compliance should match the content's visibility — the more public, the stricter the filter. That's a basic judgment for content platforms.
Sensitive words block the "obviously violating," but compliance is far more than sensitive words. An image could be original, or could infringe; a post could quote something it shouldn't. So beyond sensitive words, content has a review status field — pictures have reviewStatus, posts have status.
Content enters with status 0 (pending review). This "pending" status runs through content's whole life: reviewed and approved before it's public (status 1), reviewed and rejected to take it down (status -1 or similar). Is public content "review-before-publish" or "publish-before-review"? This project is publish-before-review — content is viewable first, then the admin reviews it retroactively. Why? Because "review-before-publish" needs review speed to keep up with content speed, unrealistic for a small team; "publish-before-review" paired with sensitive-word pre-filtering keeps the "obviously violating" at the door and lets human review cover the rest. The choice of review strategy is a trade-off between "efficiency" and "safety" — review-first is safer but slower; publish-first is faster but relies on fallbacks.
The "pending" status itself is also a record running through content's lifecycle. Content enters → pending (0) → approved (1) → if later verified as violating by a report → taken down (-1). This state machine makes "is this content currently compliant" a queryable, flowing state instead of a vague notion. Abstracting compliance into "state" is the cornerstone of content governance — without state, you can't answer "can this content be seen, searched, recommended right now."
And "pending" isn't a static flag; it affects every display position. Beyond the search filter covered in a22, recommendation, plaza, leaderboards — every "public display position" should only show approved content — review status interlocks with all "public display." An unreviewed piece of content shouldn't appear on anyone's homepage; that's the basic principle of content platforms. The review-status check has to be scattered across every place "others can see you."
The "publish-before-review" fallback relies on the review task — but there's a practical problem: an admin can't watch in real time. So review is batch + scheduled + email-reminder:
@Scheduled(cron = "0 0 1 * * ?") // every day at 1 a.m.
public void dailyCheck() {
// take "yesterday's" time range
Date startTime = yesterdayStart();
Date endTime = yesterdayEnd();
// query yesterday's unreviewed public images (reviewStatus=0)
QueryWrapper<Picture> pictureWrapper = new QueryWrapper<Picture>()
.eq("reviewStatus", 0).eq("isDelete", 0).eq("isDraft", 0)
.between("createTime", startTime, endTime)
.and(w -> w.isNull("spaceId").or().eq("spaceId", 0));
// query yesterday's unreviewed posts, friend links
List<Picture> pictures = pictureService.list(pictureWrapper);
List<Post> posts = postService.list(postWrapper);
List<FriendLink> friendLinks = friendLinkService.list(friendLinkWrapper);
// if there's unreviewed content, email the admin
if (!pictures.isEmpty() || !posts.isEmpty() || !friendLinks.isEmpty()) {
emailSenderUtil.sendReviewEmail(adminEmail, generateEmailContent(...));
}
}
Several details worth mentioning. Every day at 1 a.m. — why not real-time? Because review is "human"; pushing real-time to the admin doesn't mean the admin processes it in real time. At 1 a.m., the whole day's pending content gets aggregated into one email, and the admin looks in the morning and processes it in batch. Scheduled batch + email reminder is the most pragmatic engineering form for "human review" — it's not the system being lazy, it's matching the rhythm of human work.
Only querying "yesterday" — the time window avoids duplicate reminders. Yesterday's content is reviewed today; once reviewed, it won't reappear tomorrow. Restricting scope to "public content" — isNull("spaceId").or().eq("spaceId", 0), private-space content is the user's own responsibility; only public content needs platform review. This echoes the "scope matches visibility" principle of sensitive-word filtering.
The email template is a standalone HTML (review_notification.html), listing images, posts and friend links by category. A notification to the admin should let them scan "what to review today" at a glance — a clearly structured checklist email is far more efficient than a pile of scattered reminders.
This task also carries an expectation of "running with nobody watching": if there was no unreviewed content yesterday, no email is sent (if (!pictures.isEmpty() || ...)). Why not send an empty one? Because an empty email is noise — the admin gets used to ignoring "nothing to do" notifications, and misses the real ones when they arrive. Notifications should only fire when there's information; empty notifications corrode trust — the design principle for all reminder-type features.
One more detail: the review scope covers three content types — pictures, posts, friend links — but all limited to "public." Why review friend links too? Because friend links are "inter-site reciprocal links"; a violating friend link drags down the whole site (search engines look at outbound link quality). So every "externally exposed" entry point of the site is in the review scope. The coverage of compliance review must include everything that could affect the site's reputation — not just UGC, even corners like friend links can't be missed.
Machines and scheduled tasks can't catch everything — a piece of content looks compliant, but a user feels offended, plagiarized, harassed. That's where reporting comes in: let users help you find what machines can't see. The frontend ReportModal.vue does exactly that:
<div class="yuemu-report-item">
<div class="yuemu-report-label">Report type <span class="yuemu-required">*</span></div>
<div class="yuemu-report-select-wrap">
<!-- report types: infringement / spam / violating content / harassment ... -->
</div>
</div>
<!-- report target: targetType + targetId -->
The report modal collects three things: the report target (targetType + targetId — reporting an image, post or user), the report type (infringement, spam, violation, harassment, etc.), and the report description. These three form a report record, entering the "pending" state (status=0).
"Letting users be sentinels" has its value in: a machine's judgment is limited; a user's eyes are unlimited. Whether content discomforts some group, or plagiarizes an unregistered work — machines struggle to judge, but users can. Reporting turns "compliance detection" from "platform-only" into "platform + user, both ways" — an inevitable choice for content platforms, and the essence of its difference from pure machine review. A content platform must give users an entrance to say "there's something wrong with this" — that isn't a feature, it's the bottom line.
The "target" design is also worth noting: targetType + targetId — reporting a generic object (image/post/user) rather than writing a separate reporting flow per content type. This abstraction makes "reporting" a cross-cutting capability: any content with a targetId can hang a report button, without a custom implementation per type. Abstracting "reporting" as "an operation on any object" is a win for reusability — one implementation, usable everywhere.
The report types also need careful design: infringement, spam, violating content, harassment... these aren't arbitrary; they map to "the most common violation categories on content platforms," and they let the admin categorize quickly when receiving reports. Report types are "preprocessing for the admin" — users pick a type, the admin triages by type, far more efficient than the admin judging each report from scratch. Every field on a report ticket serves "how the admin handles it."
Reports can't pile up — an infringing post hanging for two days keeps causing harm. So reports are processed faster than content review:
@Scheduled(cron = "0 0 */2 * * ?") // every two hours
public void checkUnprocessedReports() {
// query all pending reports (status=0)
QueryWrapper<Report> reportWrapper = new QueryWrapper<Report>()
.eq("status", 0).eq("isDelete", 0);
List<Report> unprocessedReports = reportService.list(reportWrapper);
if (!unprocessedReports.isEmpty()) {
log.info("Found {} unprocessed reports", unprocessedReports.size());
// email the admin
emailSenderUtil.sendReviewEmail(adminEmail, generateReportEmailContent(unprocessedReports));
}
}
Notice the cadence difference: content review runs once a day (aggregating yesterday at 1 a.m.), report review runs every two hours. Why are reports more frequent? Because reports have "timeliness" — reporting "this content has a problem," the admin waiting a day means the problematic content stays exposed a day longer; while content review is "retroactive approval of new content," being a day late has little impact. A scheduled task's frequency should hug the business's risk level — the higher the risk of the action, the faster the cadence. That's the first principle of scheduled-task design.
Reports also carry a state flow: pending (0) → in progress → handled. When the admin finishes, the status is marked, and unprocessed reports won't re-remind. A state machine is the core of "to-do" type features — without state, you can't distinguish "not processed" from "processed," and reminders would bombard repeatedly.
The disposition result also matters: if a report is verified, content is taken down and the user is dealt with; if not verified, the report itself is rejected. A detail here — reporting isn't "reported equals guilty"; it needs a verification flow. A user report is a "suspected signal," and the admin verifies before knowing if it's real. This "report → verify → dispose/reject" flow prevents "reporting itself from being abused" — someone might use reports to retaliate, to bring down a competitor's content. The reporting mechanism must guard against "reporting being abused" — so the verification step is indispensable, which is why reporting isn't "delete content with one click," but "enters the pending queue."
The rate limiting that's appeared repeatedly in earlier articles (@BucketRateLimit) only "blocks for a moment." This project links rate-limiting to an abuse control system — when your request triggers a rate limit, that itself gets recorded as a "violation," and accumulated enough, you get degraded or banned. The trigger lives in the rate-limit aspect:
if (bucket.tryConsume(1)) {
return pjp.proceed(); // normal pass-through
}
// reaching here means the rate limit fired — that itself records a "mark"
log.warn("[bucket4j] rate limit triggered key={}", key);
if (abuseProperties.isEnabled()) {
// logged-in users by User dimension, anonymous by IP
String dimension = StpUtil.isLogin() ? "user" : "ip";
String id = StpUtil.isLogin() ? StpUtil.getLoginIdAsString() : ServletUtils.getClientIP(request);
// endpoints can have different weights (sensitive endpoints weigh more)
double weight = abuseProperties.getEndpointWeights()
.getOrDefault(limit.key(), abuseProperties.getDefaultWeight());
// record this violation, get the current total score
ScoreSnapshot snapshot = abuseScoreService.recordHit(limit.key(), dimension, id, weight);
// asynchronously check whether to escalate punishment
CompletableFuture.runAsync(() ->
abuseEscalationService.checkAndEscalate(dimension, id, snapshot.getTotal()));
}
throw new BusinessException(ErrorCode.TOO_MANY_REQUEST, limit.message());
The essence of this design: rate-limiting isn't just "rejecting" — it's "keeping a record." A normal person never triggers a rate limit, so never gets a mark; only people who repeatedly cross the threshold get scored and escalated. That's more advanced than plain rate-limiting — rate-limiting says "you're too fast, slow down"; abuse control says "you've done this too many times, I'm watching you."
One layer worth making explicit: this abuse control and rate-limiting are "symbiotic" — without rate-limiting, there's no trigger for "keeping a record"; without abuse control, rate-limiting is just "block for a moment" with no follow-up. Rate-limiting handles "stop you this instant," abuse control handles "remember who you are, be stricter next time." Together they upgrade from "preventing the transient" to "preventing the persistent." Abuse control isn't a replacement for rate-limiting, it's its amplifier — it makes every rate limit not wasted, turning it into a cumulative mark on the violator.
Two details. Dimension choice — logged-in users by User (lock the person, not the device), anonymous by IP. Because anonymous users have no account; IP is the only locatable dimension. Endpoint weight — different endpoints carry different violation weights; sensitive endpoints (posting, uploading) weigh more, ordinary queries less. The "record-keeping" granularity of abuse control follows the endpoint's sensitivity — the more harm an endpoint can do, the heavier the consequence of one cross.
Asynchronous escalation check (CompletableFuture.runAsync) is also worth noting — the escalation judgment (reading Redis, possibly banning) is a slow operation and mustn't block the rate-limited request's return. The request is already being rejected; waiting for an async judgment is pure delay. Separating the core flow (rejecting the request) from peripheral actions (escalation judgment) is the same principle recurring across earlier articles.
And the exception handling: catch (Exception e) { log.error("scoring failed, rate-limit decision unaffected", e); } — a scoring failure doesn't affect the rate-limit decision. Again the "non-core failure degrades silently" principle: whether the rate limit rejects this request shouldn't depend on whether the scoring succeeded. The abuse system is a "nice-to-have" defense layer; its failure can't drag down the main flow (rate-limiting) — that layering is written clearly in the catch.
recordHit is the scoring core of abuse control, using a Redis Lua script to guarantee the atomicity of "record + compute total":
public ScoreSnapshot recordHit(String endpointKey, String dimension, String ipOrUserId, double weight) {
// run the Lua script, atomically record this violation and compute the total across time windows
DefaultRedisScript<String> redisScript = new DefaultRedisScript<>();
redisScript.setScriptText(RECORD_HIT_LUA); // a piece of Lua
redisScript.setResultType(String.class);
String resultJson = stringRedisTemplate.execute(redisScript, ...);
// parse the 1m / 5m / 1h / 24h window scores + total
return ScoreSnapshot.builder()
.m1(json.getInt("1m", 0)).m5(json.getInt("5m", 0))
.h1(json.getInt("1h", 0)).h24(json.getInt("24h", 0))
.total(json.getDouble("total", 0.0))
.build();
}
What the Lua script does, roughly: maintains a counter per "dimension + ID + time window" (abuse:counter:{dim}:{id}:{win}), adds weight for this violation, then computes totals across the 1-minute / 5-minute / 1-hour / 24-hour windows separately, and finally cleans up the old counters that slid out of the 24-hour window. The returned result is a ScoreSnapshot — four windows' scores plus the total.
Why Lua? Because "record once + compute total" must be atomic — if done in two steps (INCR then GET), another request interleaving in between could compute the total wrong. A Lua script is atomic in Redis (the whole script runs as one unit), effectively locking "record + aggregate" into a single atomic operation. In high-concurrency counter scenarios, Lua is the standard answer for "performance and correctness at once."
A comment also mentions a detail: "keep floating-point precision to guard against low-frequency slow crawlers (0.1 weight)." Some crawlers are cunning — not high-frequency hammering but low-frequency slow scanning; a single rate limit isn't triggered (rate limits are frequency-based), but cumulatively they still harm the service. The response: record such requests with a small 0.1 weight — low frequency but persistent, and the total score still climbs. Abuse control must guard against not just "fast" but "slow and persistent" — low-frequency slow attacks rely on accumulation.
The four time windows (1 minute / 5 minutes / 1 hour / 24 hours) are also carefully chosen. Why compute four windows at once? Because they capture different violation patterns: the 1-minute window catches "instant bursts" (sudden hammering), 5 minutes catches "short attacks," 1 hour catches "sustained harassment," 24 hours catches "low-frequency slow accumulation." Multiple parallel windows let abuse control catch both "blitzkrieg" and "war of attrition" — a single window either misses the instant burst or misses the slow accumulation; four together cover everything.
The window cleanup is worth mentioning: old counters in the 24-hour window have to be cleaned, or Redis fills with expired data. The Lua script removes keys that slid out of the window — windowed counters must remember "expiry recycling," otherwise it's a leak-like accumulation. That's also why Redis instead of MySQL — this "high-frequency write + auto-expiry" scenario is naturally Redis's fit.
Scoring isn't the goal; the goal is punishment that's tiered and progressive. checkAndEscalate pushes the "violator" to different levels by total score:
public void checkAndEscalate(String dimension, String id, double totalScore) {
// decide the target level by total score
int targetLevel = 0;
if (totalScore >= thresholds.getPermBan()) targetLevel = 6; // permanent ban
else if (totalScore >= thresholds.getLongBan()) targetLevel = 5; // long-term ban
else if (totalScore >= thresholds.getFreeze()) targetLevel = 4; // account freeze
else if (totalScore >= thresholds.getIpBan()) targetLevel = 3; // IP ban
else if (totalScore >= thresholds.getWarn()) targetLevel = 2; // throttle degradation
else if (totalScore >= thresholds.getNotice()) targetLevel = 1; // warning notice
// User dimension special: 60-119 stays at L2 hardened throttle (0.2), doesn't rise to L3
if ("user".equals(dimension) && targetLevel == 3 && totalScore < thresholds.getFreeze()) {
targetLevel = 2;
}
// IP dimension caps at L3 (can't ban a user account)
int maxLevel = "ip".equals(dimension) ? 3 : 6;
if (targetLevel > maxLevel) targetLevel = maxLevel;
if (targetLevel == 0) return;
// cooldown check: within cooldown and no higher-level violation, skip
// ... write the level and throttle into Redis: abuse:level / abuse:throttle
}
The six-level ladder, each with a clear punishment:
L1 warning notice (TTL 1 hour) — remind first
L2 throttle degradation (throttle=0.5) — request allowance cut in half
L3 IP ban (throttle=0.01) — allowance down to 1%, effectively an IP block
L4 account freeze — account temporarily can't move
L5 long-term ban — account disabled long-term
L6 permanent ban — one-way ticket
The design philosophy of this ladder is worth learning: punishment isn't "one-size-fits-all," it's "progressive + reversible." A first cross is just a warning (L1), again gets throttled (L2), and only repeated offenses escalate to a ban. Why? Because "a first-time violator might be a mistake" — direct banning would wrongly punish; progressive punishment gives "those who erred" room to correct, and gives the system room to "not escalate if they stop." The punishment ladder is essentially "design that leaves room for wrongful hits" — no punishment at all can't control anything, too harsh a punishment wrongs too many, and the ladder in between is the balance.
Every level of the six-step ladder is configurable (AbuseProperties's Thresholds and Durations). That is, "how many points trigger each level" and "how long each punishment lasts" are a set of numbers in a config file, not hard-coded. This lets ops tune the strictness of abuse control without touching code — a new policy tightens (lower thresholds), a holiday relaxes (raise thresholds), both being config edits. A control system's levels and thresholds must be configurable — if hard-coded, every tune means a release, and ops would lose their minds.
Two details are also interesting. User dimension's 60-119 "hardened throttle" — when User reaches L3 (IP ban) but the score hasn't hit the freeze line, it doesn't rise to L3; it stays at L2 but pushes throttle to 0.2 (harsher than regular L2's 0.5). Because L3 is "ban the IP," which is meaningless for a logged-in user (they just change IP and get around it); so for the User dimension, it's "harder throttle" rather than "ban the IP." Punishment should be designed against the punished object's actual characteristics — the IP dimension and the User dimension should punish differently. IP caps at L3 — because IPs are shared (an office has one egress IP), banning an IP at L4 would wrong a whole group of people, so the IP dimension caps at L3.
The "async escalation" also has a subtle detail worth savoring: the escalation judgment (checkAndEscalate) is async (CompletableFuture.runAsync), but it reads the snapshot just returned by recordHit — meaning "this scoring" and "whether to escalate this time" are continuous in logic, just async in execution. "Logically sequential, executionally async" is the right way to asynchronize — you can't sacrifice correctness for asynchrony; you can only be async on the premise of correctness.
Beyond the six-level ladder, two more mechanisms make punishment smarter: recidivism multiplication and cooldown.
Recidivism multiplication — the more times the same dimension (user/IP) has been punished, the longer the punishment grows, exponentially:
// recidivism counter + duration multiplication
String recidivismKey = "abuse:recidivism:" + dimension + ":" + id;
Long recidivismCount = stringRedisTemplate.opsForValue().increment(recidivismKey);
stringRedisTemplate.expire(recidivismKey, 90, TimeUnit.DAYS);
if (recidivismCount != null && recidivismCount > 1) {
double multiplier = Math.pow(abuseProperties.getRecidivism().getMultiplier(), recidivismCount - 1);
finalDuration = (long)(baseDuration * multiplier); // punishment duration grows exponentially
}
The first L2 throttle might last hours; the third identical violation's duration becomes several times longer. This is the engineering of "repeat offenders get punished harder" — punishment isn't fixed, it grows exponentially with recidivism count. It also signals to the system: a dimension repeatedly punished is a "habitual offender," deserving heavier disposition.
Cooldown — escalation checks have a cooldownMinutes (default 30) cooldown: if just punished, and no higher-level violation occurred, no repeated punishment within the cooldown. Why? Because scoring accumulates continuously; after one violation the total keeps climbing, and without cooldown the escalation check would trigger repeatedly, sending warnings and writing Redis over and over. The cooldown keeps "already-punished violations" from being re-disposed in the short term; only "a more serious violation" (escalating to a higher level) breaks the cooldown. Abuse control must also guard against "re-disposition noise" — for the same thing, notify once is enough; don't spam.
The accumulated level and throttle value finally have to land on "this user's every request becomes slower or blocked." getThrottleMultiplier translates punishment into a rate-limit multiplier:
public double getThrottleMultiplier(String key) {
// parse dimension and ID from the rate-limit key: e.g. "user_login:user:123" → dim=user, id=123
String dim = "ip"; String id = "";
if (key.contains(":user:")) { dim = "user"; id = key.substring(key.lastIndexOf(":") + 1); }
else if (key.contains(":ip:")) { dim = "ip"; id = key.substring(key.lastIndexOf(":") + 1); }
if (!id.isEmpty()) {
// read abuse:throttle:{dim}:{id} — the punishment value written during escalation
String throttleKey = "abuse:throttle:" + dim + ":" + id;
String val = stringRedisTemplate.opsForValue().get(throttleKey);
if (val != null) return Double.parseDouble(val);
}
return 1.0; // not punished, multiplier 1.0 (normal allowance)
}
This multiplier returns to the start of BucketRateLimitAspect — remember, in a19's article I mentioned rateLimitService.getThrottleMultiplier(key) returning a multiplier, and Bucket4j's bucket scales its allowance by it. Now the loop closes: abuse escalation writes the throttle value into Redis → the next request's rate-limit bucket allowance gets scaled → the punished person's request allowance shrinks dramatically. throttle=0.5 halves the allowance, throttle=0.01 leaves 1% (effectively banned).
This "rate-limit → keep-a-record → escalate → scale allowance" loop is the most beautiful part of abuse control: it makes punishment "take effect instantly" and "continue automatically." No admin manually banning; the system automatically tightens the violator's allowance step by step based on behavior, until they stop (allowance recovers) or cross into a ban. An automated punishment system is faster, fairer and more thorough than manual disposition — these four points are what let it withstand scripts and malicious users.
I also want to point out that "throttle is a 'gradual punishment,' not a 'binary ban'": L2 cuts in half (0.5), hardened cuts to 20% (0.2), L3 cuts to 1% (0.01) — punishment isn't "usable / unusable" two states, but a continuum tightening tier by tier. Why? Because "total ban" drives users away completely, while "gradual tightening" gives users the expectation that "stop and it recovers" — punishment's reversibility retains users better than its intensity. An L2-throttled user recovers their allowance automatically by stopping; an L6-banned one is truly gone. The "reversibility degree" of punishment is the correction space you leave users.
This loop also has an underrated benefit: it requires almost no manual intervention. The admin doesn't watch violations daily and manually ban; the system scores, escalates and tightens automatically by behavior. The admin only occasionally reads logs and tunes config. The essence of automated abuse control is precipitating "the admin's judgment" into "configurable rules" — the judgment is made by humans, the execution by the system; with that division, it can withstand scale.
On the frontend, the report entry is ReportModal.vue — every piece of content has a "report" button; click it, fill in type and description, submit, and it enters the pending queue. This entry's visibility matters: users need to know "there's a place to report here" to come report when they have a problem. The button's position, copy and discoverability directly determine whether the reporting mechanism works — an entry hidden too deep is as good as no reporting mechanism at all.
An easily-ignored frontend point: consistency of the report entry across all content types. Image detail, post detail, user profile — the report button's position, copy and interaction should be unified — a user who learns to report in one place knows to report everywhere. Interaction consistency is the hidden cost of "making users able to use it" — if every place differs, users relearn every time, and the report rate drops.
The review admin lives in the backend (admin side): the admin receives the daily review email, logs into the backend, and processes the pending images, posts and reports one by one — approve/reject/take down. The review page's core is "fast judgment + batch operation": one screen shows the content, a glance judges compliance, one button handles it, then on to the next. Review is repetitive labor; the review page has to make it least-effort — that's the key to review efficiency.
The review admin also has a data-side consideration: review records (who reviewed, what, what conclusion) should leave traces. That's both audit (when something goes wrong, you can trace to the specific review action) and ops data (you can see which content types get reviewed most, which have low pass rates). Review isn't just "processing content" — it's itself a set of ops data worth recording; from review data, you can see the trend of content quality.
And there's the review status's impact on content lifecycle: content that fails review gets taken down (hidden from the frontend), but it's still in the database — if it's a soft delete, a future re-review can restore it; if hard-deleted, it's truly gone. The "disposition" of review should leave room — take-down is soft (reversible), deletion is hard (irreversible); this trade-off should be thought through in the review flow.
This compliance chain has plenty of potholes. The most memorable:
Pothole 1: The sensitive-word library overreached and wrongly hit normal content. A word in the library was too broad, flooding large amounts of normal content with "***". Later, the library switched to "exact match" instead of "contains match," and the library gets periodic manual review. The granularity of sensitive words directly determines the false-hit rate — a library would rather miss a few than wrongly hit a crowd.
Pothole 2: Review emails fired N times a day. Initially the content review task sent an email every time it found unreviewed content, flooding the admin's inbox. Switching to "only query yesterday's content" — the time window avoiding duplicate reminders — fixed it. A scheduled task's idempotency relies on "only processing new data in the window."
Pothole 3: Abuse scoring wrongly hit normal users. Early on, a normal user occasionally rate-limited (say, one batch operation) got scored and their allowance cut. Later, weights and thresholds were adjusted — ordinary endpoints have lower weights and looser thresholds; only high-frequency crossings truly trigger escalation. Abuse control's false-hit rate directly decides whether it can go live — one wrongful hit and a user is gone.
Pothole 4: Lua's atomicity was misunderstood. At first I thought "score + compute total" in recordHit could be separate (INCR then GET); under concurrency the total came out wrong. Switching to Lua fixed it. A counter's "read-write combine" must be atomic — common sense in Redis high-concurrency scenarios, but you only feel the pain after stepping in it.
Pothole 5: IP-dimension bans wrongly hit shared egress. When one person in an office (shared egress IP) violated, the whole office's IP got banned, everyone suffered. Later, the IP dimension capped at L3, and IP-ban thresholds were raised. Punishing by IP, remember IPs are shared — banning an IP wrongs a crowd, the classic pothole of abuse control.
Pothole 6: The report state flow was missing a state. Initially reports only had "pending/handled"; the admin exiting mid-processing sent the report back to pending, re-reminding. Adding an "in progress" state completed the flow. A to-do state machine, one missing state is one more misjudgment.
Pothole 7: Abuse scoring and rate-limiting coupled too tightly; tuning the limit affects the control. Abuse control depends on rate-limit triggers (only rate-limit hits get scored), so once an endpoint's rate-limit threshold was relaxed, its "scoring trigger rate" dropped, and violators' accumulated score through that endpoint slowed. This made "tune the limit" and "tune the control" interfere with each other, hard to adjust independently. Later, the rate-limit threshold and the abuse weight were considered separately — the limit governs "block or not," the weight governs "how many points to record," decoupling as much as possible. Rate-limiting and abuse control are two dimensions and must be tunable independently — too tight a coupling, and tuning one moves the other, an ops nightmare.
Pothole 8: Sensitive-word filtering only did "replace," not "alert." Initially sensitive-word filtering just replaced words with "***," but the signal "who keeps posting sensitive content" went unused — a user repeatedly posting sensitive words is a repeat offender who should enter abuse control. Later, a sensitive-word hit also records a mark into the abuse system. A sensitive-word hit is itself an "abuse signal" — replacing without recording throws away the most valuable signal.
Collect this compliance chain, and you'll see it's a complete "content security line":
Content enters → sensitive-word filter (Trie, blocked at entry)
→ marked "pending review" (reviewStatus=0)
→ daily task aggregates unreviewed content, emails the admin
→ admin reviews: approve / take down / reject
→ users can report (ReportModal) → report enters pending → emailed every 2 hours
→ abuse control: rate-limit trigger scores (Lua atomic) → six-level escalation → throttle multiplier tightens
Each gate has its division of labor: sensitive words block the "obvious," review covers the "human," reporting fills the "machine's blind spots," abuse control manages the "repeated crossers." Four gates interlocked make a complete "content from entry to disposition" defense line.
Three sentences summarize this article:
Sensitive words block at entry; review covers in batch. A Trie makes sensitive-word filtering efficient; the daily task plus email lets human review keep up with content speed; "publish-before-review" paired with sensitive-word pre-filtering is a pragmatic choice.
Reports let users be sentinels; review runs at different cadences. Problems machines can't see are caught by user reports; reports remind every two hours (high timeliness), content review runs daily (batch fallback) — scheduled-task frequency hugs the risk level.
Abuse control must be progressive, atomic and self-sustaining. Redis Lua atomic scoring, a six-level punishment ladder, recidivism multiplication, cooldown, the throttle-multiplier loop — making punishment "take effect instantly, continue automatically, and leave room for wrongful hits."
And a candid closing note: content compliance has no "done" day — the word library must update, thresholds must be tuned, new content types must be covered. But with the skeleton of these four gates standing, compliance goes from "passively beaten" to "actively defended." Compliance isn't one review; it's a defense line running through content's whole life. I hope this article lets you weave that line into the system when you build a content platform — rather than patching it after the fact.