Designing a Scalable Popularity Algorithm for Social Apps with Flutter & Firebase
Problem Statement and Requirements
In developing a social challenge app built with Flutter and Firebase, we faced a critical product question while building the "Discover" screen: How do we define "popular"?
Initially, the answer seemed simple: sort by member count. However, we quickly realized a fatal flaw. A circle with 100 members where no one wakes up is a "zombie group," while a new circle with 10 members achieving 100% success is a vibrant community. If we simply sorted by size, we would be promoting dead content while burying high-quality, emerging communities.
To solve this, we needed a ranking system that goes beyond vanity metrics. Our algorithm required a delicate balance of the following factors:
Rewards Engagement Quality: A circle with 10 highly active members must rank higher than a circle with 100 inactive members. Quality > Quantity.
Highlights Current Activity: Users want to join groups that are active right now. Circles with high daily engagement should be discoverable immediately.
Encourages Growth: New circles need a chance to shine. The system shouldn't punish a group just because it's new.
Maintains Fairness: Small but fiery circles should be able to compete with established giants on the leaderboard.
Performs Efficiently: Most importantly, as a mobile app using Firestore, the algorithm must scale to handle thousands of circles without incurring massive read costs or latency.
This document details our journey in designing and implementing a composite scoring algorithm that balances these conflicting metrics to surface the most relevant and engaging circles.
Technology Stack and Architecture
Framework: Flutter (Dart)
Backend: Firebase Firestore
State Management: GetX
Architecture Pattern: Repository Pattern + Controller Pattern
Database Structure
Circle statistics are stored in a Firestore subcollection to keep the main circle document lightweight:
circles/{circleId}/
├── name, description, memberIds, isPrivate, isActive, ...
└── stats/
├── circle/ # Circle-wide statistics
│ ├── activeToday: 15 # Members who verified today
│ ├── activeRate: 75.5 # Today's activity rate (0-100%)
│ ├── recentNewMembers: 3 # New members in last 7 days
│ ├── popularityScore: 68.2 # Cached popularity score (0-100)
│ ├── popularityUpdatedAt: timestamp
│ ├── memberJoinDates: [timestamp, ...] # Array of join dates
│ └── updatedAt: timestamp
└── {userId}/ # Individual member statistics
└── ...
Algorithm Options Considered
Before implementing our final solution, we evaluated several approaches:
Simple Member Count
Approach: Sort by total member count
Pros: Extremely simple to implement, fast query, easy to understand
Cons: Doesn't reflect actual engagement, large but inactive circles rank high, new active circles are buried, encourages quantity over quality
Verdict: Too simplistic. Member count alone doesn't indicate quality or engagement.
Recent Activity Only
Approach: Sort by today's verification count
Pros: Highlights currently active circles, rewards daily engagement, simple to calculate
Cons: Ignores long-term engagement, volatile rankings, doesn't account for circle size, small circles with 100% activity might rank higher than large circles with 80% activity
Verdict: Too narrow. Activity alone doesn't tell the full story.
Time-Decay Algorithm (Reddit-style)
Approach: Exponential decay weighting
Pros: Balances recency with historical data, smooth ranking transitions, well-tested approach
Cons: More complex to implement, requires historical data tracking, decay parameters need tuning, may not suit our use case
Verdict: Over-engineered for our needs. Our challenges are daily, so we don't need complex time decay.
Weekly Aggregation
Approach: Weekly aggregated metrics
Pros: Smooths out daily fluctuations, more stable rankings, reduces computation
Cons: Less responsive to trends, requires weekly batch jobs, stale data for new circles, doesn't highlight "hot" circles right now
Verdict: Too slow to react. We want to highlight trending circles immediately.
Composite Scoring Algorithm (Our Final Choice)
- Approach: Combine multiple normalized metrics with weighted averages. This approach balances engagement quality, scale, growth, and current activity while remaining simple to understand and maintain.
Implementation Strategy
Composite Scoring Formula
Our algorithm combines four key metrics:
Active Rate (40% weight): Percentage of members who verified today
Member Count (30% weight): Total number of members (capped at 100)
Recent Growth (20% weight): Number of new members in last 7 days
Active Today (10% weight): Raw count of members who verified today
Normalization Strategy
Each metric is normalized to a 0-1 range to prevent any single metric from dominating:
Active Rate: Already 0-100%, divide by 100
Member Count: Divide by 100 and clamp (100+ members = 1.0)
Recent Growth: Divide by 10 and clamp (10+ new members = 1.0)
Active Today: Divide by member count and clamp
Caching Strategy
Popularity scores are calculated and cached in Firestore to avoid recalculating on every query. Scores are recalculated when:
A member joins or leaves
Daily statistics are updated
Periodically (e.g., once per day)
Code Implementation
Repository Layer: Popularity Score Calculation
Core Algorithm Implementation
/// Calculate popularity score based on multiple metrics double _calculatePopularityScore({ required double activeRate, // Today's activity rate (0-100%) required int memberCount, // Total members required int recentNewMembers, // New members in last 7 days required int activeToday, // Members who verified today }) { // Weight configuration const double activeRateWeight = 0.4; // 40% - Most important const double memberCountWeight = 0.3; // 30% - Scale matters const double recentGrowthWeight = 0.2; // 20% - Growth momentum const double activeTodayWeight = 0.1; // 10% - Current activity // Normalize each metric to 0-1 range final normalizedActiveRate = activeRate / 100.0; final normalizedMemberCount = (memberCount / 100.0).clamp(0.0, 1.0); final normalizedRecentGrowth = (recentNewMembers / 10.0).clamp(0.0, 1.0); final normalizedActiveToday = memberCount > 0 ? (activeToday / memberCount).clamp(0.0, 1.0) : 0.0; // Weighted average calculation final score = (normalizedActiveRate * activeRateWeight) + (normalizedMemberCount * memberCountWeight) + (normalizedRecentGrowth * recentGrowthWeight) + (normalizedActiveToday * activeTodayWeight); // Convert to 0-100 scale for readability return score * 100.0; }Fetching Popular Circles
/// Get popular circles sorted by popularity score Future<List<CircleModel>> getPopularCircles({int limit = 20}) async { try { // Fetch public, active circles final querySnapshot = await _firestore .collection(_collection) .where('isPrivate', isEqualTo: false) .where('isActive', isEqualTo: true) .limit(100) // Fetch more than needed for sorting .get(); final circles = querySnapshot.docs .map((doc) => CircleModel.fromSnapshot(doc)) .toList(); // Calculate scores for each circle final circlesWithScores = <Map<String, dynamic>>[]; for (final circle in circles) { try { // Fetch statistics from subcollection final statsDoc = await _firestore .collection(_collection) .doc(circle.id) .collection('stats') .doc('circle') .get(); final stats = statsDoc.data(); final activeToday = stats?['activeToday'] as int? ?? 0; final activeRate = (stats?['activeRate'] as num?)?.toDouble() ?? 0.0; final recentNewMembers = stats?['recentNewMembers'] as int? ?? 0; final popularityScore = (stats?['popularityScore'] as num?)?.toDouble(); // Use cached score if available, otherwise calculate double score; if (popularityScore != null && popularityScore > 0) { score = popularityScore; // Use cached score } else { // Calculate on-demand score = _calculatePopularityScore( activeRate: activeRate, memberCount: circle.memberIds.length, recentNewMembers: recentNewMembers, activeToday: activeToday, ); } circlesWithScores.add({ 'circle': circle, 'score': score, }); } catch (e) { print('Error getting stats for circle ${circle.id}: $e'); // Use default score on error final score = _calculatePopularityScore( activeRate: 0.0, memberCount: circle.memberIds.length, recentNewMembers: 0, activeToday: 0, ); circlesWithScores.add({ 'circle': circle, 'score': score, }); } } // Sort by score (descending) circlesWithScores.sort( (a, b) => (b['score'] as double).compareTo(a['score'] as double)); // Return top N circles return circlesWithScores .take(limit) .map((item) => item['circle'] as CircleModel) .toList(); } catch (e) { print('Error fetching popular circles: $e'); return []; } }
Statistics Calculation and Updates
Calculating Daily Statistics
/// Calculate and update all circle statistics Future<bool> calculateAndUpdateCircleStats(String circleId) async { try { final circle = await getCircle(circleId); if (circle == null) return false; final now = DateTime.now(); final startOfDay = DateTime(now.year, now.month, now.day); final endOfDay = startOfDay.add(const Duration(days: 1)); final sevenDaysAgo = now.subtract(const Duration(days: 7)); // Calculate activeToday: Members who verified today final todayVerificationsSnapshot = await _firestore .collection('circles') .doc(circleId) .collection('verifications') .where('isSuccess', isEqualTo: true) .limit(100) .get(); final activeTodaySet = <String>{}; for (var doc in todayVerificationsSnapshot.docs) { final data = doc.data(); final verifiedAt = data['verifiedAt'] as Timestamp?; if (verifiedAt == null) continue; final verifiedDate = verifiedAt.toDate(); final isToday = verifiedDate.isAfter(startOfDay.subtract(const Duration(milliseconds: 1))) && verifiedDate.isBefore(endOfDay); if (isToday) { final userId = data['userId'] as String?; if (userId != null) { activeTodaySet.add(userId); } } } final activeToday = activeTodaySet.length; // Calculate activeRate: (activeToday / totalMembers) * 100 final activeRate = circle.memberIds.isNotEmpty ? (activeToday / circle.memberIds.length) * 100.0 : 0.0; // Calculate recentNewMembers: Count from memberJoinDates array int recentNewMembers = 0; final statsDoc = await _firestore .collection(_collection) .doc(circleId) .collection('stats') .doc('circle') .get(); if (statsDoc.exists) { final statsData = statsDoc.data() as Map<String, dynamic>; final memberJoinDates = statsData['memberJoinDates'] as List<dynamic>?; if (memberJoinDates != null) { for (final date in memberJoinDates) { if (date is Timestamp) { final joinDate = date.toDate(); if (joinDate.isAfter(sevenDaysAgo)) { recentNewMembers++; } } } } } // Calculate popularity score final popularityScore = _calculatePopularityScore( activeRate: activeRate, memberCount: circle.memberIds.length, recentNewMembers: recentNewMembers, activeToday: activeToday, ); // Update all statistics in single write await updateCircleStats( circleId: circleId, activeToday: activeToday, activeRate: activeRate, recentNewMembers: recentNewMembers, popularityScore: popularityScore, ); return true; } catch (e) { print('Error calculating circle stats: $e'); return false; } }Updating Statistics
/// Update circle statistics in subcollection Future<bool> updateCircleStats({ required String circleId, int? activeToday, double? activeRate, int? recentNewMembers, double? popularityScore, List<Timestamp>? memberJoinDates, }) async { try { final statsRef = _firestore .collection(_collection) .doc(circleId) .collection('stats') .doc('circle'); final updateData = <String, dynamic>{ 'updatedAt': Timestamp.fromDate(DateTime.now()), }; if (activeToday != null) { updateData['activeToday'] = activeToday; } if (activeRate != null) { updateData['activeRate'] = activeRate; } if (recentNewMembers != null) { updateData['recentNewMembers'] = recentNewMembers; } if (popularityScore != null) { updateData['popularityScore'] = popularityScore; updateData['popularityUpdatedAt'] = Timestamp.fromDate(DateTime.now()); } if (memberJoinDates != null) { updateData['memberJoinDates'] = memberJoinDates; } // Use merge to create document if it doesn't exist await statsRef.set(updateData, SetOptions(merge: true)); return true; } catch (e) { print('Error updating circle stats: $e'); return false; } }
Controller Layer: Statistics Initialization
Initializing Statistics on Circle Creation
/// Initialize statistics when creating a new circle Future<bool> createCircle({...}) async { try { // ... Create circle document ... // Initialize statistics with creator as first member await _circleRepository.updateCircleStats( circleId: circleId, activeToday: 0, activeRate: 0.0, recentNewMembers: 1, // Creator is the first member popularityScore: 0.0, memberJoinDates: [Timestamp.fromDate(newCircle.createdAt)], // Track creator's join date ); // Initialize individual member stats for creator final challengeRepository = ChallengeRepository(); await challengeRepository.initializeMemberStats(circleId, userId); return true; } catch (e) { print('Error creating circle: $e'); return false; } }Tracking Member Joins
/// Add member to circle and track join date Future<bool> addMemberToCircle(String circleId, String userId) async { try { final now = DateTime.now(); // Add member to circle await _firestore.collection(_collection).doc(circleId).update({ 'memberIds': FieldValue.arrayUnion([userId]), 'updatedAt': Timestamp.fromDate(now), }); // Track join date in statistics final statsRef = _firestore .collection(_collection) .doc(circleId) .collection('stats') .doc('circle'); await statsRef.set({ 'memberJoinDates': FieldValue.arrayUnion([Timestamp.fromDate(now)]), 'updatedAt': Timestamp.fromDate(now), }, SetOptions(merge: true)); return true; } catch (e) { print('Error adding member to circle: $e'); return false; } }
Performance Optimization and Considerations
Score Caching Strategy
Problem: Calculating popularity scores on every query is expensive.
Solution: Cache calculated scores in Firestore and recalculate only when necessary.
Benefits: Reduces computation on read queries, faster response times for popular circles list, lower Firestore read costs.
When to Recalculate: When a member joins or leaves, when daily statistics are updated, periodically (e.g., once per day via Cloud Functions).
Client-Side Sorting
Problem: Firestore doesn't support sorting by computed fields (popularity score).
Solution: Fetch more circles than needed (e.g., 100), calculate scores client-side, then sort and return top N.
Why This Works: Sorting 100 items client-side is fast, more flexible than complex Firestore queries, allows for dynamic score calculation.
Batch Statistics Updates
Problem: Updating statistics individually causes multiple Firestore writes.
Solution: Calculate all metrics first, then update in a single write operation.
Benefits: Reduces Firestore write costs, atomic updates (all or nothing), better performance.
Normalization Thresholds
Rationale for Thresholds:
Member Count (100): Based on analysis showing 95% of circles have < 100 members. Prevents extremely large circles from dominating.
Recent Growth (10): Typical growth is 1-5 new members per week. 10 represents "high growth" threshold.
Active Rate: Already 0-100%, no threshold needed.
Future Improvement: Make thresholds dynamic based on percentiles of actual data.
Challenges and Solutions
New Circles Have No Data
Problem: A newly created circle has no activity data, so it scores 0 and never appears in popular circles.
Solution: Initialize statistics upon circle creation with creator as first member:
await updateCircleStats( circleId: circleId, recentNewMembers: 1, // Creator is the first member memberJoinDates: [Timestamp.fromDate(createdAt)], // Track creator's join date );This ensures new circles have a baseline score and can appear in rankings (albeit low).
Tracking Member Join Dates
Problem: We need to track when each member joined to calculate
recentNewMembers.Solution: Maintain a
memberJoinDatesarray in the stats document usingFieldValue.arrayUnion().Trade-off: This array grows over time. For very old circles, we might need to prune old dates, but for now, the array size is manageable.
Weight Selection
Problem: How do we determine the optimal weights for each metric?
Solution: Start with reasonable defaults based on business logic:
Active Rate (40%): Most important - engagement quality > quantity
Member Count (30%): Important for scale, but not dominant
Recent Growth (20%): Shows momentum and attractiveness
Active Today (10%): Small boost for currently active circles
Future: A/B test different weight combinations and measure impact on join rates and engagement.
Results and Impact
Before Implementation: Popular circles were sorted by member count only, large inactive circles dominated rankings, new active circles were invisible, user engagement in discover screen was low.
After Implementation: Active, engaged circles rank higher, small but active circles are discoverable, growing circles get visibility, more diverse circle recommendations, increased join rates for recommended circles.
Metrics to Monitor
Join Rate: Are users joining recommended circles?
Engagement: Do recommended circles have higher retention?
Diversity: Are we showing a variety of circles?
User Feedback: Do users find recommendations relevant?
Future Improvements
Machine Learning
Replace fixed weights with ML models trained on user behavior:
Which circles do users actually join?
Which circles have high retention?
What patterns predict success?
Personalization
Factor in user preferences:
User's past circle types
User's activity patterns
Similar users' preferences
Time-Based Adjustments
Adjust weights based on time of day or day of week:
Morning: Weight activeRate higher (people are starting challenges)
Evening: Weight memberCount higher (people are browsing)
A/B Testing Framework
Build infrastructure to test different algorithms:
Test different weight combinations
Test different normalization thresholds
Measure impact on key metrics
Conclusion
Designing a popularity algorithm is a balancing act. Too simple, and you miss important signals. Too complex, and you over-engineer. Our composite scoring approach strikes a balance:
Simple enough to understand and maintain
Flexible enough to tune based on data
Effective enough to surface the right circles
Performant enough to scale with thousands of circles
The key is to start with a reasonable algorithm, measure its impact, and iterate based on real user behavior. The weights and thresholds we chose are starting points, not final answers. As our app grows and we gather more data, we'll refine the algorithm to better serve our users.
Remember: The best algorithm is the one that helps users find circles they'll actually engage with.