In developing a social challenge app built with Flutter and Firebase, we faced a critical product question while building the "Discover" screen: How do we define "popular"?

Initially, the answer seemed simple: sort by member count. However, we quickly realized a fatal flaw. A circle with 100 members where no one wakes up is a "zombie group," while a new circle with 10 members achieving 100% success is a vibrant community. If we simply sorted by size, we would be promoting dead content while burying high-quality, emerging communities.

To solve this, we needed a ranking system that goes beyond vanity metrics. Our algorithm required a delicate balance of the following factors:

• Rewards Engagement Quality: A circle with 10 highly active members must rank higher than a circle with 100 inactive members. Quality > Quantity.

• Highlights Current Activity: Users want to join groups that are active right now. Circles with high daily engagement should be discoverable immediately.

• Encourages Growth: New circles need a chance to shine. The system shouldn't punish a group just because it's new.

• Maintains Fairness: Small but fiery circles should be able to compete with established giants on the leaderboard.

• Performs Efficiently: Most importantly, as a mobile app using Firestore, the algorithm must scale to handle thousands of circles without incurring massive read costs or latency.

This document details our journey in designing and implementing a composite scoring algorithm that balances these conflicting metrics to surface the most relevant and engaging circles.

Technology Stack and Architecture

• Framework: Flutter (Dart)

• Backend: Firebase Firestore

• State Management: GetX

• Architecture Pattern: Repository Pattern + Controller Pattern

Database Structure

Circle statistics are stored in a Firestore subcollection to keep the main circle document lightweight:

circles/{circleId}/
  ├── name, description, memberIds, isPrivate, isActive, ...
  └── stats/
      ├── circle/                    # Circle-wide statistics
      │   ├── activeToday: 15        # Members who verified today
      │   ├── activeRate: 75.5       # Today's activity rate (0-100%)
      │   ├── recentNewMembers: 3    # New members in last 7 days
      │   ├── popularityScore: 68.2  # Cached popularity score (0-100)
      │   ├── popularityUpdatedAt: timestamp
      │   ├── memberJoinDates: [timestamp, ...]  # Array of join dates
      │   └── updatedAt: timestamp
      └── {userId}/                  # Individual member statistics
          └── ...

Algorithm Options Considered

Before implementing our final solution, we evaluated several approaches:

1. Simple Member Count

   • Approach: Sort by total member count

   • Pros: Extremely simple to implement, fast query, easy to understand

   • Cons: Doesn't reflect actual engagement, large but inactive circles rank high, new active circles are buried, encourages quantity over quality

   • Verdict: Too simplistic. Member count alone doesn't indicate quality or engagement.

2. Recent Activity Only

   • Approach: Sort by today's verification count

   • Pros: Highlights currently active circles, rewards daily engagement, simple to calculate

   • Cons: Ignores long-term engagement, volatile rankings, doesn't account for circle size, small circles with 100% activity might rank higher than large circles with 80% activity

   • Verdict: Too narrow. Activity alone doesn't tell the full story.

3. Time-Decay Algorithm (Reddit-style)

   • Approach: Exponential decay weighting

   • Pros: Balances recency with historical data, smooth ranking transitions, well-tested approach

   • Cons: More complex to implement, requires historical data tracking, decay parameters need tuning, may not suit our use case

   • Verdict: Over-engineered for our needs. Our challenges are daily, so we don't need complex time decay.

4. Weekly Aggregation

   • Approach: Weekly aggregated metrics

   • Pros: Smooths out daily fluctuations, more stable rankings, reduces computation

   • Cons: Less responsive to trends, requires weekly batch jobs, stale data for new circles, doesn't highlight "hot" circles right now

   • Verdict: Too slow to react. We want to highlight trending circles immediately.

5. Composite Scoring Algorithm (Our Final Choice)

   • Approach: Combine multiple normalized metrics with weighted averages. This approach balances engagement quality, scale, growth, and current activity while remaining simple to understand and maintain.

Implementation Strategy

1. Composite Scoring Formula

   • Our algorithm combines four key metrics:

     • Active Rate (40% weight): Percentage of members who verified today

     • Member Count (30% weight): Total number of members (capped at 100)

     • Recent Growth (20% weight): Number of new members in last 7 days

     • Active Today (10% weight): Raw count of members who verified today

2. Normalization Strategy

   • Each metric is normalized to a 0-1 range to prevent any single metric from dominating:

     • Active Rate: Already 0-100%, divide by 100

     • Member Count: Divide by 100 and clamp (100+ members = 1.0)

     • Recent Growth: Divide by 10 and clamp (10+ new members = 1.0)

     • Active Today: Divide by member count and clamp

3. Caching Strategy

   • Popularity scores are calculated and cached in Firestore to avoid recalculating on every query. Scores are recalculated when:

     • A member joins or leaves

     • Daily statistics are updated

     • Periodically (e.g., once per day)

Code Implementation

1. Repository Layer: Popularity Score Calculation

   • Core Algorithm Implementation

       /// Calculate popularity score based on multiple metrics
       double _calculatePopularityScore({
         required double activeRate,        // Today's activity rate (0-100%)
         required int memberCount,          // Total members
         required int recentNewMembers,     // New members in last 7 days
         required int activeToday,          // Members who verified today
       }) {
         // Weight configuration
         const double activeRateWeight = 0.4;      // 40% - Most important
         const double memberCountWeight = 0.3;     // 30% - Scale matters
         const double recentGrowthWeight = 0.2;    // 20% - Growth momentum
         const double activeTodayWeight = 0.1;     // 10% - Current activity
     
         // Normalize each metric to 0-1 range
         final normalizedActiveRate = activeRate / 100.0;
         final normalizedMemberCount = (memberCount / 100.0).clamp(0.0, 1.0);
         final normalizedRecentGrowth = (recentNewMembers / 10.0).clamp(0.0, 1.0);
         final normalizedActiveToday = memberCount > 0 
           ? (activeToday / memberCount).clamp(0.0, 1.0) 
           : 0.0;
     
         // Weighted average calculation
         final score = (normalizedActiveRate * activeRateWeight) +
                       (normalizedMemberCount * memberCountWeight) +
                       (normalizedRecentGrowth * recentGrowthWeight) +
                       (normalizedActiveToday * activeTodayWeight);
     
         // Convert to 0-100 scale for readability
         return score * 100.0;
       }
     

   • Fetching Popular Circles

       /// Get popular circles sorted by popularity score
       Future<List<CircleModel>> getPopularCircles({int limit = 20}) async {
         try {
           // Fetch public, active circles
           final querySnapshot = await _firestore
               .collection(_collection)
               .where('isPrivate', isEqualTo: false)
               .where('isActive', isEqualTo: true)
               .limit(100)  // Fetch more than needed for sorting
               .get();
     
           final circles = querySnapshot.docs
               .map((doc) => CircleModel.fromSnapshot(doc))
               .toList();
     
           // Calculate scores for each circle
           final circlesWithScores = <Map<String, dynamic>>[];
     
           for (final circle in circles) {
             try {
               // Fetch statistics from subcollection
               final statsDoc = await _firestore
                   .collection(_collection)
                   .doc(circle.id)
                   .collection('stats')
                   .doc('circle')
                   .get();
     
               final stats = statsDoc.data();
               final activeToday = stats?['activeToday'] as int? ?? 0;
               final activeRate = (stats?['activeRate'] as num?)?.toDouble() ?? 0.0;
               final recentNewMembers = stats?['recentNewMembers'] as int? ?? 0;
               final popularityScore = (stats?['popularityScore'] as num?)?.toDouble();
     
               // Use cached score if available, otherwise calculate
               double score;
               if (popularityScore != null && popularityScore > 0) {
                 score = popularityScore;  // Use cached score
               } else {
                 // Calculate on-demand
                 score = _calculatePopularityScore(
                   activeRate: activeRate,
                   memberCount: circle.memberIds.length,
                   recentNewMembers: recentNewMembers,
                   activeToday: activeToday,
                 );
               }
     
               circlesWithScores.add({
                 'circle': circle,
                 'score': score,
               });
             } catch (e) {
               print('Error getting stats for circle ${circle.id}: $e');
               // Use default score on error
               final score = _calculatePopularityScore(
                 activeRate: 0.0,
                 memberCount: circle.memberIds.length,
                 recentNewMembers: 0,
                 activeToday: 0,
               );
               circlesWithScores.add({
                 'circle': circle,
                 'score': score,
               });
             }
           }
     
           // Sort by score (descending)
           circlesWithScores.sort(
               (a, b) => (b['score'] as double).compareTo(a['score'] as double));
     
           // Return top N circles
           return circlesWithScores
               .take(limit)
               .map((item) => item['circle'] as CircleModel)
               .toList();
         } catch (e) {
           print('Error fetching popular circles: $e');
           return [];
         }
       }
     

2. Statistics Calculation and Updates

   • Calculating Daily Statistics

       /// Calculate and update all circle statistics
       Future<bool> calculateAndUpdateCircleStats(String circleId) async {
         try {
           final circle = await getCircle(circleId);
           if (circle == null) return false;
     
           final now = DateTime.now();
           final startOfDay = DateTime(now.year, now.month, now.day);
           final endOfDay = startOfDay.add(const Duration(days: 1));
           final sevenDaysAgo = now.subtract(const Duration(days: 7));
     
           // Calculate activeToday: Members who verified today
           final todayVerificationsSnapshot = await _firestore
               .collection('circles')
               .doc(circleId)
               .collection('verifications')
               .where('isSuccess', isEqualTo: true)
               .limit(100)
               .get();
     
           final activeTodaySet = <String>{};
           for (var doc in todayVerificationsSnapshot.docs) {
             final data = doc.data();
             final verifiedAt = data['verifiedAt'] as Timestamp?;
             if (verifiedAt == null) continue;
     
             final verifiedDate = verifiedAt.toDate();
             final isToday = verifiedDate.isAfter(startOfDay.subtract(const Duration(milliseconds: 1))) &&
                             verifiedDate.isBefore(endOfDay);
     
             if (isToday) {
               final userId = data['userId'] as String?;
               if (userId != null) {
                 activeTodaySet.add(userId);
               }
             }
           }
           final activeToday = activeTodaySet.length;
     
           // Calculate activeRate: (activeToday / totalMembers) * 100
           final activeRate = circle.memberIds.isNotEmpty
               ? (activeToday / circle.memberIds.length) * 100.0
               : 0.0;
     
           // Calculate recentNewMembers: Count from memberJoinDates array
           int recentNewMembers = 0;
           final statsDoc = await _firestore
               .collection(_collection)
               .doc(circleId)
               .collection('stats')
               .doc('circle')
               .get();
     
           if (statsDoc.exists) {
             final statsData = statsDoc.data() as Map<String, dynamic>;
             final memberJoinDates = statsData['memberJoinDates'] as List<dynamic>?;
     
             if (memberJoinDates != null) {
               for (final date in memberJoinDates) {
                 if (date is Timestamp) {
                   final joinDate = date.toDate();
                   if (joinDate.isAfter(sevenDaysAgo)) {
                     recentNewMembers++;
                   }
                 }
               }
             }
           }
     
           // Calculate popularity score
           final popularityScore = _calculatePopularityScore(
             activeRate: activeRate,
             memberCount: circle.memberIds.length,
             recentNewMembers: recentNewMembers,
             activeToday: activeToday,
           );
     
           // Update all statistics in single write
           await updateCircleStats(
             circleId: circleId,
             activeToday: activeToday,
             activeRate: activeRate,
             recentNewMembers: recentNewMembers,
             popularityScore: popularityScore,
           );
     
           return true;
         } catch (e) {
           print('Error calculating circle stats: $e');
           return false;
         }
       }
     

   • Updating Statistics

       /// Update circle statistics in subcollection
       Future<bool> updateCircleStats({
         required String circleId,
         int? activeToday,
         double? activeRate,
         int? recentNewMembers,
         double? popularityScore,
         List<Timestamp>? memberJoinDates,
       }) async {
         try {
           final statsRef = _firestore
               .collection(_collection)
               .doc(circleId)
               .collection('stats')
               .doc('circle');
     
           final updateData = <String, dynamic>{
             'updatedAt': Timestamp.fromDate(DateTime.now()),
           };
     
           if (activeToday != null) {
             updateData['activeToday'] = activeToday;
           }
           if (activeRate != null) {
             updateData['activeRate'] = activeRate;
           }
           if (recentNewMembers != null) {
             updateData['recentNewMembers'] = recentNewMembers;
           }
           if (popularityScore != null) {
             updateData['popularityScore'] = popularityScore;
             updateData['popularityUpdatedAt'] = Timestamp.fromDate(DateTime.now());
           }
           if (memberJoinDates != null) {
             updateData['memberJoinDates'] = memberJoinDates;
           }
     
           // Use merge to create document if it doesn't exist
           await statsRef.set(updateData, SetOptions(merge: true));
           return true;
         } catch (e) {
           print('Error updating circle stats: $e');
           return false;
         }
       }
     

3. Controller Layer: Statistics Initialization

   • Initializing Statistics on Circle Creation

       /// Initialize statistics when creating a new circle
       Future<bool> createCircle({...}) async {
         try {
           // ... Create circle document ...
     
           // Initialize statistics with creator as first member
           await _circleRepository.updateCircleStats(
             circleId: circleId,
             activeToday: 0,
             activeRate: 0.0,
             recentNewMembers: 1,  // Creator is the first member
             popularityScore: 0.0,
             memberJoinDates: [Timestamp.fromDate(newCircle.createdAt)],  // Track creator's join date
           );
     
           // Initialize individual member stats for creator
           final challengeRepository = ChallengeRepository();
           await challengeRepository.initializeMemberStats(circleId, userId);
     
           return true;
         } catch (e) {
           print('Error creating circle: $e');
           return false;
         }
       }
     

   • Tracking Member Joins

       /// Add member to circle and track join date
       Future<bool> addMemberToCircle(String circleId, String userId) async {
         try {
           final now = DateTime.now();
     
           // Add member to circle
           await _firestore.collection(_collection).doc(circleId).update({
             'memberIds': FieldValue.arrayUnion([userId]),
             'updatedAt': Timestamp.fromDate(now),
           });
     
           // Track join date in statistics
           final statsRef = _firestore
               .collection(_collection)
               .doc(circleId)
               .collection('stats')
               .doc('circle');
     
           await statsRef.set({
             'memberJoinDates': FieldValue.arrayUnion([Timestamp.fromDate(now)]),
             'updatedAt': Timestamp.fromDate(now),
           }, SetOptions(merge: true));
     
           return true;
         } catch (e) {
           print('Error adding member to circle: $e');
           return false;
         }
       }
     

Performance Optimization and Considerations

1. Score Caching Strategy

   • Problem: Calculating popularity scores on every query is expensive.

   • Solution: Cache calculated scores in Firestore and recalculate only when necessary.

   • Benefits: Reduces computation on read queries, faster response times for popular circles list, lower Firestore read costs.

   • When to Recalculate: When a member joins or leaves, when daily statistics are updated, periodically (e.g., once per day via Cloud Functions).

2. Client-Side Sorting

   • Problem: Firestore doesn't support sorting by computed fields (popularity score).

   • Solution: Fetch more circles than needed (e.g., 100), calculate scores client-side, then sort and return top N.

   • Why This Works: Sorting 100 items client-side is fast, more flexible than complex Firestore queries, allows for dynamic score calculation.

3. Batch Statistics Updates

   • Problem: Updating statistics individually causes multiple Firestore writes.

   • Solution: Calculate all metrics first, then update in a single write operation.

   • Benefits: Reduces Firestore write costs, atomic updates (all or nothing), better performance.

4. Normalization Thresholds

   • Rationale for Thresholds:

     • Member Count (100): Based on analysis showing 95% of circles have < 100 members. Prevents extremely large circles from dominating.

     • Recent Growth (10): Typical growth is 1-5 new members per week. 10 represents "high growth" threshold.

     • Active Rate: Already 0-100%, no threshold needed.

   • Future Improvement: Make thresholds dynamic based on percentiles of actual data.

Challenges and Solutions

1. New Circles Have No Data

   • Problem: A newly created circle has no activity data, so it scores 0 and never appears in popular circles.

   • Solution: Initialize statistics upon circle creation with creator as first member:

       await updateCircleStats(
         circleId: circleId,
         recentNewMembers: 1,  // Creator is the first member
         memberJoinDates: [Timestamp.fromDate(createdAt)],  // Track creator's join date
       );
     

   • This ensures new circles have a baseline score and can appear in rankings (albeit low).

2. Tracking Member Join Dates

   • Problem: We need to track when each member joined to calculate recentNewMembers.

   • Solution: Maintain a memberJoinDates array in the stats document using FieldValue.arrayUnion().

   • Trade-off: This array grows over time. For very old circles, we might need to prune old dates, but for now, the array size is manageable.

3. Weight Selection

   • Problem: How do we determine the optimal weights for each metric?

   • Solution: Start with reasonable defaults based on business logic:

     • Active Rate (40%): Most important - engagement quality > quantity

     • Member Count (30%): Important for scale, but not dominant

     • Recent Growth (20%): Shows momentum and attractiveness

     • Active Today (10%): Small boost for currently active circles

   • Future: A/B test different weight combinations and measure impact on join rates and engagement.

Results and Impact

• Before Implementation: Popular circles were sorted by member count only, large inactive circles dominated rankings, new active circles were invisible, user engagement in discover screen was low.

• After Implementation: Active, engaged circles rank higher, small but active circles are discoverable, growing circles get visibility, more diverse circle recommendations, increased join rates for recommended circles.

Metrics to Monitor

1. Join Rate: Are users joining recommended circles?

2. Engagement: Do recommended circles have higher retention?

3. Diversity: Are we showing a variety of circles?

4. User Feedback: Do users find recommendations relevant?

Future Improvements

1. Machine Learning

   • Replace fixed weights with ML models trained on user behavior:

     • Which circles do users actually join?

     • Which circles have high retention?

     • What patterns predict success?

2. Personalization

   • Factor in user preferences:

     • User's past circle types

     • User's activity patterns

     • Similar users' preferences

3. Time-Based Adjustments

   • Adjust weights based on time of day or day of week:

     • Morning: Weight activeRate higher (people are starting challenges)

     • Evening: Weight memberCount higher (people are browsing)

4. A/B Testing Framework

   • Build infrastructure to test different algorithms:

     • Test different weight combinations

     • Test different normalization thresholds

     • Measure impact on key metrics

Conclusion

Designing a popularity algorithm is a balancing act. Too simple, and you miss important signals. Too complex, and you over-engineer. Our composite scoring approach strikes a balance:

1. Simple enough to understand and maintain

2. Flexible enough to tune based on data

3. Effective enough to surface the right circles

4. Performant enough to scale with thousands of circles

The key is to start with a reasonable algorithm, measure its impact, and iterate based on real user behavior. The weights and thresholds we chose are starting points, not final answers. As our app grows and we gather more data, we'll refine the algorithm to better serve our users.

Remember: The best algorithm is the one that helps users find circles they'll actually engage with.

While developing a chat application, we faced a dilemma regarding message loading strategy:

• 100 message limit: Fast initial loading, but poor user experience when checking older conversations

• 1000+ messages loaded: Loading all messages at once increases initial loading time, network costs, and Firestore read costs

• Optimal solution: Load only the latest 100 messages initially, and progressively load older messages when users scroll up using pagination

This document provides a detailed explanation of implementing efficient chat message pagination using Firebase Firestore.

Technology Stack and Architecture

• Backend: Firebase Firestore

• State Management: GetX

• Architecture Pattern: Repository Pattern + Controller Pattern

Database Structure

In Firestore, chat messages are stored in the following structure:

circles/{circleId}/chatRooms/{chatRoomId}/messages/{messageId}

Each message document contains the following fields:

• createdAt: Timestamp (message creation time)

• senderId: String (sender ID)

• senderName: String (sender name)

• message: String (message content)

Implementation Strategy

1. Initial Loading: Latest 100 Messages

We use Firestore's orderBy and limit queries to fetch only the latest 100 messages.

2. Pagination: Cursor-Based Pagination

Firestore uses cursor-based pagination rather than offset-based pagination. This is more performant and ensures data consistency in real-time databases.

Principle of Cursor-Based Pagination:

• Query the next page based on the field value (e.g., createdAt) of the last loaded document

• Combine where clause with orderBy to filter messages before a specific point in time

Code Implementation

1. Repository Layer: Firestore Query Optimization

Previous Code (Issues)

Stream<List<ChatMessage>> getChatMessagesStream(String circleId, String chatRoomId) {
  return _getMessagesRef(circleId, chatRoomId)
    .orderBy('createdAt', descending: true)
    .limit(1000) // Loading all messages at once
    .snapshots()
    .map((snapshot) {
      final messages = snapshot.docs
        .map((doc) => ChatMessage.fromSnapshot(doc))
        .toList();
      messages.sort((a, b) => a.createdAt.compareTo(b.createdAt));
      return messages;
    });
}

Issues:

• Initial loading requires reading 1000 documents

• Increased Firestore read costs (1000 reads per load)

• Wasted network bandwidth

• Increased initial loading time

Improved Code: Optimized Initial Loading

Stream<List<ChatMessage>> getChatMessagesStream(String circleId, String chatRoomId) {
  try {
    // Fetch messages from subcollection - get latest messages first, then reverse sort
    return _getMessagesRef(circleId, chatRoomId)
      .orderBy('createdAt', descending: true) // Latest messages first
      .limit(100) // Load only the latest 100 messages
      .snapshots()
      .map((snapshot) {
        final messages = snapshot.docs
          .map((doc) => ChatMessage.fromSnapshot(doc))
          .toList();
        // Reverse sort so latest messages come first (oldest → newest)
        messages.sort((a, b) => a.createdAt.compareTo(b.createdAt));
        return messages;
      });
  } catch (e) {
    print('Error in getChatMessagesStream: $e');
    return Stream.value([]);
  }
}

Improvements:

• limit(100) reduces initial loading costs (100 reads per load)

• Maintains snapshots() stream for real-time updates

• Client-side chronological sorting (for UI display)

New Method: Pagination for Older Messages

/// Load older messages (pagination)
Future<List<ChatMessage>> loadOlderMessages({
  required String circleId,
  required String chatRoomId,
  required DateTime beforeDate,
  int limit = 100,
}) async {
  try {
    // Cursor-based pagination: query only messages before beforeDate
    final query = _getMessagesRef(circleId, chatRoomId)
      .orderBy('createdAt', descending: true)
      .where('createdAt', isLessThan: Timestamp.fromDate(beforeDate))
      .limit(limit);
    final snapshot = await query.get();
    final messages = snapshot.docs
      .map((doc) => ChatMessage.fromSnapshot(doc))
      .toList();
    // Sort in ascending order (oldest first for UI display)
    messages.sort((a, b) => a.createdAt.compareTo(b.createdAt));
    return messages;
  } catch (e) {
    print('Error loading older messages: $e');
    return [];
  }
}

Key Points:

1. Cursor-based pagination: where('createdAt', isLessThan: Timestamp.fromDate(beforeDate))

   • beforeDate is the createdAt value of the oldest message currently loaded

   • Only queries messages before this value to prevent duplicates

2. Query optimization: Using orderBy and where together

   • Firestore may require a composite index

   • Index on createdAt field is automatically created

3. Consistent sorting: Client-side chronological sorting

   • Firestore fetches newest first with descending: true

   • Re-sorted in ascending order for UI display

2. Controller Layer: State Management and Business Logic

Adding State Variables

class ChatController extends GetxController {
  // Existing state variables...
  final RxList<ChatMessage> currentMessages = <ChatMessage>[].obs;
  final RxBool isLoadingMessages = false.obs;

  // New state variables for pagination
  final RxBool isLoadingOlderMessages = false.obs; // Loading state for older messages
  final RxBool hasMoreMessages = true.obs; // Whether there are more messages to load
}

Implementing Pagination Method

/// Load older messages (pagination)
Future<bool> loadOlderMessages(String chatRoomId) async {
  try {
    // Prevent duplicate loads and check conditions
    if (isLoadingOlderMessages.value || !hasMoreMessages.value || currentMessages.isEmpty) {
      return false;
    }
    isLoadingOlderMessages.value = true;
    final circleId = chatRoomId;
    final oldestMessage = currentMessages.first; // Oldest message currently loaded

    // Load older messages through Repository
    final olderMessages = await _chatRepository.loadOlderMessages(
      circleId: circleId,
      chatRoomId: chatRoomId,
      beforeDate: oldestMessage.createdAt, // Cursor: time of oldest message
      limit: 100,
    );

    // No more messages to load
    if (olderMessages.isEmpty) {
      hasMoreMessages.value = false;
      return false;
    }

    // Insert at the beginning of existing message list (already sorted oldest first)
    currentMessages.insertAll(0, olderMessages);

    // If less than 100, this is the last page
    if (olderMessages.length < 100) {
      hasMoreMessages.value = false;
    }
    return true;
  } catch (e, stackTrace) {
    print('Error loading older messages: $e');
    print('Stack trace: $stackTrace');
    return false;
  } finally {
    isLoadingOlderMessages.value = false;
  }
}

Implementation Details:

1. Prevent Duplicate Loads

   • isLoadingOlderMessages flag prevents concurrent requests

   • hasMoreMessages blocks requests when no more messages are available

2. Determine Cursor Value

   • currentMessages.first: Oldest message currently loaded

   • Pass this message's createdAt as beforeDate

3. Merge Messages

   • insertAll(0, olderMessages): Insert at the beginning of existing list

   • Chronological order is maintained

4. Detect Last Page

   • If loaded messages are less than 100, consider it the last page

   • Set hasMoreMessages.value = false to block additional requests

Updating State During Initial Loading

void loadChatMessages(String chatRoomId) {
  try {
    isLoadingMessages.value = true;
    currentMessages.clear();
    final circleId = chatRoomId;

    // Subscribe to real-time stream
    _currentChatMessageStream = _chatRepository.getChatMessagesStream(circleId, chatRoomId).listen(
      (messages) {
        currentMessages.assignAll(messages);
        // If less than 100 messages, no more messages to load
        hasMoreMessages.value = messages.length >= 100;
        isLoadingMessages.value = false;
        _markMessagesAsRead(circleId, chatRoomId);
      },
      onError: (error) {
        print('Error loading messages: $error');
        isLoadingMessages.value = false;
      },
    );
  } catch (e) {
    print('Error loading chat messages: $e');
    isLoadingMessages.value = false;
  }
}

3. UI Layer: Scroll Detection and Auto-Loading

Implementing Scroll Listener

class _ChatRoomScreenState extends State<ChatRoomScreen> {
  final ScrollController _scrollController = ScrollController();
  final ChatController _chatController = Get.find<ChatController>();

  @override
  void initState() {
    super.initState();
    // Add scroll listener
    _scrollController.addListener(_onScroll);
  }

  void _onScroll() {
    if (_scrollController.hasClients) {
      final currentScroll = _scrollController.position.pixels;
      // Load older messages when scroll is near the top
      // Trigger auto-load when within 200px
      if (currentScroll < 200 && !_chatController.isLoadingOlderMessages.value && _chatController.hasMoreMessages.value) {
        _loadOlderMessages();
      }
    }
  }

  Future<void> _loadOlderMessages() async {
    if (_chatRoomId == null) return;

    // Save message count to maintain scroll position
    final previousCount = _chatController.currentMessages.length;

    // Load older messages
    final success = await _chatController.loadOlderMessages(_chatRoomId!);
    if (success && mounted) {
      // Maintain scroll position (new messages are added at the top)
      WidgetsBinding.instance.addPostFrameCallback((_) {
        if (_scrollController.hasClients) {
          final newCount = _chatController.currentMessages.length;
          final addedCount = newCount - previousCount;
          // Adjust scroll position by estimated height of added messages
          // Estimate approximately 60px per message (can be dynamically calculated)
          final estimatedHeight = addedCount * 60.0;
          final newScrollPosition = _scrollController.position.pixels + estimatedHeight;
          if (newScrollPosition <= _scrollController.position.maxScrollExtent) {
            _scrollController.jumpTo(newScrollPosition);
          }
        }
      });
    }
  }
}

Scroll Position Maintenance Logic:

1. Problem: When new messages are added at the top of the list, scroll position changes

2. Solution: Adjust scroll position by the estimated height of added messages

3. Implementation: Use jumpTo() to immediately change scroll position (no animation)

UI: Loading Indicator and Empty State Handling

ListView.builder(
  controller: _scrollController,
  padding: const EdgeInsets.symmetric(horizontal: 16, vertical: 16),
  itemCount: messages.length + 2, // +1 for loading indicator, +1 for spacing
  itemBuilder: (context, index) {
    // Show loading indicator at the top
    if (index == 0) {
      return Obx(() {
        if (_chatController.isLoadingOlderMessages.value) {
          return Container(
            padding: const EdgeInsets.all(16),
            alignment: Alignment.center,
            child: const CircularProgressIndicator(
              valueColor: AlwaysStoppedAnimation<Color>(AppTheme.lime400),
            ),
          );
        }
        if (!_chatController.hasMoreMessages.value) {
          return Container(
            padding: const EdgeInsets.all(16),
            alignment: Alignment.center,
            child: Text(
              'No more messages',
              style: TextStyle(
                fontSize: 12,
                color: AppTheme.textDarkTertiary,
              ),
            ),
          );
        }
        return const SizedBox.shrink();
      });
    }
    // Add empty space at the end (for scrolling)
    if (index == messages.length + 1) {
      return const SizedBox(height: 20);
    }
    // Render actual message
    final messageIndex = index - 1; // -1 because of loading indicator
    final message = messages[messageIndex];
    // ... Render message UI
  },
)

Performance Optimization and Considerations

1. Firestore Read Cost Optimization

Cost Comparison:

• Previous approach: 1000 reads on initial load

• Improved approach: 100 reads on initial load + 100 reads per scroll

• Cost Reduction: 90% cost savings if user doesn't scroll

2. Network Bandwidth Optimization

• Reduced initial loading time: Only 100 messages sent, reducing network latency

• Progressive loading: Load only what's needed, reducing total bandwidth usage

3. Memory Management

• currentMessages maintains only hundreds of messages in memory

• Older messages loaded only when needed, optimizing memory usage

4. Harmony with Real-Time Updates

• Maintain snapshots() stream so new messages are added in real-time

• Paginated older messages and real-time messages merge naturally

Firestore Index Requirements

Cursor-based pagination requires the following index:

Collection: circles/{circleId}/chatRooms/{chatRoomId}/messages
Fields:
- createdAt (Descending)
- createdAt (Ascending) - used with where clause

Firebase Console automatically provides an index creation link, or you can create it manually if needed.

Application to Personal Chat Rooms

We applied the same pattern to personal chat rooms:

// PersonalChatRepository
Future<List<ChatMessage>> loadOlderPersonalMessages({
  required String chatRoomId,
  required DateTime beforeDate,
  int limit = 100,
}) async {
  final query = _getMessagesRef(chatRoomId)
    .orderBy('timestamp', descending: true)
    .where('timestamp', isLessThan: Timestamp.fromDate(beforeDate))
    .limit(limit);
  final snapshot = await query.get();
  // ... Transform and sort messages
}

Conclusion

We optimized chat message loading by leveraging Firebase Firestore's cursor-based pagination. Key achievements:

1. Cost Reduction: 90% reduction in initial loading costs

2. Improved User Experience: Fast initial loading + ability to check older messages when needed

3. Scalability: Efficiently handles tens of thousands of messages

4. Real-Time Synchronization: Pagination and real-time updates work seamlessly together

Through these optimizations, we've achieved stable and efficient message loading even in large-scale chat applications.