How to handle 1 lakh data receiving from database at once in spring Boot?

 

Spring Boot 

Handle Large Data Receiving From Database

Handling large datasets, such as 1 lakh (100,000) records, in a Spring Boot application requires careful design to manage memory and performance. Here's a step-by-step approach:

---

1. Use Pagination

Fetching all records at once can overwhelm your application's memory. Instead, implement pagination to fetch data in smaller chunks.

Implementation:

• Add pagination support to your repository using Spring Data JPA:

  @Repository
  public interface MyRepository extends JpaRepository<MyEntity, Long> {
      Page<MyEntity> findAll(Pageable pageable);
  }
  

• Use it in your service:

  public Page<MyEntity> getData(int page, int size) {
      Pageable pageable = PageRequest.of(page, size);
      return myRepository.findAll(pageable);
  }
  

• Return paginated responses to the client:

  @GetMapping("/data")
  public Page<MyEntity> fetchPaginatedData(
          @RequestParam int page, 
          @RequestParam int size) {
      return myService.getData(page, size);
  }
  

---

2. Streaming Data

For scenarios where the client needs the entire dataset but the server cannot hold it in memory, use streaming with Hibernate ScrollableResults or Spring Data JPA Stream.

Implementation:

• Use Stream with Spring Data:

  @Query("SELECT e FROM MyEntity e")
  Stream<MyEntity> streamAll();
  

• Stream and process data:

  @Transactional(readOnly = true)
  public void processLargeDataset() {
      try (Stream<MyEntity> stream = myRepository.streamAll()) {
          stream.forEach(entity -> {
              // Process each record
          });
      }
  }
  

---

3. Batch Processing

If you need to perform operations on a large dataset, use batch processing with frameworks like Spring Batch.

Implementation:

• Configure a Job and Step in Spring Batch.
• Use a chunk-oriented processing model:

  @Bean
  public Step step1() {
      return stepBuilderFactory.get("step1")
              .<InputType, OutputType>chunk(1000)
              .reader(itemReader())
              .processor(itemProcessor())
              .writer(itemWriter())
              .build();
  }
  

---

4. Optimize Database Query

Ensure your query retrieves only the necessary columns and filters unnecessary data at the database level.

Example:

SELECT id, name FROM my_table WHERE condition = 'value';

• Use projections in JPA:

  @Query("SELECT new com.example.MyDTO(e.id, e.name) FROM MyEntity e WHERE e.condition = :condition")
  List<MyDTO> fetchPartialData(@Param("condition") String condition);
  

---

5. Async Processing

If fetching the data takes a long time, process it asynchronously to avoid blocking the main thread.

Implementation:

• Use @Async:

  @Async
  public CompletableFuture<List<MyEntity>> fetchDataAsync() {
      return CompletableFuture.completedFuture(myRepository.findAll());
  }
  

---

6. Data Compression

If transferring large data over the network, consider compressing the response using GZIP:

• Enable compression in Spring Boot:

  server:
    compression:
      enabled: true
      mime-types: application/json,application/xml
      min-response-size: 1024
  

---

Key Considerations:

1. Memory Management: Avoid loading all data into memory at once.
2. Database Tuning: Use indexes and optimized queries.
3. Scalability: Test with larger datasets to ensure scalability.
4. Client Handling: Implement proper client-side handling for large or paginated datasets.

These strategies ensure efficient handling of large datasets while maintaining application performance and reliability.