首页 > 应用开发 > Java > Java大Excel处理实战:高效导入导出解决方案
# Java大Excel处理实战:高效导入导出解决方案
在企业级应用中,大Excel文件的处理是常见需求。传统的POI库在处理大文件时容易导致内存溢出,本文将介绍多种高效处理大Excel文件的解决方案。
## 问题分析与技术选型
### 传统方式的问题
```java
// 传统POI方式 - 容易内存溢出
public void traditionalExcelRead(String filePath) throws IOException {
// 一次性加载整个文件到内存
FileInputStream fis = new FileInputStream(filePath);
Workbook workbook = new XSSFWorkbook(fis); // 对于xlsx文件
Sheet sheet = workbook.getSheetAt(0);
for (Row row : sheet) {
for (Cell cell : row) {
// 处理单元格数据
System.out.println(cell.getStringCellValue());
}
}
workbook.close();
fis.close();
// 当Excel很大时,这里会抛出OutOfMemoryError
}
```
### 技术方案对比
| 方案 | 适用场景 | 内存占用 | 性能 |
|------|----------|----------|------|
| POI SXSSF | 大文件导出 | 低 | 高 |
| POI Event API | 大文件导入 | 很低 | 高 |
| EasyExcel | 大文件读写 | 低 | 很高 |
| 分片处理 | 超大文件 | 很低 | 中 |
## 基于POI SXSSF的流式导出
### 基础导出实现
```java
@Service
@Slf4j
public class BigExcelExportService {
private static final int ROW_ACCESS_WINDOW_SIZE = 100;
private static final int DEFAULT_SHEET_SIZE = 100000;
/**
* 流式导出大数据到Excel
*/
public void exportLargeData(String filePath, List<String> headers,
Consumer<SXSSFSheet> dataProvider) throws IOException {
SXSSFWorkbook workbook = null;
FileOutputStream fos = null;
try {
// 创建SXSSF工作簿,设置行访问窗口大小
workbook = new SXSSFWorkbook(ROW_ACCESS_WINDOW_SIZE);
// 配置工作簿
setupWorkbookStyle(workbook);
// 创建Sheet
SXSSFSheet sheet = workbook.createSheet("数据导出");
// 创建表头
createHeaderRow(sheet, headers);
// 填充数据
dataProvider.accept(sheet);
// 写入文件
fos = new FileOutputStream(filePath);
workbook.write(fos);
log.info("Excel导出完成,文件路径: {}", filePath);
} finally {
// 清理临时文件
if (workbook != null) {
workbook.dispose();
}
if (fos != null) {
fos.close();
}
}
}
/**
* 配置工作簿样式
*/
<"deqxw.i0396.com">
<"xjcov.i0396.com">
<"tqnfc.i0396.com">
private void setupWorkbookStyle(SXSSFWorkbook workbook) {
// 创建字体
Font headerFont = workbook.createFont();
headerFont.setBold(true);
headerFont.setFontHeightInPoints((short) 12);
// 创建单元格样式
CellStyle headerStyle = workbook.createCellStyle();
headerStyle.setFont(headerFont);
headerStyle.setFillForegroundColor(IndexedColors.GREY_25_PERCENT.getIndex());
headerStyle.setFillPattern(FillPatternType.SOLID_FOREGROUND);
headerStyle.setBorderBottom(BorderStyle.THIN);
// 存储样式供后续使用
workbook.setSheetName(0, "数据");
}
/**
* 创建表头行
*/
private void createHeaderRow(SXSSFSheet sheet, List<String> headers) {
Row headerRow = sheet.createRow(0);
for (int i = 0; i < headers.size(); i++) {
Cell cell = headerRow.createCell(i);
cell.setCellValue(headers.get(i));
// 设置列宽自适应
sheet.trackColumnForAutoSizing(i);
sheet.autoSizeColumn(i);
}
}
/**
* 分页查询数据并导出
*/
public void exportWithPagination(String filePath,
Pageable pageable,
DataQueryService dataService) throws IOException {
List<String> headers = Arrays.asList("ID", "姓名", "邮箱", "手机号", "创建时间");
exportLargeData(filePath, headers, sheet -> {
int currentRow = 1; // 从第二行开始(表头占第一行)
int page = 0;
final int pageSize = pageable.getPageSize();
while (true) {
// 分页查询数据
Page<User> userPage = dataService.getUsers(PageRequest.of(page, pageSize));
List<User> users = userPage.getContent();
if (users.isEmpty()) {
break;
}
// 写入当前页数据
for (User user : users) {
Row row = sheet.createRow(currentRow++);
createUserDataRow(row, user);
}
// 手动刷新数据到磁盘,防止内存堆积
if (currentRow % ROW_ACCESS_WINDOW_SIZE == 0) {
try {
((SXSSFSheet) sheet).flushRows(ROW_ACCESS_WINDOW_SIZE);
} catch (IOException e) {
log.error("刷新行数据失败", e);
}
}
log.info("已导出第 {} 页数据,共 {} 条", page + 1, users.size());
if (!userPage.hasNext()) {
break;
}
page++;
}
});
}
/**
* 创建数据行
*/
private void createUserDataRow(Row row, User user) {
int col = 0;
row.createCell(col++).setCellValue(user.getId());
row.createCell(col++).setCellValue(user.getUsername());
row.createCell(col++).setCellValue(user.getEmail());
row.createCell(col++).setCellValue(user.getPhone());
Cell dateCell = row.createCell(col);
if (user.getCreateTime() != null) {
dateCell.setCellValue(user.getCreateTime().toString());
}
}
/**
* 多Sheet导出大量数据
*/
public void exportToMultipleSheets(String filePath,
Map<String, List<String>> sheetHeaders,
Map<String, Consumer<SXSSFSheet>> sheetDataProviders) throws IOException {
try (SXSSFWorkbook workbook = new SXSSFWorkbook(ROW_ACCESS_WINDOW_SIZE);
FileOutputStream fos = new FileOutputStream(filePath)) {
setupWorkbookStyle(workbook);
for (Map.Entry<String, List<String>> entry : sheetHeaders.entrySet()) {
String sheetName = entry.getKey();
List<String> headers = entry.getValue();
SXSSFSheet sheet = workbook.createSheet(sheetName);
createHeaderRow(sheet, headers);
// 调用数据提供者填充数据
Consumer<SXSSFSheet> dataProvider = sheetDataProviders.get(sheetName);
if (dataProvider != null) {
dataProvider.accept(sheet);
}
}
workbook.write(fos);
log.info("多Sheet Excel导出完成,文件路径: {}", filePath);
}
}
}
```
### Web导出控制器
```java
@RestController
@RequestMapping("/api/excel")
@Slf4j
<"bzcgd.i0396.com">
<"gsnfn.i0396.com">
<"zindw.i0396.com">
public class ExcelExportController {
@Autowired
private BigExcelExportService excelExportService;
@GetMapping("/export/users")
public void exportUsers(HttpServletResponse response,
@RequestParam(defaultValue = "0") int page,
@RequestParam(defaultValue = "1000") int size) throws IOException {
// 设置响应头
setupExportResponse(response, "用户数据导出.xlsx");
try {
Pageable pageable = PageRequest.of(page, size);
excelExportService.exportWithPagination(response.getOutputStream(), pageable);
} catch (Exception e) {
log.error("导出用户数据失败", e);
sendErrorResponse(response, "导出失败: " + e.getMessage());
}
}
/**
* 设置导出响应头
*/
private void setupExportResponse(HttpServletResponse response, String fileName) {
try {
response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
response.setCharacterEncoding("UTF-8");
String encodedFileName = URLEncoder.encode(fileName, "UTF-8")
.replaceAll("\\+", "%20");
response.setHeader("Content-Disposition",
"attachment; filename*=UTF-8''" + encodedFileName);
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate");
} catch (UnsupportedEncodingException e) {
log.error("文件名编码失败", e);
}
}
/**
* 发送错误响应
*/
private void sendErrorResponse(HttpServletResponse response, String errorMessage) {
try {
response.reset();
response.setContentType("application/json");
response.setCharacterEncoding("UTF-8");
response.getWriter().write("{\"error\": \"" + errorMessage + "\"}");
} catch (IOException e) {
log.error("发送错误响应失败", e);
}
}
}
```
## 基于POI Event Model的流式导入
### 事件模型导入实现
```java
@Service
@Slf4j
public class BigExcelImportService {
private static final int BATCH_SIZE = 1000;
/**
* 使用事件模型导入大Excel文件
*/
public ExcelImportResult importLargeExcel(File excelFile,
RowProcessor rowProcessor) throws Exception {
ExcelImportResult result = new ExcelImportResult();
long startTime = System.currentTimeMillis();
try (FileInputStream fis = new FileInputStream(excelFile)) {
// 创建OPCPackage
OPCPackage pkg = OPCPackage.open(fis);
// 创建XSSF Reader
XSSFReader reader = new XSSFReader(pkg);
// 获取共享字符串表
SharedStringsTable sst = reader.getSharedStringsTable();
// 创建样式表
StylesTable styles = reader.getStylesTable();
// 创建XML解析器
XMLReader parser = getSheetParser(sst, styles, rowProcessor, result);
// 读取第一个Sheet
InputStream sheetInputStream = reader.getSheet("rId1");
InputSource sheetSource = new InputSource(sheetInputStream);
// 解析Sheet
parser.parse(sheetSource);
sheetInputStream.close();
pkg.close();
} catch (Exception e) {
log.error("导入Excel文件失败", e);
result.setSuccess(false);
result.setErrorMessage(e.getMessage());
}
long endTime = System.currentTimeMillis();
result.setProcessTime(endTime - startTime);
log.info("Excel导入完成,总行数: {}, 处理时间: {}ms",
result.getTotalRows(), result.getProcessTime());
return result;
}
/**
* 获取Sheet解析器
*/
<"nrdpu.i0396.com">
<"xzpgm.i0396.com">
<"ylcrd.i0396.com">
private XMLReader getSheetParser(SharedStringsTable sst,
StylesTable styles,
RowProcessor rowProcessor,
ExcelImportResult result) throws Exception {
XMLReader parser = XMLReaderFactory.createXMLReader("org.apache.xerces.parsers.SAXParser");
// 创建内容处理器
SheetHandler handler = new SheetHandler(sst, styles, rowProcessor, result);
parser.setContentHandler(handler);
return parser;
}
/**
* Sheet内容处理器
*/
private static class SheetHandler extends DefaultHandler {
private final SharedStringsTable sst;
private final StylesTable styles;
private final RowProcessor rowProcessor;
private final ExcelImportResult result;
private String lastContents;
private boolean nextIsString;
private boolean nextIsDate;
private int currentRow = 0;
private int currentCol = 0;
private List<String> currentRowData;
private List<List<String>> batchData;
private int processedRows = 0;
public SheetHandler(SharedStringsTable sst, StylesTable styles,
RowProcessor rowProcessor, ExcelImportResult result) {
this.sst = sst;
this.styles = styles;
this.rowProcessor = rowProcessor;
this.result = result;
this.batchData = new ArrayList<>(BATCH_SIZE);
}
@Override
public void startElement(String uri, String localName, String name,
Attributes attributes) throws SAXException {
if ("row".equals(name)) {
// 新行开始
currentRow++;
currentCol = 0;
currentRowData = new ArrayList<>();
} else if ("c".equals(name)) {
// 单元格开始
String cellType = attributes.getValue("t");
String cellStyle = attributes.getValue("s");
nextIsString = "s".equals(cellType);
nextIsDate = isDateCell(cellStyle);
} else if ("v".equals(name)) {
// 单元格值开始
lastContents = "";
}
}
@Override
public void endElement(String uri, String localName, String name) throws SAXException {
if ("v".equals(name)) {
// 单元格值结束
String value = lastContents;
if (nextIsString) {
int idx = Integer.parseInt(value);
value = sst.getItemAt(idx).getString();
} else if (nextIsDate) {
value = convertExcelDate(value);
}
currentRowData.add(value);
currentCol++;
} else if ("row".equals(name)) {
// 行结束,处理行数据
processRowData();
}
}
@Override
public void characters(char[] ch, int start, int length) throws SAXException {
lastContents += new String(ch, start, length);
}
/**
* 处理行数据
*/
private void processRowData() {
try {
// 跳过表头(第一行)
if (currentRow > 1 && !currentRowData.isEmpty()) {
batchData.add(new ArrayList<>(currentRowData));
result.incrementTotalRows();
// 批量处理
if (batchData.size() >= BATCH_SIZE) {
processBatch();
}
}
} catch (Exception e) {
log.error("处理第 {} 行数据失败", currentRow, e);
result.incrementErrorRows();
}
}
/**
* 处理批次数据
*/
private void processBatch() {
if (!batchData.isEmpty()) {
try {
rowProcessor.processBatch(batchData);
processedRows += batchData.size();
log.debug("已处理 {} 行数据", processedRows);
} catch (Exception e) {
log.error("处理批次数据失败", e);
result.incrementErrorRowsBy(batchData.size());
}
batchData.clear();
}
}
/**
* 判断是否为日期单元格
*/
private boolean isDateCell(String cellStyle) {
if (cellStyle != null) {
int styleIndex = Integer.parseInt(cellStyle);
CellStyle style = styles.getStyleAt(styleIndex);
short formatIndex = style.getDataFormat();
String formatString = style.getDataFormatString();
return isDateFormat(formatIndex, formatString);
}
return false;
}
private boolean isDateFormat(short formatIndex, String formatString) {
return formatString.contains("yy") || formatString.contains("mm") ||
formatString.contains("dd") || formatString.contains("hh");
}
private String convertExcelDate(String excelDate) {
try {
double excelValue = Double.parseDouble(excelDate);
Date javaDate = DateUtil.getJavaDate(excelValue);
return new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(javaDate);
} catch (Exception e) {
return excelDate;
}
}
@Override
public void endDocument() throws SAXException {
// 文档结束,处理剩余数据
<"vqwqi.i0396.com">
<"vzarg.i0396.com">
<"kquqq.i0396.com">
processBatch();
result.setSuccess(result.getErrorRows() == 0);
}
}
}
/**
* 行处理器接口
*/
public interface RowProcessor {
void processBatch(List<List<String>> batchData) throws Exception;
}
/**
* 导入结果
*/
@Data
class ExcelImportResult {
private boolean success;
private long totalRows;
private long errorRows;
private long processTime;
private String errorMessage;
public void incrementTotalRows() {
totalRows++;
}
public void incrementErrorRows() {
errorRows++;
}
public void incrementErrorRowsBy(long count) {
errorRows += count;
}
}
```
### 具体业务处理器
```java
@Service
@Slf4j
public class UserExcelProcessor implements RowProcessor {
@Autowired
private UserRepository userRepository;
@Override
@Transactional
public void processBatch(List<List<String>> batchData) throws Exception {
List<User> users = new ArrayList<>();
for (List<String> rowData : batchData) {
try {
User user = convertToUser(rowData);
if (user != null) {
users.add(user);
}
} catch (Exception e) {
log.warn("转换行数据失败: {}", rowData, e);
}
}
if (!users.isEmpty()) {
userRepository.saveAll(users);
log.info("批量保存用户数据: {} 条", users.size());
}
}
/**
* 将行数据转换为用户对象
*/
private User convertToUser(List<String> rowData) {
if (rowData.size() < 5) {
return null;
}
try {
User user = new User();
user.setUsername(getStringValue(rowData, 0));
user.setEmail(getStringValue(rowData, 1));
user.setPhone(getStringValue(rowData, 2));
user.setAge(getIntegerValue(rowData, 3));
user.setCreateTime(getDateValue(rowData, 4));
// 数据验证
if (!isValidUser(user)) {
return null;
}
return user;
} catch (Exception e) {
log.warn("用户数据转换失败: {}", rowData, e);
return null;
}
}
private String getStringValue(List<String> rowData, int index) {
return index < rowData.size() ? rowData.get(index) : null;
}
private Integer getIntegerValue(List<String> rowData, int index) {
try {
String value = getStringValue(rowData, index);
return value != null ? Integer.parseInt(value) : null;
} catch (NumberFormatException e) {
return null;
}
}
private LocalDateTime getDateValue(List<String> rowData, int index) {
try {
String value = getStringValue(rowData, index);
if (value != null) {
return LocalDateTime.parse(value,
DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"));
}
} catch (Exception e) {
// 解析失败,返回null
}
return null;
}
private boolean isValidUser(User user) {
return user.getUsername() != null &&
user.getEmail() != null &&
user.getEmail().contains("@");
}
}
```
## 基于EasyExcel的简化方案
### EasyExcel依赖配置
```xml
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>easyexcel</artifactId>
<version>3.3.2</version>
</dependency>
<"gsyvv.i0396.com">
<"dswgw.i0396.com">
<"iknny.i0396.com">
```
### EasyExcel导入导出
```java
@Service
@Slf4j
public class EasyExcelService {
/**
* 使用EasyExcel导出大数据
*/
public void exportWithEasyExcel(String filePath, List<User> data) {
try {
EasyExcel.write(filePath, User.class)
.sheet("用户数据")
.doWrite(data);
log.info("EasyExcel导出完成,文件路径: {}", filePath);
} catch (Exception e) {
log.error("EasyExcel导出失败", e);
throw new RuntimeException("导出失败", e);
}
}
/**
* 流式导出大量数据
*/
public void exportLargeDataWithEasyExcel(String filePath,
DataSupplier<User> dataSupplier) {
try {
EasyExcel.write(filePath, User.class)
.sheet("用户数据")
.doWrite(() -> dataSupplier.getData());
} catch (Exception e) {
log.error("EasyExcel流式导出失败", e);
throw new RuntimeException("导出失败", e);
}
}
/**
* 使用EasyExcel读取大文件
*/
public List<User> readLargeExcel(String filePath) {
List<User> result = new ArrayList<>();
EasyExcel.read(filePath, User.class, new AnalysisEventListener<User>() {
private static final int BATCH_COUNT = 1000;
private List<User> cachedData = new ArrayList<>(BATCH_COUNT);
@Override
public void invoke(User user, AnalysisContext context) {
cachedData.add(user);
if (cachedData.size() >= BATCH_COUNT) {
saveData();
cachedData = new ArrayList<>(BATCH_COUNT);
}
}
@Override
public void doAfterAllAnalysed(AnalysisContext context) {
saveData();
log.info("Excel读取完成");
}
private void saveData() {
if (!cachedData.isEmpty()) {
result.addAll(cachedData);
log.info("保存 {} 条数据", cachedData.size());
}
}
}).sheet().doRead();
return result;
}
}
/**
* 数据供应接口
*/
public interface DataSupplier<T> {
Iterator<T> getData();
<"arbpw.i0396.com">
<"rbvix.i0396.com">
<"zdfvo.i0396.com">
}
// 用户实体类(需要注解)
@Data
public class User {
@ExcelProperty("ID")
private Long id;
@ExcelProperty("用户名")
private String username;
@ExcelProperty("邮箱")
private String email;
@ExcelProperty("手机号")
private String phone;
@ExcelProperty("年龄")
private Integer age;
@ExcelProperty("创建时间")
private String createTime;
@ExcelIgnore
private String ignoreField; // 忽略的字段
}
```
## 性能优化与监控
### 内存监控与调优
```java
@Component
@Slf4j
public class ExcelMemoryMonitor {
private static final long MEMORY_THRESHOLD = 100 * 1024 * 1024; // 100MB
/**
* 检查内存使用情况
*/
public boolean checkMemoryHealth() {
Runtime runtime = Runtime.getRuntime();
long usedMemory = runtime.totalMemory() - runtime.freeMemory();
long maxMemory = runtime.maxMemory();
long availableMemory = maxMemory - usedMemory;
log.debug("内存使用情况 - 已用: {}MB, 最大: {}MB, 可用: {}MB",
usedMemory / 1024 / 1024,
maxMemory / 1024 / 1024,
availableMemory / 1024 / 1024);
return availableMemory > MEMORY_THRESHOLD;
}
/**
* 执行垃圾回收
*/
public void performGarbageCollection() {
log.info("执行垃圾回收...");
System.gc();
try {
Thread.sleep(1000); // 等待GC完成
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
/**
* 监控导出任务
*/
public void monitorExportTask(Runnable exportTask, String taskName) {
long startTime = System.currentTimeMillis();
long startMemory = getUsedMemory();
try {
if (!checkMemoryHealth()) {
performGarbageCollection();
}
exportTask.run();
} finally {
long endTime = System.currentTimeMillis();
long endMemory = getUsedMemory();
long memoryUsed = endMemory - startMemory;
log.info("任务 {} 完成 - 耗时: {}ms, 内存使用: {}MB",
taskName, (endTime - startTime), memoryUsed / 1024 / 1024);
}
}
private long getUsedMemory() {
Runtime runtime = Runtime.getRuntime();
return runtime.totalMemory() - runtime.freeMemory();
}
<"imokk.i0396.com">
<"zjzno.i0396.com">
}
```
### 分片处理策略
```java
@Service
@Slf4j
public class ShardingExcelService {
/**
* 分片导出大文件
*/
public void exportWithSharding(String basePath,
int totalRecords,
int shardSize,
Function<Integer, List<User>> dataFetcher) throws IOException {
int totalShards = (int) Math.ceil((double) totalRecords / shardSize);
for (int shard = 0; shard < totalShards; shard++) {
String filePath = basePath + "_part_" + (shard + 1) + ".xlsx";
// 获取当前分片数据
List<User> shardData = dataFetcher.apply(shard);
// 导出分片
exportShardData(filePath, shardData, shard + 1, totalShards);
log.info("分片 {} / {} 导出完成", shard + 1, totalShards);
// 分片间执行GC,释放内存
System.gc();
}
}
private void exportShardData(String filePath, List<User> data,
int currentShard, int totalShards) throws IOException {
try (SXSSFWorkbook workbook = new SXSSFWorkbook(100);
FileOutputStream fos = new FileOutputStream(filePath)) {
SXSSFSheet sheet = workbook.createSheet("数据分片");
// 创建表头
createShardHeader(sheet, currentShard, totalShards);
// 填充数据
int rowNum = 1;
for (User user : data) {
Row row = sheet.createRow(rowNum++);
fillUserRow(row, user);
}
workbook.write(fos);
}
}
private void createShardHeader(SXSSFSheet sheet, int currentShard, int totalShards) {
Row headerRow = sheet.createRow(0);
headerRow.createCell(0).setCellValue("分片信息: " + currentShard + "/" + totalShards);
Row dataHeader = sheet.createRow(1);
String[] headers = {"ID", "用户名", "邮箱", "手机号"};
for (int i = 0; i < headers.length; i++) {
dataHeader.createCell(i).setCellValue(headers[i]);
}
}
private void fillUserRow(Row row, User user) {
int col = 0;
row.createCell(col++).setCellValue(user.getId());
row.createCell(col++).setCellValue(user.getUsername());
row.createCell(col++).setCellValue(user.getEmail());
row.createCell(col++).setCellValue(user.getPhone());
}
}
```
## 最佳实践总结
### 1. 方案选择指南
- **小文件(<10MB)**:使用传统POI HSSF/XSSF
- **大文件导出**:使用POI SXSSF或EasyExcel
- **大文件导入**:使用POI Event Model或EasyExcel
- **超大文件(>100MB)**:使用分片处理
### 2. 内存管理要点
```java
// 正确的资源清理
public void safeExcelOperation() {
Workbook workbook = null;
FileInputStream fis = null;
try {
fis = new FileInputStream("file.xlsx");
workbook = new XSSFWorkbook(fis);
// 处理逻辑
} catch (IOException e) {
log.error("处理Excel失败", e);
} finally {
// 确保资源被正确关闭
try {
if (workbook != null) {
workbook.close();
}
if (fis != null) {
fis.close();
}
} catch (IOException e) {
log.error("关闭资源失败", e);
}
}
}
```
### 3. 性能优化建议
- 设置合适的批处理大小
- 定期刷新流式写入的数据
- 避免在循环中创建对象
- 使用连接池管理数据库连接
- 监控内存使用,适时执行GC
通过本文介绍的各种方案,开发者可以根据具体场景选择合适的大Excel处理方式,有效避免内存溢出问题,提升系统稳定性。
来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/19545/viewspace-3102467/ ,如需转载,请注明出处,否则将追究法律责任。