Compare commits
16 Commits
a904137b60
...
vue
| Author | SHA1 | Date | |
|---|---|---|---|
| 2614af7808 | |||
| 7bfba04199 | |||
| d78dc655ee | |||
| bd46d8f907 | |||
| 0648770a6a | |||
| 40118ec508 | |||
| b9270428db | |||
| 4f504447a1 | |||
| a2408fa952 | |||
| 61c93882d6 | |||
| 4653b1d7b9 | |||
| ad659c4ff0 | |||
| e3766b86be | |||
| ed03bd2032 | |||
| fb70356f5d | |||
| f2c856ab05 |
5
.env
Normal file
5
.env
Normal file
@@ -0,0 +1,5 @@
|
||||
# 服务器端口配置
|
||||
PORT=5000
|
||||
|
||||
# Firecrawl API Key（在 https://www.firecrawl.dev/app/api-keys 获取）
|
||||
FIRECRAWL_API_KEY=fc-595dd922780442f8a907202666a522ef
|
||||
@@ -1,11 +1,15 @@
|
||||
# 服务器端口配置
|
||||
PORT=5000
|
||||
|
||||
# Firecrawl API Key（在 https://www.firecrawl.dev/app/api-keys 获取）
|
||||
FIRECRAWL_API_KEY=fc-your-api-key-here
|
||||
|
||||
# 环境说明：
|
||||
# - 开发环境：通常使用 5000
|
||||
# - 生产环境：可以使用 80、8080 等
|
||||
#
|
||||
# 使用方法：
|
||||
# 1. 复制此文件为 .env
|
||||
# 2. 修改端口号
|
||||
# 3. 启动服务时会自动读取
|
||||
# 2. 填写 FIRECRAWL_API_KEY
|
||||
# 3. 修改端口号（可选）
|
||||
# 4. 启动服务时会自动读取
|
||||
|
||||
@@ -3,7 +3,7 @@ name: Deploy Vue App
|
||||
# 触发条件：推送到 master 分支时执行
|
||||
on:
|
||||
push:
|
||||
branches: ['master']
|
||||
branches: ["南京公共资源交易中心"]
|
||||
|
||||
env:
|
||||
IP: 106.15.181.192
|
||||
@@ -13,11 +13,20 @@ jobs:
|
||||
# 运行在已注册的 runner 上（确保 runner 标签匹配，默认无标签可省略）
|
||||
runs-on: [test]
|
||||
steps:
|
||||
# 步骤 4：部署到服务器
|
||||
- name: Deploy to server
|
||||
# 步骤 1：拉取代码
|
||||
- name: Checkout code
|
||||
uses: https://gitee.com/skr2005/checkout@v4
|
||||
# 步骤 2：停止服务
|
||||
- name: Stop service
|
||||
run: net stop tool-node
|
||||
|
||||
# 步骤 3：拷贝源码到目标目录
|
||||
- name: Copy source to target
|
||||
run: xcopy /E /I /Y "${{ github.workspace }}\*" "D:\tools\tool-node\"
|
||||
|
||||
# 步骤 4：安装依赖并启动服务
|
||||
- name: Install and start service
|
||||
run: |
|
||||
cd D:\tools\tool-node
|
||||
npm install
|
||||
git pull
|
||||
net stop tool-node
|
||||
net start tool-node
|
||||
|
||||
7
.gitignore
vendored
7
.gitignore
vendored
@@ -8,12 +8,17 @@ yarn-error.log*
|
||||
pnpm-debug.log*
|
||||
|
||||
# 环境变量文件
|
||||
.env
|
||||
# .env (已注释，不再忽略)
|
||||
.env.local
|
||||
.env.*.local
|
||||
|
||||
# 配置文件（包含敏感信息）
|
||||
config.json
|
||||
data/
|
||||
*.sqlite
|
||||
*.sqlite-shm
|
||||
*.sqlite-wal
|
||||
*.migrated.bak
|
||||
|
||||
# 编辑器目录和文件
|
||||
.vscode/
|
||||
|
||||
619
README.md
619
README.md
@@ -1,536 +1,191 @@
|
||||
# 南京公共工程建设中心 - 公告采集工具
|
||||
# 公告抓取与分析工具
|
||||
|
||||
一个用于采集南京公共工程建设中心公告信息的 Web 可视化工具，支持定时任务和邮件推送。
|
||||
一个用于管理公告抓取任务、查看抓取结果、查询项目数据并维护系统配置的工具。
|
||||
|
||||
## 功能特性
|
||||
当前项目结构已经升级为：
|
||||
|
||||
- ✅ 采集公告列表(支持分页)
|
||||
- ✅ 按时间范围智能采集
|
||||
- ✅ 采集公告详情内容
|
||||
- ✅ 智能提取预算金额
|
||||
- ✅ 生成统计报告
|
||||
- ✅ Web 可视化界面
|
||||
- ✅ 导出 Word/Markdown 报告
|
||||
- ✅ RESTful API 支持
|
||||
- ✅ **定时任务自动采集**
|
||||
- ✅ **邮件推送 HTML 报告**
|
||||
- ✅ **Web 配置界面**
|
||||
- ✅ **无需数据库，轻量部署**
|
||||
- 前端：Vue 3 + JavaScript + Vite + Vue Router + Axios + Element Plus
|
||||
- 后端：Node.js + Express
|
||||
- 数据存储：better-sqlite3
|
||||
- 调度：node-cron
|
||||
- 邮件：nodemailer
|
||||
|
||||
## 快速开始
|
||||
## 功能概览
|
||||
|
||||
### 1. 安装依赖
|
||||
- 任务管理：新增、编辑、启用、禁用、删除任务
|
||||
- 手动执行：支持单任务运行和批量运行
|
||||
- 结果查看：按城市、板块、类型筛选抓取结果
|
||||
- 项目查询：按项目名称、金额、日期范围过滤项目
|
||||
- 系统设置：维护 Agent、定时任务、邮件配置
|
||||
- 定时调度：支持 cron 表达式配置
|
||||
- 数据持久化：任务与结果保存在 SQLite
|
||||
|
||||
## 环境要求
|
||||
|
||||
- Node.js 20 及以上可运行当前项目
|
||||
- 建议使用 Node.js 22
|
||||
|
||||
说明：
|
||||
|
||||
- 当前依赖 `@mendable/firecrawl-js` 会提示要求 Node `>=22`
|
||||
- 目前在 Node `20.19.4` 下可以安装、构建和启动
|
||||
- 如果后续采集运行时出现环境兼容问题，优先升级到 Node 22
|
||||
|
||||
## 安装依赖
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
### 2. 配置文件
|
||||
## 配置文件
|
||||
|
||||
首次使用需要创建配置文件：
|
||||
项目根目录下使用 `config.json` 作为运行配置文件。
|
||||
|
||||
如果你还没有该文件，可以参考：
|
||||
|
||||
```bash
|
||||
# 复制示例配置文件
|
||||
cp config.example.json config.json
|
||||
|
||||
# 编辑配置文件（或通过 Web 界面配置）
|
||||
# 填写邮件服务器信息和定时任务设置
|
||||
```
|
||||
|
||||
**配置文件说明：**
|
||||
需要重点配置的内容包括：
|
||||
|
||||
- `config.example.json` - 配置模板（不含敏感信息，可提交到 Git）
|
||||
- `config.json` - 实际配置（包含密码等敏感信息，已在 .gitignore 中忽略）
|
||||
- `agent.baseUrl`
|
||||
- `scheduler.enabled`
|
||||
- `scheduler.cronTime`
|
||||
- `email.smtpHost`
|
||||
- `email.smtpUser`
|
||||
- `email.smtpPass`
|
||||
- `email.recipients`
|
||||
|
||||
## 使用方法
|
||||
## 启动方式
|
||||
|
||||
### 1. 启动服务器
|
||||
### 开发模式
|
||||
|
||||
前后端同时启动：
|
||||
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
|
||||
启动后地址如下：
|
||||
|
||||
- 前端开发服务：`http://localhost:5173`
|
||||
- 后端 API 服务：`http://localhost:5000`
|
||||
|
||||
说明：
|
||||
|
||||
- Vite 前端会通过代理把 `/api` 请求转发到 `5000`
|
||||
- 开发时建议直接访问 `http://localhost:5173`
|
||||
|
||||
### 生产模式
|
||||
|
||||
先构建前端，再启动 Express：
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
npm start
|
||||
```
|
||||
|
||||
### 2. 访问界面
|
||||
启动后访问：
|
||||
|
||||
打开浏览器访问: **http://localhost:5000** （或您配置的端口）
|
||||
- `http://localhost:5000`
|
||||
|
||||
### 3. 功能介绍
|
||||
说明：
|
||||
|
||||
**公告列表标签**
|
||||
- `npm run build` 会把前端构建到 `dist/`
|
||||
- `npm start` 会由 Express 托管 `dist/` 静态资源和 `/api`
|
||||
|
||||
- 快速查看所有公告
|
||||
- 支持分页浏览
|
||||
- 一键获取最新公告列表
|
||||
|
||||
**详情采集标签**
|
||||
|
||||
- 批量采集公告详情
|
||||
- 支持按时间范围采集
|
||||
- 自动提取预算金额
|
||||
- 可自定义采集数量
|
||||
|
||||
**生成报告标签**
|
||||
|
||||
- 支持按时间范围生成报告
|
||||
- 设置金额阈值筛选项目
|
||||
- 实时统计项目信息
|
||||
- 一键导出 Word/Markdown 报告
|
||||
|
||||
**定时任务标签** ⭐ 新增
|
||||
|
||||
- Web 界面配置定时任务
|
||||
- 支持 Cron 表达式自定义执行时间
|
||||
- 可选时间范围（今日/本周/本月）
|
||||
- 设置金额阈值自动筛选
|
||||
- 实时查看任务运行状态
|
||||
- 立即测试运行功能
|
||||
|
||||
**邮件配置标签** ⭐ 新增
|
||||
|
||||
- Web 界面配置 SMTP 邮件服务
|
||||
- 支持主流邮箱（QQ、163、Gmail 等）
|
||||
- 测试连接功能验证配置
|
||||
- 支持多个收件人
|
||||
- 自动发送精美 HTML 报告
|
||||
|
||||
## 报告示例
|
||||
|
||||
```markdown
|
||||
# 南京公共工程建设项目报告
|
||||
|
||||
**生成时间**: 2025/12/12 11:00:03
|
||||
|
||||
## 统计摘要
|
||||
|
||||
- 总项目数: 10
|
||||
- 超过 50 万元的项目: 3
|
||||
- 总金额: 5395.50 万元
|
||||
|
||||
## 项目列表
|
||||
|
||||
### 1. 项目名称
|
||||
|
||||
- **发布日期**: 2025-12-12
|
||||
- **发布时间**: 2025-12-12 10:35:00
|
||||
- **预算金额**: 5000 万元
|
||||
- **链接**: https://...
|
||||
```
|
||||
|
||||
## API 接口文档
|
||||
|
||||
服务器启动后提供以下 RESTful API 接口:
|
||||
|
||||
### 1. 获取公告列表
|
||||
|
||||
```
|
||||
GET /api/list?url=<列表页URL>&page=<页码>
|
||||
```
|
||||
|
||||
参数:
|
||||
|
||||
- `url` (可选): 列表页 URL,默认为官网首页
|
||||
- `page` (可选): 页码,默认为 1
|
||||
|
||||
### 2. 按时间范围获取列表
|
||||
|
||||
```
|
||||
POST /api/list-daterange
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"startDate": "2025-11-01",
|
||||
"endDate": "2025-12-31",
|
||||
"maxPages": 23
|
||||
}
|
||||
```
|
||||
|
||||
### 3. 批量获取详情
|
||||
|
||||
```
|
||||
POST /api/details
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"items": [{ "title": "...", "href": "...", "date": "..." }],
|
||||
"limit": 10
|
||||
}
|
||||
```
|
||||
|
||||
### 4. 生成报告
|
||||
|
||||
```
|
||||
POST /api/report
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"url": "https://gjzx.nanjing.gov.cn/gggs/",
|
||||
"limit": 15,
|
||||
"threshold": 50
|
||||
}
|
||||
```
|
||||
|
||||
### 5. 按时间范围生成报告
|
||||
|
||||
```
|
||||
POST /api/report-daterange
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"startDate": "2025-11-01",
|
||||
"endDate": "2025-12-31",
|
||||
"threshold": 50,
|
||||
"maxPages": 23
|
||||
}
|
||||
```
|
||||
|
||||
### 6. 获取配置
|
||||
|
||||
```
|
||||
GET /api/config
|
||||
```
|
||||
|
||||
返回当前配置信息（密码会被隐藏）
|
||||
|
||||
### 7. 更新配置
|
||||
|
||||
```
|
||||
POST /api/config
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"scheduler": {
|
||||
"enabled": true,
|
||||
"cronTime": "0 9 * * *",
|
||||
"threshold": 100000,
|
||||
"timeRange": "today"
|
||||
},
|
||||
"email": {
|
||||
"smtpHost": "smtp.qq.com",
|
||||
"smtpPort": 587,
|
||||
"smtpUser": "your-email@qq.com",
|
||||
"smtpPass": "your-password",
|
||||
"recipients": "recipient@example.com"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 8. 获取定时任务状态
|
||||
|
||||
```
|
||||
GET /api/scheduler/status
|
||||
```
|
||||
|
||||
### 9. 手动触发定时任务
|
||||
|
||||
```
|
||||
POST /api/run-scheduled-task
|
||||
```
|
||||
|
||||
## 定时任务配置
|
||||
|
||||
### 通过 Web 界面配置（推荐）
|
||||
|
||||
1. 访问 `http://localhost:5000` （或您配置的端口）
|
||||
2. 切换到 **"定时任务"** 标签
|
||||
3. 配置以下选项：
|
||||
- **启用定时任务**：勾选启用
|
||||
- **执行时间**：选择预设时间或自定义 Cron 表达式
|
||||
- **时间范围**：今日/本周/本月
|
||||
- **金额阈值**：设置最低金额（单位：万元）
|
||||
4. 切换到 **"邮件配置"** 标签
|
||||
5. 配置 SMTP 邮件服务器信息
|
||||
6. 点击"测试连接"验证配置
|
||||
7. 点击"保存配置"，定时任务自动生效
|
||||
|
||||
### 通过配置文件
|
||||
|
||||
编辑 `config.json`：
|
||||
|
||||
```json
|
||||
{
|
||||
"scheduler": {
|
||||
"enabled": true, // 是否启用
|
||||
"cronTime": "0 9 * * *", // Cron表达式：每天9点
|
||||
"threshold": 100000, // 金额阈值（万元，10亿）
|
||||
"description": "每天9点采集大于10亿的项目",
|
||||
"timeRange": "today" // 采集范围: today/thisWeek/thisMonth
|
||||
},
|
||||
"email": {
|
||||
"smtpHost": "smtp.qq.com", // SMTP服务器
|
||||
"smtpPort": 587, // 端口
|
||||
"smtpUser": "your@qq.com", // 发件邮箱
|
||||
"smtpPass": "授权码", // QQ邮箱授权码
|
||||
"recipients": "to@qq.com" // 收件人（多个用逗号分隔）
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cron 表达式说明
|
||||
|
||||
格式：`分 时 日 月 周`
|
||||
|
||||
常用示例：
|
||||
- `0 9 * * *` - 每天 9:00
|
||||
- `0 9,18 * * *` - 每天 9:00 和 18:00
|
||||
- `0 9 * * 1` - 每周一 9:00
|
||||
- `0 9 1 * *` - 每月1号 9:00
|
||||
- `0 */6 * * *` - 每 6 小时
|
||||
|
||||
### 邮件服务配置参考
|
||||
|
||||
**QQ 邮箱：**
|
||||
- SMTP: `smtp.qq.com`
|
||||
- 端口: `587`
|
||||
- 密码: 使用授权码（在 QQ 邮箱设置中生成）
|
||||
|
||||
**163 邮箱：**
|
||||
- SMTP: `smtp.163.com`
|
||||
- 端口: `465`
|
||||
- 密码: 使用授权码
|
||||
|
||||
**Gmail：**
|
||||
- SMTP: `smtp.gmail.com`
|
||||
- 端口: `587`
|
||||
- 需要开启"不够安全的应用访问"
|
||||
|
||||
## 服务器部署
|
||||
|
||||
### 方法 1：使用 PM2（推荐）
|
||||
## 常用脚本
|
||||
|
||||
```bash
|
||||
# 安装 PM2
|
||||
npm install -g pm2
|
||||
|
||||
# 启动服务
|
||||
pm2 start src/server.js --name gjzx-scraper
|
||||
|
||||
# 查看状态
|
||||
pm2 status
|
||||
|
||||
# 查看日志
|
||||
pm2 logs gjzx-scraper
|
||||
|
||||
# 设置开机自启
|
||||
pm2 startup
|
||||
pm2 save
|
||||
npm run dev
|
||||
npm run build
|
||||
npm run preview
|
||||
npm start
|
||||
```
|
||||
|
||||
### 方法 2：使用 systemd
|
||||
脚本说明：
|
||||
|
||||
创建服务文件 `/etc/systemd/system/gjzx-scraper.service`：
|
||||
- `npm run dev`：开发模式，前后端同时启动
|
||||
- `npm run build`：构建前端生产包
|
||||
- `npm run preview`：本地预览前端构建结果
|
||||
- `npm start`：启动后端服务，并托管前端构建产物
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=GJZX Scraper Service
|
||||
After=network.target
|
||||
## 页面说明
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=your-user
|
||||
WorkingDirectory=/path/to/tool-node
|
||||
ExecStart=/usr/bin/node src/server.js
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
### 任务配置
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
- 管理抓取任务
|
||||
- 支持单独运行和批量运行
|
||||
- 支持查看运行状态
|
||||
|
||||
启动服务：
|
||||
### 抓取结果
|
||||
|
||||
```bash
|
||||
sudo systemctl enable gjzx-scraper
|
||||
sudo systemctl start gjzx-scraper
|
||||
sudo systemctl status gjzx-scraper
|
||||
```
|
||||
- 查看每次抓取生成的记录
|
||||
- 支持按条件筛选
|
||||
- 支持删除结果
|
||||
|
||||
### 注意事项
|
||||
### 项目管理
|
||||
|
||||
- ✅ **无需数据库**：项目采用无数据库架构，轻量部署
|
||||
- ✅ **配置持久化**：所有配置保存在 `config.json` 文件中
|
||||
- ✅ **进程保活**：使用 PM2 或 systemd 确保进程持续运行
|
||||
- ⚠️ **防火墙**：确保配置的端口可访问（默认 5000）
|
||||
- ⚠️ **配置安全**：不要将 `config.json` 提交到公开仓库
|
||||
- 对抓取出的项目做去重查询
|
||||
- 支持金额和日期范围过滤
|
||||
|
||||
### 系统设置
|
||||
|
||||
- 配置 Agent 服务地址
|
||||
- 配置定时任务
|
||||
- 配置邮件发送参数
|
||||
|
||||
## 技术栈
|
||||
|
||||
- **后端**: Node.js + Express
|
||||
- **爬虫**: Axios + Cheerio
|
||||
- **定时任务**: node-cron
|
||||
- **邮件服务**: nodemailer
|
||||
- **前端**: 原生 HTML/CSS/JavaScript
|
||||
- **编码处理**: iconv-lite (支持 GBK/UTF-8)
|
||||
- **文档导出**: docx.js
|
||||
- **架构**: 无数据库设计
|
||||
- 前端：Vue 3、Vite、Vue Router、Axios、Element Plus
|
||||
- 后端：Express
|
||||
- 数据库：better-sqlite3
|
||||
- 调度：node-cron
|
||||
- 邮件：nodemailer
|
||||
|
||||
## 项目结构
|
||||
|
||||
```
|
||||
```text
|
||||
.
|
||||
├── src/
|
||||
│ ├── server.js # Web服务器及API
|
||||
│ ├── scheduler.js # 定时任务调度器
|
||||
│ └── emailService.js # 邮件发送服务
|
||||
├── public/
|
||||
│ ├── index.html # Web界面
|
||||
│ └── app.js # 前端逻辑
|
||||
├── config.json # 配置文件（不提交到Git）
|
||||
├── config.example.json # 配置示例
|
||||
├── package.json
|
||||
└── README.md
|
||||
├─ client/ # Vue 3 前端源码
|
||||
│ ├─ index.html
|
||||
│ └─ src/
|
||||
│ ├─ api/
|
||||
│ ├─ components/
|
||||
│ ├─ pages/
|
||||
│ ├─ router/
|
||||
│ ├─ App.vue
|
||||
│ ├─ main.js
|
||||
│ └─ styles.css
|
||||
├─ dist/ # Vite 构建产物
|
||||
├─ src/ # Express 服务端
|
||||
│ ├─ server.js
|
||||
│ ├─ scheduler.js
|
||||
│ ├─ resultStore.js
|
||||
│ ├─ agentService.js
|
||||
│ └─ emailService.js
|
||||
├─ data/ # SQLite 数据文件目录
|
||||
├─ config.json # 运行配置
|
||||
├─ config.example.json # 配置示例
|
||||
├─ package.json
|
||||
├─ vite.config.js
|
||||
└─ README.md
|
||||
```
|
||||
|
||||
## 架构特点
|
||||
## 数据说明
|
||||
|
||||
### 无数据库设计
|
||||
- 任务数据和抓取结果保存在 SQLite
|
||||
- 默认数据库路径位于 `data/results.sqlite`
|
||||
- 配置数据保存在根目录 `config.json`
|
||||
|
||||
本项目采用**无数据库架构**，具有以下特点：
|
||||
## 部署建议
|
||||
|
||||
- ✅ **轻量部署**：无需安装和配置数据库
|
||||
- ✅ **实时数据**：每次从源站实时抓取最新数据
|
||||
- ✅ **配置简单**：只需配置 config.json 文件
|
||||
- ✅ **邮件归档**：报告通过邮件发送，邮箱即为历史记录
|
||||
- ✅ **低资源消耗**：内存占用小，适合小型服务器
|
||||
- 使用 PM2 或 systemd 保持进程常驻
|
||||
- 通过反向代理暴露 `5000` 端口
|
||||
- 不要把 `config.json`、`data/`、`.env` 提交到公开仓库
|
||||
|
||||
### 数据流程
|
||||
## 备注
|
||||
|
||||
```
|
||||
定时触发 → 抓取网站数据 → 解析提取 → 筛选过滤 → 生成报告 → 发送邮件
|
||||
↑ ↓
|
||||
└──────────────────── 配置文件 ──────────────────────────┘
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. **采集速度**：已限制为每条延迟 500ms-1s，避免请求过快
|
||||
2. **域名支持**：仅支持 gjzx.nanjing.gov.cn 域名的详情页解析
|
||||
3. **金额提取**：基于正则匹配，支持多种格式（预算金额、最高限价等）
|
||||
4. **端口配置**：Web 服务器默认端口 5000，支持通过环境变量 PORT 修改
|
||||
5. **智能停止**：按时间范围采集会在检测到所有公告早于起始日期时自动停止
|
||||
6. **编码处理**：自动识别，支持 GBK 和 UTF-8 网页
|
||||
7. **配置安全**：config.json 包含敏感信息，已加入 .gitignore，不要提交到公开仓库
|
||||
8. **进程保活**：部署时使用 PM2 或 systemd 确保定时任务持续运行
|
||||
|
||||
## 核心功能说明
|
||||
|
||||
### 时间范围采集逻辑
|
||||
|
||||
按时间范围采集时,程序会:
|
||||
|
||||
1. 从第一页开始顺序采集
|
||||
2. 检查每页公告的日期是否在指定范围内
|
||||
3. 如果某页所有公告都早于起始日期,自动停止采集
|
||||
4. 支持设置最大页数限制,避免过度采集
|
||||
|
||||
### 金额提取规则
|
||||
|
||||
支持识别以下格式:
|
||||
|
||||
- 预算金额: XX 万元
|
||||
- 最高限价: XX 万元
|
||||
- 预算: XX 万元
|
||||
- 金额: XX 万元
|
||||
- 直接数字: XX 万元
|
||||
|
||||
### 编码处理
|
||||
|
||||
自动识别网页编码:
|
||||
|
||||
- 优先读取 Content-Type 中的 charset
|
||||
- 自动处理 GBK、GB2312 编码
|
||||
- 默认使用 UTF-8
|
||||
|
||||
## 常见问题
|
||||
|
||||
### Q: 为什么采集速度比较慢？
|
||||
|
||||
A: 为了避免对服务器造成过大压力，程序限制了请求频率（每条延迟 500ms-1s）。这是一个负责任的爬虫设计。
|
||||
|
||||
### Q: 如何采集指定日期范围的公告？
|
||||
|
||||
A: 在 Web 界面的"详情采集"和"生成报告"标签中勾选"按时间范围采集"，然后输入起始和结束日期即可。
|
||||
|
||||
### Q: 导出的报告在哪里？
|
||||
|
||||
A: 点击"导出 Word"或"导出 Markdown"按钮后会自动下载到浏览器的默认下载目录。
|
||||
|
||||
### Q: 可以采集其他网站吗？
|
||||
|
||||
A: 需要修改 server.js 中的 BASE_URL 和相应的解析函数，因为不同网站的 HTML 结构不同。
|
||||
|
||||
### Q: 需要安装数据库吗？
|
||||
|
||||
A: **不需要！** 本项目采用无数据库架构，只需安装 Node.js 依赖即可运行。所有配置保存在 config.json 文件中，报告通过邮件发送。
|
||||
|
||||
### Q: 定时任务配置后不生效怎么办？
|
||||
|
||||
A: 请检查以下几点：
|
||||
1. `config.json` 中 `scheduler.enabled` 是否为 `true`
|
||||
2. 邮件配置是否正确（SMTP 服务器、用户名、密码）
|
||||
3. Node.js 进程是否持续运行（使用 PM2 查看状态）
|
||||
4. 查看服务器日志是否有错误信息
|
||||
|
||||
### Q: 如何修改定时任务的执行时间？
|
||||
|
||||
A: 有两种方式：
|
||||
1. **Web 界面**（推荐）：访问网页，切换到"定时任务"标签，选择预设时间或自定义 Cron 表达式，点击保存
|
||||
2. **配置文件**：编辑 `config.json` 中的 `scheduler.cronTime` 字段
|
||||
|
||||
### Q: 邮件发送失败怎么办？
|
||||
|
||||
A: 常见原因：
|
||||
1. **QQ 邮箱**：需要使用授权码，不是登录密码（在 QQ 邮箱设置 → 账户 → POP3/SMTP 服务中生成）
|
||||
2. **端口问题**：QQ 邮箱使用 587，某些邮箱可能需要 465（SSL）
|
||||
3. **防火墙**：确保服务器可以访问 SMTP 端口
|
||||
4. **测试连接**：在"邮件配置"标签中使用"测试连接"功能验证配置
|
||||
|
||||
### Q: 如何查看定时任务运行日志？
|
||||
|
||||
A: 如果使用 PM2 部署：
|
||||
```bash
|
||||
pm2 logs gjzx-scraper
|
||||
```
|
||||
|
||||
如果使用 systemd：
|
||||
```bash
|
||||
sudo journalctl -u gjzx-scraper -f
|
||||
```
|
||||
|
||||
### Q: 服务器重启后定时任务还会运行吗？
|
||||
|
||||
A: 需要设置开机自启：
|
||||
- **PM2 方式**：执行 `pm2 startup` 和 `pm2 save`
|
||||
- **systemd 方式**：执行 `sudo systemctl enable gjzx-scraper`
|
||||
|
||||
### Q: 可以设置多个定时任务吗？
|
||||
|
||||
A: 当前版本只支持一个定时任务。如需多个任务，可以：
|
||||
1. 使用 Cron 表达式设置多个时间点（如 `0 9,18 * * *` 表示每天 9:00 和 18:00）
|
||||
2. 部署多个实例，使用不同的配置文件和端口
|
||||
|
||||
## 更新日志
|
||||
|
||||
### v1.1.0 (2025-12-15)
|
||||
|
||||
- ✨ 新增定时任务功能（node-cron）
|
||||
- ✨ 新增邮件推送服务（nodemailer）
|
||||
- ✨ 新增 Web 配置界面（定时任务 + 邮件配置）
|
||||
- ✨ 支持多时间范围（今日/本周/本月）
|
||||
- ✨ 配置 API（读取/更新配置）
|
||||
- ✨ 定时任务状态查询
|
||||
- ✨ 手动触发任务功能
|
||||
- 📝 完善文档，新增部署指南
|
||||
|
||||
### v1.0.0 (2025-12-12)
|
||||
|
||||
- Web 可视化界面
|
||||
- 支持按时间范围采集
|
||||
- 支持分页浏览
|
||||
- 支持导出 Word/Markdown 报告
|
||||
- RESTful API 接口
|
||||
- 自动编码识别
|
||||
- 智能金额提取
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
- 旧版 `public/` 页面已不再作为当前主前端入口
|
||||
- 当前主前端以 `client/` 目录为准
|
||||
|
||||
12
client/index.html
Normal file
12
client/index.html
Normal file
@@ -0,0 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>公告抓取与分析工具</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script type="module" src="/src/main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
10
client/src/App.vue
Normal file
10
client/src/App.vue
Normal file
@@ -0,0 +1,10 @@
|
||||
<script setup>
|
||||
import zhCn from 'element-plus/es/locale/lang/zh-cn';
|
||||
import AppShell from './components/AppShell.vue';
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<el-config-provider :locale="zhCn">
|
||||
<AppShell />
|
||||
</el-config-provider>
|
||||
</template>
|
||||
24
client/src/api/http.js
Normal file
24
client/src/api/http.js
Normal file
@@ -0,0 +1,24 @@
|
||||
import axios from 'axios';
|
||||
|
||||
const http = axios.create({
|
||||
baseURL: '/api',
|
||||
timeout: 30000,
|
||||
});
|
||||
|
||||
http.interceptors.response.use(
|
||||
(response) => {
|
||||
const payload = response.data;
|
||||
|
||||
if (payload && payload.success === false) {
|
||||
return Promise.reject(new Error(payload.error || '请求失败'));
|
||||
}
|
||||
|
||||
return payload;
|
||||
},
|
||||
(error) => {
|
||||
const message = error.response?.data?.error || error.message || '请求失败';
|
||||
return Promise.reject(new Error(message));
|
||||
},
|
||||
);
|
||||
|
||||
export default http;
|
||||
25
client/src/api/results.js
Normal file
25
client/src/api/results.js
Normal file
@@ -0,0 +1,25 @@
|
||||
import http from './http';
|
||||
|
||||
export function fetchResults(params = {}) {
|
||||
return http.get('/results', { params }).then((payload) => payload);
|
||||
}
|
||||
|
||||
export function fetchResultFilters(params = {}) {
|
||||
return http.get('/results/filters', { params }).then((payload) => payload.data || {});
|
||||
}
|
||||
|
||||
export function deleteResult(id) {
|
||||
return http.delete(`/results/${id}`);
|
||||
}
|
||||
|
||||
export function clearResults() {
|
||||
return http.delete('/results');
|
||||
}
|
||||
|
||||
export function fetchProjects(params = {}) {
|
||||
return http.get('/projects', { params }).then((payload) => payload);
|
||||
}
|
||||
|
||||
export function fetchProjectFilters() {
|
||||
return http.get('/projects/filters').then((payload) => payload.data || {});
|
||||
}
|
||||
17
client/src/api/settings.js
Normal file
17
client/src/api/settings.js
Normal file
@@ -0,0 +1,17 @@
|
||||
import http from './http';
|
||||
|
||||
export function fetchConfig() {
|
||||
return http.get('/config').then((payload) => payload.data || {});
|
||||
}
|
||||
|
||||
export function saveConfig(payload) {
|
||||
return http.post('/config', payload);
|
||||
}
|
||||
|
||||
export function fetchSchedulerStatus() {
|
||||
return http.get('/scheduler/status').then((payload) => payload.data || {});
|
||||
}
|
||||
|
||||
export function triggerScheduledTask() {
|
||||
return http.post('/run-scheduled-task');
|
||||
}
|
||||
34
client/src/api/tasks.js
Normal file
34
client/src/api/tasks.js
Normal file
@@ -0,0 +1,34 @@
|
||||
import http from './http';
|
||||
|
||||
export function fetchTasks() {
|
||||
return http.get('/tasks').then((payload) => payload.data || []);
|
||||
}
|
||||
|
||||
export function createTask(payload) {
|
||||
return http.post('/tasks', payload).then((response) => response.data);
|
||||
}
|
||||
|
||||
export function updateTask(id, payload) {
|
||||
return http.put(`/tasks/${id}`, payload).then((response) => response.data);
|
||||
}
|
||||
|
||||
export function deleteTask(id) {
|
||||
return http.delete(`/tasks/${id}`);
|
||||
}
|
||||
|
||||
export function runTask(id) {
|
||||
return http.post(`/tasks/${id}/run`);
|
||||
}
|
||||
|
||||
export function runAllTasks(ids = []) {
|
||||
const body = ids.length ? { ids } : {};
|
||||
return http.post('/tasks/run', body);
|
||||
}
|
||||
|
||||
export function toggleTask(id, enabled) {
|
||||
return http.put(`/tasks/${id}`, { enabled }).then((response) => response.data);
|
||||
}
|
||||
|
||||
export function fetchTaskStatus() {
|
||||
return http.get('/tasks/status').then((payload) => payload.data || { isRunning: false });
|
||||
}
|
||||
41
client/src/components/AppShell.vue
Normal file
41
client/src/components/AppShell.vue
Normal file
@@ -0,0 +1,41 @@
|
||||
<script setup>
|
||||
import { computed } from 'vue';
|
||||
import { useRoute } from 'vue-router';
|
||||
|
||||
const route = useRoute();
|
||||
|
||||
const subtitle = computed(() => route.meta?.subtitle || '把抓取任务、分析结果和调度配置收在同一套前端工作台里。');
|
||||
|
||||
const menuItems = [
|
||||
{ index: '/tasks', label: '任务配置' },
|
||||
{ index: '/results', label: '抓取结果' },
|
||||
{ index: '/projects', label: '项目管理' },
|
||||
{ index: '/settings', label: '系统设置' },
|
||||
];
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div class="app-shell">
|
||||
<div class="app-frame">
|
||||
<header class="app-topbar">
|
||||
<div class="app-brand">
|
||||
<div class="app-brand__mark">公</div>
|
||||
<div class="app-brand__text">
|
||||
<h1>公告抓取与分析工具</h1>
|
||||
<p>{{ subtitle }}</p>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<el-menu :default-active="route.path" class="app-menu" mode="horizontal" router>
|
||||
<el-menu-item v-for="item in menuItems" :key="item.index" :index="item.index">
|
||||
{{ item.label }}
|
||||
</el-menu-item>
|
||||
</el-menu>
|
||||
|
||||
<main class="app-main">
|
||||
<RouterView />
|
||||
</main>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
55
client/src/components/ResultRecordCard.vue
Normal file
55
client/src/components/ResultRecordCard.vue
Normal file
@@ -0,0 +1,55 @@
|
||||
<script setup>
|
||||
import { computed } from 'vue';
|
||||
import { formatDateTime, formatYuan, pickResultLink, pickResultName } from '@/utils/format';
|
||||
|
||||
const props = defineProps({
|
||||
record: {
|
||||
type: Object,
|
||||
required: true,
|
||||
},
|
||||
});
|
||||
|
||||
const emit = defineEmits(['delete']);
|
||||
|
||||
const items = computed(() => props.record.data?.results || []);
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<el-card shadow="hover" class="result-card-plus">
|
||||
<template #header>
|
||||
<div class="result-card-plus__header">
|
||||
<div>
|
||||
<div class="result-card-plus__title">{{ record.city || '未命名城市' }}</div>
|
||||
<div class="result-card-plus__meta">
|
||||
<el-tag size="small" type="primary">{{ items.length }} 条结果</el-tag>
|
||||
<el-tag v-if="record.error" size="small" type="danger">执行失败</el-tag>
|
||||
</div>
|
||||
</div>
|
||||
<div class="result-card-plus__time">{{ formatDateTime(record.scrapedAt) }}</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<el-alert v-if="record.error" :title="`错误信息：${record.error}`" type="error" :closable="false" show-icon />
|
||||
|
||||
<div v-else-if="items.length" class="result-card-plus__list">
|
||||
<div v-for="(item, index) in items" :key="`${record.id}-${index}`" class="result-card-plus__item">
|
||||
<div class="result-card-plus__type">{{ item.type || '-' }}</div>
|
||||
<div class="result-card-plus__name">{{ pickResultName(item) }}</div>
|
||||
<div class="result-card-plus__amount">{{ formatYuan(item.amount_yuan) }}</div>
|
||||
<div class="result-card-plus__date">{{ item.date || '-' }}</div>
|
||||
<div class="result-card-plus__action">
|
||||
<el-link v-if="pickResultLink(item)" :href="pickResultLink(item)" target="_blank" type="primary">查看详情</el-link>
|
||||
<span v-else>-</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<el-empty v-else description="当前记录里没有可展示的数据" :image-size="88" />
|
||||
|
||||
<template #footer>
|
||||
<div class="result-card-plus__footer">
|
||||
<el-button type="danger" plain @click="emit('delete', record)">删除记录</el-button>
|
||||
</div>
|
||||
</template>
|
||||
</el-card>
|
||||
</template>
|
||||
105
client/src/components/TaskDialog.vue
Normal file
105
client/src/components/TaskDialog.vue
Normal file
@@ -0,0 +1,105 @@
|
||||
<script setup>
|
||||
import { computed, reactive, watch } from 'vue';
|
||||
import { TASK_MODE_OPTIONS } from '@/constants/taskModes';
|
||||
|
||||
const props = defineProps({
|
||||
modelValue: {
|
||||
type: Boolean,
|
||||
default: false,
|
||||
},
|
||||
task: {
|
||||
type: Object,
|
||||
default: null,
|
||||
},
|
||||
});
|
||||
|
||||
const emit = defineEmits(['update:modelValue', 'submit']);
|
||||
|
||||
const form = reactive({
|
||||
city: '',
|
||||
plateName: '',
|
||||
prompt: '',
|
||||
mode: TASK_MODE_OPTIONS[0],
|
||||
enabled: true,
|
||||
});
|
||||
|
||||
const dialogTitle = computed(() => (props.task?.id ? '编辑任务' : '新增任务'));
|
||||
const modeOptions = computed(() => {
|
||||
const set = new Set(TASK_MODE_OPTIONS);
|
||||
if (props.task?.mode) set.add(props.task.mode);
|
||||
return [...set];
|
||||
});
|
||||
|
||||
watch(
|
||||
() => [props.modelValue, props.task],
|
||||
() => {
|
||||
form.city = props.task?.city || '';
|
||||
form.plateName = props.task?.plateName || '';
|
||||
form.prompt = props.task?.prompt || '';
|
||||
form.mode = props.task?.mode || TASK_MODE_OPTIONS[0];
|
||||
form.enabled = props.task?.enabled ?? true;
|
||||
},
|
||||
{ immediate: true },
|
||||
);
|
||||
|
||||
function close() {
|
||||
emit('update:modelValue', false);
|
||||
}
|
||||
|
||||
function submit() {
|
||||
emit('submit', {
|
||||
city: form.city.trim(),
|
||||
plateName: form.plateName.trim(),
|
||||
prompt: form.prompt.trim(),
|
||||
mode: form.mode.trim() || TASK_MODE_OPTIONS[0],
|
||||
enabled: form.enabled,
|
||||
});
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<el-dialog :model-value="modelValue" :title="dialogTitle" width="760px" destroy-on-close @close="close">
|
||||
<el-form label-position="top">
|
||||
<el-form-item label="城市名称">
|
||||
<el-input v-model="form.city" placeholder="如：南京市" />
|
||||
</el-form-item>
|
||||
|
||||
<el-form-item label="板块名称">
|
||||
<el-input v-model="form.plateName" placeholder="如：工程" />
|
||||
</el-form-item>
|
||||
|
||||
<el-form-item label="提示词">
|
||||
<el-input
|
||||
v-model="form.prompt"
|
||||
type="textarea"
|
||||
:autosize="{ minRows: 6, maxRows: 12 }"
|
||||
placeholder="请访问目标网站，抓取今天的招标公告和中标公告信息……"
|
||||
/>
|
||||
</el-form-item>
|
||||
|
||||
<el-row :gutter="16">
|
||||
<el-col :xs="24" :sm="12">
|
||||
<el-form-item label="模型">
|
||||
<el-select v-model="form.mode" placeholder="选择模型" style="width: 100%;">
|
||||
<el-option v-for="option in modeOptions" :key="option" :label="option" :value="option" />
|
||||
</el-select>
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12">
|
||||
<el-form-item label="状态">
|
||||
<el-switch v-model="form.enabled" inline-prompt active-text="启用" inactive-text="禁用" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
</el-row>
|
||||
</el-form>
|
||||
|
||||
<el-alert title="建议把目标网址、筛选规则和字段要求都写进提示词里，后续维护会更稳定。" type="info" :closable="false" show-icon />
|
||||
|
||||
<template #footer>
|
||||
<div class="dialog-footer">
|
||||
<el-button @click="close">取消</el-button>
|
||||
<el-button type="primary" @click="submit">保存任务</el-button>
|
||||
</div>
|
||||
</template>
|
||||
</el-dialog>
|
||||
</template>
|
||||
10
client/src/constants/taskModes.js
Normal file
10
client/src/constants/taskModes.js
Normal file
@@ -0,0 +1,10 @@
|
||||
export const TASK_MODE_OPTIONS = [
|
||||
'qwen3.5-plus',
|
||||
'qwen3-max-2026-01-23',
|
||||
'qwen3-coder-next',
|
||||
'qwen3-coder-plus',
|
||||
'glm-5',
|
||||
'glm-4.7',
|
||||
'kimi-k2.5',
|
||||
'MiniMax-M2.5',
|
||||
];
|
||||
6
client/src/main.js
Normal file
6
client/src/main.js
Normal file
@@ -0,0 +1,6 @@
|
||||
import { createApp } from 'vue';
|
||||
import App from './App.vue';
|
||||
import router from './router';
|
||||
import './styles.css';
|
||||
|
||||
createApp(App).use(router).mount('#app');
|
||||
233
client/src/pages/ProjectsPage.vue
Normal file
233
client/src/pages/ProjectsPage.vue
Normal file
@@ -0,0 +1,233 @@
|
||||
<script setup>
|
||||
import { onMounted, reactive, ref } from 'vue';
|
||||
import { ElMessage } from 'element-plus';
|
||||
import { fetchProjectFilters, fetchProjects } from '@/api/results';
|
||||
import { formatYuan } from '@/utils/format';
|
||||
|
||||
const form = reactive({
|
||||
city: '',
|
||||
section: '',
|
||||
projectName: '',
|
||||
minAmount: '',
|
||||
maxAmount: '',
|
||||
startDate: '',
|
||||
endDate: '',
|
||||
});
|
||||
|
||||
const activeFilters = reactive({
|
||||
city: '',
|
||||
section: '',
|
||||
projectName: '',
|
||||
minAmount: '',
|
||||
maxAmount: '',
|
||||
startDate: '',
|
||||
endDate: '',
|
||||
});
|
||||
|
||||
const options = reactive({
|
||||
cities: [],
|
||||
sections: [],
|
||||
});
|
||||
|
||||
const state = reactive({
|
||||
records: [],
|
||||
page: 1,
|
||||
pageSize: 10,
|
||||
total: 0,
|
||||
loading: true,
|
||||
});
|
||||
|
||||
const summaryText = ref('加载中…');
|
||||
|
||||
onMounted(async () => {
|
||||
await Promise.all([loadFilters(), loadProjects()]);
|
||||
});
|
||||
|
||||
async function loadFilters() {
|
||||
try {
|
||||
const data = await fetchProjectFilters();
|
||||
options.cities = data.cities || [];
|
||||
options.sections = data.sections || [];
|
||||
} catch (error) {
|
||||
ElMessage.error(`项目筛选项加载失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
function validateForm() {
|
||||
if (form.minAmount && form.maxAmount && Number(form.minAmount) > Number(form.maxAmount)) {
|
||||
throw new Error('最小金额不能大于最大金额');
|
||||
}
|
||||
|
||||
if (form.startDate && form.endDate && form.startDate > form.endDate) {
|
||||
throw new Error('开始日期不能晚于结束日期');
|
||||
}
|
||||
}
|
||||
|
||||
function syncActiveFilters() {
|
||||
Object.assign(activeFilters, form);
|
||||
}
|
||||
|
||||
async function loadProjects() {
|
||||
state.loading = true;
|
||||
|
||||
try {
|
||||
const payload = await fetchProjects({
|
||||
...activeFilters,
|
||||
page: state.page,
|
||||
pageSize: state.pageSize,
|
||||
});
|
||||
|
||||
state.records = payload.data || [];
|
||||
state.total = payload.total || 0;
|
||||
state.page = payload.page || 1;
|
||||
|
||||
if (!state.records.length) {
|
||||
summaryText.value = '当前筛选条件下没有项目';
|
||||
return;
|
||||
}
|
||||
|
||||
const startIndex = (state.page - 1) * state.pageSize + 1;
|
||||
const endIndex = Math.min(state.page * state.pageSize, state.total);
|
||||
summaryText.value = `显示第 ${startIndex} 至 ${endIndex} 条记录，共 ${state.total} 条`;
|
||||
} catch (error) {
|
||||
state.records = [];
|
||||
state.total = 0;
|
||||
summaryText.value = `项目加载失败：${error.message}`;
|
||||
ElMessage.error(`项目加载失败：${error.message}`);
|
||||
} finally {
|
||||
state.loading = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function search() {
|
||||
try {
|
||||
validateForm();
|
||||
syncActiveFilters();
|
||||
state.page = 1;
|
||||
await loadProjects();
|
||||
} catch (error) {
|
||||
ElMessage.error(error.message);
|
||||
}
|
||||
}
|
||||
|
||||
async function reset() {
|
||||
Object.keys(form).forEach((key) => {
|
||||
form[key] = '';
|
||||
activeFilters[key] = '';
|
||||
});
|
||||
state.page = 1;
|
||||
await loadProjects();
|
||||
}
|
||||
|
||||
async function changePage(page) {
|
||||
state.page = page;
|
||||
await loadProjects();
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<section class="page-grid">
|
||||
<header class="page-head">
|
||||
<div>
|
||||
<h2>项目管理</h2>
|
||||
<p>用 Element Plus 表单和表格组织项目去重查询，适合继续扩展导出、排序和列配置。</p>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<el-card shadow="never">
|
||||
<el-form label-position="top">
|
||||
<el-row :gutter="16">
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-form-item label="城市">
|
||||
<el-select v-model="form.city" placeholder="全部城市" clearable style="width: 100%;">
|
||||
<el-option v-for="item in options.cities" :key="item" :label="item" :value="item" />
|
||||
</el-select>
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-form-item label="板块">
|
||||
<el-select v-model="form.section" placeholder="全部板块" clearable style="width: 100%;">
|
||||
<el-option v-for="item in options.sections" :key="item" :label="item" :value="item" />
|
||||
</el-select>
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-form-item label="项目名称">
|
||||
<el-input v-model="form.projectName" placeholder="输入项目名关键字" clearable />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-form-item label="最小金额">
|
||||
<el-input-number v-model="form.minAmount" :min="0" :precision="2" :controls="false" style="width: 100%;" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-form-item label="最大金额">
|
||||
<el-input-number v-model="form.maxAmount" :min="0" :precision="2" :controls="false" style="width: 100%;" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-form-item label="开始日期">
|
||||
<el-date-picker v-model="form.startDate" type="date" value-format="YYYY-MM-DD" placeholder="选择日期" style="width: 100%;" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-form-item label="结束日期">
|
||||
<el-date-picker v-model="form.endDate" type="date" value-format="YYYY-MM-DD" placeholder="选择日期" style="width: 100%;" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
</el-row>
|
||||
</el-form>
|
||||
|
||||
<div class="toolbar">
|
||||
<div class="summary-text">按项目名称去重后展示，方便快速判断有哪些有效项目。</div>
|
||||
<div class="toolbar__group">
|
||||
<el-button @click="reset">重置</el-button>
|
||||
<el-button type="primary" @click="search">查询项目</el-button>
|
||||
</div>
|
||||
</div>
|
||||
</el-card>
|
||||
|
||||
<el-card shadow="never">
|
||||
<template #header>
|
||||
<div class="toolbar">
|
||||
<span>{{ summaryText }}</span>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<el-table v-loading="state.loading" :data="state.records" style="width: 100%;" empty-text="暂无项目数据">
|
||||
<el-table-column prop="city" label="城市" min-width="120" />
|
||||
<el-table-column label="板块" min-width="140">
|
||||
<template #default="{ row }">
|
||||
{{ row.section || row.type || '-' }}
|
||||
</template>
|
||||
</el-table-column>
|
||||
<el-table-column prop="projectName" label="项目名称" min-width="280" show-overflow-tooltip />
|
||||
<el-table-column label="金额" min-width="150">
|
||||
<template #default="{ row }">
|
||||
{{ formatYuan(row.amountYuan) }}
|
||||
</template>
|
||||
</el-table-column>
|
||||
<el-table-column prop="date" label="发布日期" min-width="140" />
|
||||
<el-table-column label="详情" min-width="120" fixed="right">
|
||||
<template #default="{ row }">
|
||||
<el-link v-if="row.detailLink" :href="row.detailLink" type="primary" target="_blank">查看详情</el-link>
|
||||
<span v-else>-</span>
|
||||
</template>
|
||||
</el-table-column>
|
||||
</el-table>
|
||||
|
||||
<div class="pagination-wrap">
|
||||
<el-pagination
|
||||
v-if="state.total"
|
||||
background
|
||||
layout="total, prev, pager, next"
|
||||
:current-page="state.page"
|
||||
:page-size="state.pageSize"
|
||||
:total="state.total"
|
||||
@current-change="changePage"
|
||||
/>
|
||||
</div>
|
||||
</el-card>
|
||||
</section>
|
||||
</template>
|
||||
200
client/src/pages/ResultsPage.vue
Normal file
200
client/src/pages/ResultsPage.vue
Normal file
@@ -0,0 +1,200 @@
|
||||
<script setup>
|
||||
import { computed, onMounted, reactive } from 'vue';
|
||||
import { ElMessage, ElMessageBox } from 'element-plus';
|
||||
import ResultRecordCard from '@/components/ResultRecordCard.vue';
|
||||
import { deleteResult, fetchResultFilters, fetchResults } from '@/api/results';
|
||||
|
||||
const filters = reactive({
|
||||
city: '',
|
||||
section: '',
|
||||
type: '',
|
||||
});
|
||||
|
||||
const options = reactive({
|
||||
cities: [],
|
||||
sections: [],
|
||||
types: [],
|
||||
});
|
||||
|
||||
const stats = reactive({
|
||||
total: 0,
|
||||
success: 0,
|
||||
failed: 0,
|
||||
cities: 0,
|
||||
});
|
||||
|
||||
const state = reactive({
|
||||
records: [],
|
||||
page: 1,
|
||||
pageSize: 10,
|
||||
total: 0,
|
||||
loading: true,
|
||||
});
|
||||
|
||||
const hasRecords = computed(() => state.records.length > 0);
|
||||
|
||||
onMounted(async () => {
|
||||
await refreshAll();
|
||||
});
|
||||
|
||||
async function loadFilters() {
|
||||
const data = await fetchResultFilters();
|
||||
options.cities = data.cities || [];
|
||||
options.sections = data.sections || [];
|
||||
options.types = data.types || [];
|
||||
}
|
||||
|
||||
async function loadStats() {
|
||||
const payload = await fetchResults({ page: 1, pageSize: 500 });
|
||||
const items = payload.data || [];
|
||||
stats.total = payload.total || 0;
|
||||
stats.success = items.filter((item) => !item.error).length;
|
||||
stats.failed = items.filter((item) => item.error).length;
|
||||
stats.cities = new Set(items.map((item) => item.city).filter(Boolean)).size;
|
||||
}
|
||||
|
||||
async function loadRecords() {
|
||||
state.loading = true;
|
||||
|
||||
try {
|
||||
const payload = await fetchResults({
|
||||
page: state.page,
|
||||
pageSize: state.pageSize,
|
||||
city: filters.city || undefined,
|
||||
section: filters.section || undefined,
|
||||
type: filters.type || undefined,
|
||||
});
|
||||
|
||||
state.records = payload.data || [];
|
||||
state.total = payload.total || 0;
|
||||
state.page = payload.page || 1;
|
||||
} catch (error) {
|
||||
ElMessage.error(`结果加载失败：${error.message}`);
|
||||
state.records = [];
|
||||
state.total = 0;
|
||||
} finally {
|
||||
state.loading = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function refreshAll() {
|
||||
try {
|
||||
await Promise.all([loadFilters(), loadStats(), loadRecords()]);
|
||||
} catch (error) {
|
||||
ElMessage.error(`页面刷新失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function applyFilters() {
|
||||
state.page = 1;
|
||||
await loadRecords();
|
||||
}
|
||||
|
||||
async function changePage(page) {
|
||||
state.page = page;
|
||||
await loadRecords();
|
||||
}
|
||||
|
||||
async function handleDelete(record) {
|
||||
try {
|
||||
await ElMessageBox.confirm('确定要删除这条抓取记录吗？', '删除确认', {
|
||||
type: 'warning',
|
||||
});
|
||||
await deleteResult(record.id);
|
||||
ElMessage.success('抓取记录已删除');
|
||||
await refreshAll();
|
||||
} catch (error) {
|
||||
if (error !== 'cancel' && error !== 'close') {
|
||||
ElMessage.error(`删除失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function resetFilters() {
|
||||
filters.city = '';
|
||||
filters.section = '';
|
||||
filters.type = '';
|
||||
applyFilters();
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<section class="page-grid">
|
||||
<header class="page-head">
|
||||
<div>
|
||||
<h2>抓取结果</h2>
|
||||
<p>结果页已经切到 Element Plus 组件风格，适合继续加更复杂的筛选、导出和批量操作。</p>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<el-row :gutter="16">
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-statistic title="总记录数" :value="stats.total" />
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-statistic title="成功条数" :value="stats.success" />
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-statistic title="失败条数" :value="stats.failed" />
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12" :lg="6">
|
||||
<el-statistic title="来源城市" :value="stats.cities" />
|
||||
</el-col>
|
||||
</el-row>
|
||||
|
||||
<el-card shadow="never">
|
||||
<el-form label-position="top">
|
||||
<el-row :gutter="16">
|
||||
<el-col :xs="24" :sm="8">
|
||||
<el-form-item label="城市">
|
||||
<el-select v-model="filters.city" placeholder="全部城市" clearable style="width: 100%;" @change="applyFilters">
|
||||
<el-option v-for="item in options.cities" :key="item" :label="item" :value="item" />
|
||||
</el-select>
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="8">
|
||||
<el-form-item label="板块">
|
||||
<el-select v-model="filters.section" placeholder="全部板块" clearable style="width: 100%;" @change="applyFilters">
|
||||
<el-option v-for="item in options.sections" :key="item" :label="item" :value="item" />
|
||||
</el-select>
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="8">
|
||||
<el-form-item label="类型">
|
||||
<el-select v-model="filters.type" placeholder="全部类型" clearable style="width: 100%;" @change="applyFilters">
|
||||
<el-option v-for="item in options.types" :key="item" :label="item" :value="item" />
|
||||
</el-select>
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
</el-row>
|
||||
</el-form>
|
||||
|
||||
<div class="toolbar">
|
||||
<div class="summary-text">当前页展示 {{ state.records.length }} 条记录。</div>
|
||||
<div class="toolbar__group">
|
||||
<el-button @click="resetFilters">重置筛选</el-button>
|
||||
<el-button type="primary" @click="refreshAll">刷新数据</el-button>
|
||||
</div>
|
||||
</div>
|
||||
</el-card>
|
||||
|
||||
<div v-loading="state.loading" class="page-grid">
|
||||
<template v-if="hasRecords">
|
||||
<ResultRecordCard v-for="item in state.records" :key="item.id" :record="item" @delete="handleDelete" />
|
||||
</template>
|
||||
<el-empty v-else description="暂无抓取结果" :image-size="96" />
|
||||
</div>
|
||||
|
||||
<div class="pagination-wrap">
|
||||
<el-pagination
|
||||
v-if="state.total"
|
||||
background
|
||||
layout="total, prev, pager, next"
|
||||
:current-page="state.page"
|
||||
:page-size="state.pageSize"
|
||||
:total="state.total"
|
||||
@current-change="changePage"
|
||||
/>
|
||||
</div>
|
||||
</section>
|
||||
</template>
|
||||
228
client/src/pages/SettingsPage.vue
Normal file
228
client/src/pages/SettingsPage.vue
Normal file
@@ -0,0 +1,228 @@
|
||||
<script setup>
|
||||
import { onMounted, reactive, ref } from 'vue';
|
||||
import { ElMessage, ElMessageBox } from 'element-plus';
|
||||
import { fetchConfig, fetchSchedulerStatus, saveConfig, triggerScheduledTask } from '@/api/settings';
|
||||
|
||||
const loading = ref(true);
|
||||
const saveStatus = ref('');
|
||||
const schedulerText = ref('未获取');
|
||||
|
||||
const form = reactive({
|
||||
agent: {
|
||||
baseUrl: '',
|
||||
pollInterval: 3000,
|
||||
timeout: 300000,
|
||||
},
|
||||
scheduler: {
|
||||
enabled: false,
|
||||
cronTime: '0 9 * * *',
|
||||
description: '',
|
||||
},
|
||||
email: {
|
||||
smtpHost: '',
|
||||
smtpPort: 587,
|
||||
smtpUser: '',
|
||||
smtpPass: '',
|
||||
recipients: '',
|
||||
},
|
||||
});
|
||||
|
||||
onMounted(async () => {
|
||||
await refresh();
|
||||
});
|
||||
|
||||
async function refresh() {
|
||||
loading.value = true;
|
||||
|
||||
try {
|
||||
const [config, schedulerStatus] = await Promise.all([fetchConfig(), fetchSchedulerStatus()]);
|
||||
|
||||
form.agent.baseUrl = config.agent?.baseUrl || '';
|
||||
form.agent.pollInterval = config.agent?.pollInterval || 3000;
|
||||
form.agent.timeout = config.agent?.timeout || 300000;
|
||||
|
||||
form.scheduler.enabled = config.scheduler?.enabled ?? false;
|
||||
form.scheduler.cronTime = config.scheduler?.cronTime || '0 9 * * *';
|
||||
form.scheduler.description = config.scheduler?.description || '';
|
||||
|
||||
form.email.smtpHost = config.email?.smtpHost || '';
|
||||
form.email.smtpPort = config.email?.smtpPort || 587;
|
||||
form.email.smtpUser = config.email?.smtpUser || '';
|
||||
form.email.smtpPass = config.email?.smtpPass || '';
|
||||
form.email.recipients = config.email?.recipients || '';
|
||||
|
||||
schedulerText.value = schedulerStatus.isRunning
|
||||
? `运行中（启用任务 ${schedulerStatus.enabledTasks || 0} 个）`
|
||||
: `未运行（启用任务 ${schedulerStatus.enabledTasks || 0} 个）`;
|
||||
} catch (error) {
|
||||
ElMessage.error(`设置加载失败：${error.message}`);
|
||||
} finally {
|
||||
loading.value = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function handleSave() {
|
||||
try {
|
||||
await saveConfig({
|
||||
agent: {
|
||||
baseUrl: form.agent.baseUrl.trim(),
|
||||
pollInterval: Number(form.agent.pollInterval) || 3000,
|
||||
timeout: Number(form.agent.timeout) || 300000,
|
||||
},
|
||||
scheduler: {
|
||||
enabled: Boolean(form.scheduler.enabled),
|
||||
cronTime: form.scheduler.cronTime.trim() || '0 9 * * *',
|
||||
description: form.scheduler.description.trim(),
|
||||
},
|
||||
email: {
|
||||
smtpHost: form.email.smtpHost.trim(),
|
||||
smtpPort: Number(form.email.smtpPort) || 587,
|
||||
smtpUser: form.email.smtpUser.trim(),
|
||||
smtpPass: form.email.smtpPass.trim(),
|
||||
recipients: form.email.recipients.trim(),
|
||||
},
|
||||
});
|
||||
|
||||
saveStatus.value = '设置已保存';
|
||||
ElMessage.success('系统设置已保存');
|
||||
await refresh();
|
||||
setTimeout(() => {
|
||||
saveStatus.value = '';
|
||||
}, 3000);
|
||||
} catch (error) {
|
||||
saveStatus.value = '';
|
||||
ElMessage.error(`保存失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function handleTriggerScheduler() {
|
||||
try {
|
||||
await ElMessageBox.confirm('确定要立即执行一次定时任务吗？', '执行确认', {
|
||||
type: 'warning',
|
||||
});
|
||||
const payload = await triggerScheduledTask();
|
||||
ElMessage.success(payload.message || '定时任务已在后台触发');
|
||||
await refresh();
|
||||
} catch (error) {
|
||||
if (error !== 'cancel' && error !== 'close') {
|
||||
ElMessage.error(`触发失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<section class="page-grid">
|
||||
<header class="page-head">
|
||||
<div>
|
||||
<h2>系统设置</h2>
|
||||
<p>这一页已经改用 Element Plus 表单和卡片来管理 Agent、定时任务和邮件配置。</p>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div v-loading="loading" class="page-grid">
|
||||
<el-card shadow="never">
|
||||
<template #header>
|
||||
<span>Agent 服务配置</span>
|
||||
</template>
|
||||
|
||||
<el-form label-position="top">
|
||||
<el-row :gutter="16">
|
||||
<el-col :xs="24" :sm="24">
|
||||
<el-form-item label="Agent 服务地址">
|
||||
<el-input v-model="form.agent.baseUrl" placeholder="http://127.0.0.1:18625" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12">
|
||||
<el-form-item label="轮询间隔（毫秒）">
|
||||
<el-input-number v-model="form.agent.pollInterval" :min="1000" :step="100" style="width: 100%;" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12">
|
||||
<el-form-item label="超时时间（毫秒）">
|
||||
<el-input-number v-model="form.agent.timeout" :min="1000" :step="1000" style="width: 100%;" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
</el-row>
|
||||
</el-form>
|
||||
</el-card>
|
||||
|
||||
<el-card shadow="never">
|
||||
<template #header>
|
||||
<div class="toolbar">
|
||||
<span>定时任务</span>
|
||||
<el-tag type="info">{{ schedulerText }}</el-tag>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<el-form label-position="top">
|
||||
<el-row :gutter="16">
|
||||
<el-col :xs="24" :sm="8">
|
||||
<el-form-item label="是否启用">
|
||||
<el-switch v-model="form.scheduler.enabled" inline-prompt active-text="启用" inactive-text="禁用" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="8">
|
||||
<el-form-item label="Cron 表达式">
|
||||
<el-input v-model="form.scheduler.cronTime" placeholder="0 9 * * *" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="8">
|
||||
<el-form-item label="描述">
|
||||
<el-input v-model="form.scheduler.description" placeholder="每天 9 点执行" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
</el-row>
|
||||
</el-form>
|
||||
|
||||
<el-button @click="handleTriggerScheduler">立即执行一次</el-button>
|
||||
</el-card>
|
||||
|
||||
<el-card shadow="never">
|
||||
<template #header>
|
||||
<span>邮件配置</span>
|
||||
</template>
|
||||
|
||||
<el-form label-position="top">
|
||||
<el-row :gutter="16">
|
||||
<el-col :xs="24" :sm="12">
|
||||
<el-form-item label="SMTP 服务地址">
|
||||
<el-input v-model="form.email.smtpHost" placeholder="smtp.qq.com" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12">
|
||||
<el-form-item label="端口">
|
||||
<el-input-number v-model="form.email.smtpPort" :min="1" style="width: 100%;" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12">
|
||||
<el-form-item label="用户名">
|
||||
<el-input v-model="form.email.smtpUser" placeholder="your-email@example.com" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12">
|
||||
<el-form-item label="密码或授权码">
|
||||
<el-input v-model="form.email.smtpPass" type="password" show-password placeholder="留空会沿用已保存的掩码值" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="24">
|
||||
<el-form-item label="收件人">
|
||||
<el-input v-model="form.email.recipients" placeholder="a@example.com, b@example.com" />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
</el-row>
|
||||
</el-form>
|
||||
</el-card>
|
||||
|
||||
<el-card shadow="never">
|
||||
<div class="toolbar">
|
||||
<div class="summary-text">保存后会自动刷新定时任务配置。</div>
|
||||
<div class="toolbar__group">
|
||||
<span v-if="saveStatus" class="summary-text" style="color: var(--el-color-success);">{{ saveStatus }}</span>
|
||||
<el-button type="primary" @click="handleSave">保存全部设置</el-button>
|
||||
</div>
|
||||
</div>
|
||||
</el-card>
|
||||
</div>
|
||||
</section>
|
||||
</template>
|
||||
347
client/src/pages/TasksPage.vue
Normal file
347
client/src/pages/TasksPage.vue
Normal file
@@ -0,0 +1,347 @@
|
||||
<script setup>
|
||||
import { computed, onBeforeUnmount, onMounted, reactive, ref, watch } from 'vue';
|
||||
import { ElMessage, ElMessageBox } from 'element-plus';
|
||||
import TaskDialog from '@/components/TaskDialog.vue';
|
||||
import {
|
||||
createTask,
|
||||
deleteTask,
|
||||
fetchTasks,
|
||||
fetchTaskStatus,
|
||||
runAllTasks,
|
||||
runTask,
|
||||
toggleTask,
|
||||
updateTask,
|
||||
} from '@/api/tasks';
|
||||
import { formatElapsed } from '@/utils/format';
|
||||
|
||||
const tasks = ref([]);
|
||||
const isLoading = ref(true);
|
||||
const dialogOpen = ref(false);
|
||||
const editingTask = ref(null);
|
||||
|
||||
const filters = reactive({
|
||||
city: '',
|
||||
plate: '',
|
||||
});
|
||||
|
||||
const pagination = reactive({
|
||||
page: 1,
|
||||
pageSize: 10,
|
||||
});
|
||||
|
||||
const runner = reactive({
|
||||
timer: null,
|
||||
polling: false,
|
||||
});
|
||||
|
||||
const status = reactive({
|
||||
visible: false,
|
||||
type: 'info',
|
||||
message: '',
|
||||
});
|
||||
|
||||
const filteredTasks = computed(() => {
|
||||
const cityKeyword = filters.city.trim().toLowerCase();
|
||||
const plateKeyword = filters.plate.trim().toLowerCase();
|
||||
|
||||
return tasks.value.filter((task) => {
|
||||
const city = (task.city || '').toLowerCase();
|
||||
const plate = (task.plateName || '').toLowerCase();
|
||||
return (!cityKeyword || city.includes(cityKeyword)) && (!plateKeyword || plate.includes(plateKeyword));
|
||||
});
|
||||
});
|
||||
|
||||
const enabledCount = computed(() => tasks.value.filter((item) => item.enabled).length);
|
||||
const pagedTasks = computed(() => {
|
||||
const start = (pagination.page - 1) * pagination.pageSize;
|
||||
return filteredTasks.value.slice(start, start + pagination.pageSize);
|
||||
});
|
||||
|
||||
const summaryText = computed(
|
||||
() => `共 ${tasks.value.length} 个任务，已启用 ${enabledCount.value} 个，筛选后 ${filteredTasks.value.length} 个`,
|
||||
);
|
||||
|
||||
watch(
|
||||
() => [filters.city, filters.plate],
|
||||
() => {
|
||||
pagination.page = 1;
|
||||
},
|
||||
);
|
||||
|
||||
onMounted(async () => {
|
||||
await loadTasks();
|
||||
await restoreRunningStatus();
|
||||
});
|
||||
|
||||
onBeforeUnmount(() => {
|
||||
stopPolling();
|
||||
});
|
||||
|
||||
async function loadTasks() {
|
||||
isLoading.value = true;
|
||||
|
||||
try {
|
||||
tasks.value = await fetchTasks();
|
||||
} catch (error) {
|
||||
ElMessage.error(`任务加载失败：${error.message}`);
|
||||
} finally {
|
||||
isLoading.value = false;
|
||||
}
|
||||
}
|
||||
|
||||
function openCreateDialog() {
|
||||
editingTask.value = null;
|
||||
dialogOpen.value = true;
|
||||
}
|
||||
|
||||
function openEditDialog(task) {
|
||||
editingTask.value = task;
|
||||
dialogOpen.value = true;
|
||||
}
|
||||
|
||||
function updateStatus(type, message) {
|
||||
status.visible = true;
|
||||
status.type = type;
|
||||
status.message = message;
|
||||
}
|
||||
|
||||
function stopPolling() {
|
||||
if (runner.timer) {
|
||||
clearInterval(runner.timer);
|
||||
runner.timer = null;
|
||||
}
|
||||
runner.polling = false;
|
||||
}
|
||||
|
||||
function applyRunStatus(data) {
|
||||
if (!data?.isRunning && !data?.finished) {
|
||||
stopPolling();
|
||||
return;
|
||||
}
|
||||
|
||||
if (data.finished) {
|
||||
stopPolling();
|
||||
|
||||
if (data.error) {
|
||||
updateStatus('error', `运行失败：${data.error}`);
|
||||
ElMessage.error(`任务执行失败：${data.error}`);
|
||||
return;
|
||||
}
|
||||
|
||||
const results = data.results || [];
|
||||
if (results.length <= 1) {
|
||||
const total = results[0]?.data?.total ?? 0;
|
||||
updateStatus('success', `运行完成，抓取到 ${total} 条结果。`);
|
||||
ElMessage.success(`任务完成，抓取到 ${total} 条结果`);
|
||||
return;
|
||||
}
|
||||
|
||||
const successCount = results.filter((item) => !item.error).length;
|
||||
const failCount = results.filter((item) => item.error).length;
|
||||
updateStatus('success', `批量运行完成，成功 ${successCount} 个，失败 ${failCount} 个。`);
|
||||
ElMessage.success(`批量运行完成，成功 ${successCount} 个`);
|
||||
return;
|
||||
}
|
||||
|
||||
updateStatus(
|
||||
'info',
|
||||
`正在执行：${data.city || '任务'}（${data.current || 0}/${data.total || 0}），已用时 ${formatElapsed(data.elapsed)}`,
|
||||
);
|
||||
}
|
||||
|
||||
async function pollStatus() {
|
||||
try {
|
||||
const data = await fetchTaskStatus();
|
||||
applyRunStatus(data);
|
||||
if (data.finished) {
|
||||
await loadTasks();
|
||||
}
|
||||
} catch (error) {
|
||||
stopPolling();
|
||||
updateStatus('error', `状态轮询失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
function startPolling() {
|
||||
stopPolling();
|
||||
runner.polling = true;
|
||||
runner.timer = setInterval(pollStatus, 2000);
|
||||
}
|
||||
|
||||
async function restoreRunningStatus() {
|
||||
try {
|
||||
const data = await fetchTaskStatus();
|
||||
applyRunStatus(data);
|
||||
if (data.isRunning) {
|
||||
startPolling();
|
||||
}
|
||||
} catch (error) {
|
||||
ElMessage.error(`任务状态恢复失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function submitTask(payload) {
|
||||
try {
|
||||
if (editingTask.value?.id) {
|
||||
await updateTask(editingTask.value.id, payload);
|
||||
ElMessage.success('任务已更新');
|
||||
} else {
|
||||
await createTask(payload);
|
||||
ElMessage.success('任务已创建');
|
||||
}
|
||||
|
||||
dialogOpen.value = false;
|
||||
await loadTasks();
|
||||
} catch (error) {
|
||||
ElMessage.error(`保存失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function handleDelete(task) {
|
||||
try {
|
||||
await ElMessageBox.confirm(`确定要删除“${task.city || '未命名任务'}”吗？`, '删除确认', {
|
||||
type: 'warning',
|
||||
});
|
||||
await deleteTask(task.id);
|
||||
ElMessage.success('任务已删除');
|
||||
await loadTasks();
|
||||
} catch (error) {
|
||||
if (error !== 'cancel' && error !== 'close') {
|
||||
ElMessage.error(`删除失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function handleToggle(task, enabled) {
|
||||
try {
|
||||
await toggleTask(task.id, enabled);
|
||||
ElMessage.success(enabled ? '任务已启用' : '任务已禁用');
|
||||
await loadTasks();
|
||||
} catch (error) {
|
||||
ElMessage.error(`状态更新失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function handleRun(task) {
|
||||
if (runner.polling) {
|
||||
ElMessage.info('当前已有任务在运行，请稍后再试');
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
await runTask(task.id);
|
||||
updateStatus('info', `已开始运行任务：${task.city}`);
|
||||
ElMessage.success(`已开始运行：${task.city}`);
|
||||
startPolling();
|
||||
} catch (error) {
|
||||
updateStatus('error', `启动失败：${error.message}`);
|
||||
ElMessage.error(`启动失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function handleRunAll() {
|
||||
if (!enabledCount.value) {
|
||||
ElMessage.info('没有已启用的任务可以运行');
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
await ElMessageBox.confirm(`确定要运行全部 ${enabledCount.value} 个已启用任务吗？`, '批量执行确认', {
|
||||
type: 'warning',
|
||||
});
|
||||
await runAllTasks();
|
||||
updateStatus('info', '已开始运行全部启用任务，请稍候…');
|
||||
ElMessage.success('已开始运行全部启用任务');
|
||||
startPolling();
|
||||
} catch (error) {
|
||||
if (error !== 'cancel' && error !== 'close') {
|
||||
updateStatus('error', `批量启动失败：${error.message}`);
|
||||
ElMessage.error(`批量启动失败：${error.message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<section class="page-grid">
|
||||
<header class="page-head">
|
||||
<div>
|
||||
<h2>任务配置</h2>
|
||||
|
||||
</div>
|
||||
</header>
|
||||
|
||||
|
||||
|
||||
<el-card shadow="never">
|
||||
<div class="toolbar">
|
||||
<div class="toolbar__group">
|
||||
<el-button type="primary" @click="openCreateDialog">新增任务</el-button>
|
||||
<el-button type="success" :disabled="runner.polling" @click="handleRunAll">运行全部启用</el-button>
|
||||
</div>
|
||||
<div class="summary-text">{{ summaryText }}</div>
|
||||
</div>
|
||||
|
||||
<el-form label-position="top" class="filter-form">
|
||||
<el-row :gutter="16">
|
||||
<el-col :xs="24" :sm="12">
|
||||
<el-form-item label="搜索城市">
|
||||
<el-input v-model="filters.city" placeholder="输入城市关键字" clearable />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
<el-col :xs="24" :sm="12">
|
||||
<el-form-item label="搜索板块">
|
||||
<el-input v-model="filters.plate" placeholder="输入板块关键字" clearable />
|
||||
</el-form-item>
|
||||
</el-col>
|
||||
</el-row>
|
||||
</el-form>
|
||||
|
||||
<el-alert v-if="status.visible" :title="status.message" :type="status.type" :closable="false" show-icon />
|
||||
|
||||
<el-table v-loading="isLoading" :data="pagedTasks" style="width: 100%; margin-top: 18px;" empty-text="没有匹配的任务">
|
||||
<el-table-column prop="city" label="城市" min-width="120" />
|
||||
<el-table-column prop="plateName" label="板块" min-width="120" />
|
||||
<el-table-column label="提示词" min-width="280">
|
||||
<template #default="{ row }">
|
||||
<div class="table-prompt">{{ row.prompt || '-' }}</div>
|
||||
</template>
|
||||
</el-table-column>
|
||||
<el-table-column label="模型" min-width="160">
|
||||
<template #default="{ row }">
|
||||
<el-tag type="info">{{ row.mode || 'qwen3.5-plus' }}</el-tag>
|
||||
</template>
|
||||
</el-table-column>
|
||||
<el-table-column label="状态" width="100">
|
||||
<template #default="{ row }">
|
||||
<el-tag :type="row.enabled ? 'success' : 'danger'">{{ row.enabled ? '启用' : '禁用' }}</el-tag>
|
||||
</template>
|
||||
</el-table-column>
|
||||
<el-table-column label="操作" min-width="260" fixed="right">
|
||||
<template #default="{ row }">
|
||||
<div class="table-actions">
|
||||
<el-button size="small" @click="openEditDialog(row)">编辑</el-button>
|
||||
<el-button size="small" type="primary" :disabled="runner.polling" @click="handleRun(row)">运行</el-button>
|
||||
<el-button size="small" plain @click="handleToggle(row, !row.enabled)">{{ row.enabled ? '禁用' : '启用' }}</el-button>
|
||||
<el-button size="small" type="danger" plain @click="handleDelete(row)">删除</el-button>
|
||||
</div>
|
||||
</template>
|
||||
</el-table-column>
|
||||
</el-table>
|
||||
|
||||
<div class="pagination-wrap">
|
||||
<el-pagination
|
||||
v-if="filteredTasks.length"
|
||||
background
|
||||
layout="total, prev, pager, next"
|
||||
:current-page="pagination.page"
|
||||
:page-size="pagination.pageSize"
|
||||
:total="filteredTasks.length"
|
||||
@current-change="pagination.page = $event"
|
||||
/>
|
||||
</div>
|
||||
</el-card>
|
||||
|
||||
<TaskDialog v-model="dialogOpen" :task="editingTask" @submit="submitTask" />
|
||||
</section>
|
||||
</template>
|
||||
56
client/src/router/index.js
Normal file
56
client/src/router/index.js
Normal file
@@ -0,0 +1,56 @@
|
||||
import { createRouter, createWebHistory } from 'vue-router';
|
||||
|
||||
const routes = [
|
||||
{
|
||||
path: '/',
|
||||
redirect: '/tasks',
|
||||
},
|
||||
{
|
||||
path: '/tasks',
|
||||
name: 'tasks',
|
||||
component: () => import('@/pages/TasksPage.vue'),
|
||||
meta: {
|
||||
title: '任务配置',
|
||||
subtitle: '管理城市任务、切换模型，并手动触发抓取流程。',
|
||||
},
|
||||
},
|
||||
{
|
||||
path: '/results',
|
||||
name: 'results',
|
||||
component: () => import('@/pages/ResultsPage.vue'),
|
||||
meta: {
|
||||
title: '抓取结果',
|
||||
subtitle: '查看抓取记录、按条件筛选，并快速定位失败任务。',
|
||||
},
|
||||
},
|
||||
{
|
||||
path: '/projects',
|
||||
name: 'projects',
|
||||
component: () => import('@/pages/ProjectsPage.vue'),
|
||||
meta: {
|
||||
title: '项目管理',
|
||||
subtitle: '对抓取出的项目去重、筛选与金额范围查询。',
|
||||
},
|
||||
},
|
||||
{
|
||||
path: '/settings',
|
||||
name: 'settings',
|
||||
component: () => import('@/pages/SettingsPage.vue'),
|
||||
meta: {
|
||||
title: '系统设置',
|
||||
subtitle: '维护 Agent、定时任务与邮件推送配置。',
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
const router = createRouter({
|
||||
history: createWebHistory(),
|
||||
routes,
|
||||
});
|
||||
|
||||
router.afterEach((to) => {
|
||||
const title = to.meta?.title ? `${to.meta.title} | 公告抓取与分析工具` : '公告抓取与分析工具';
|
||||
document.title = title;
|
||||
});
|
||||
|
||||
export default router;
|
||||
383
client/src/styles.css
Normal file
383
client/src/styles.css
Normal file
@@ -0,0 +1,383 @@
|
||||
@import url('https://fonts.googleapis.com/css2?family=Plus+Jakarta+Sans:wght@400;500;600;700;800&family=Noto+Sans+SC:wght@400;500;700&family=IBM+Plex+Mono:wght@500&display=swap');
|
||||
|
||||
:root {
|
||||
--bg-top: #edf7ff;
|
||||
--bg-mid: #f8fcf8;
|
||||
--bg-bottom: #fff8ef;
|
||||
--line: rgba(15, 34, 52, 0.1);
|
||||
--text: #11263a;
|
||||
--muted: #60758a;
|
||||
--shadow-lg: 0 24px 54px rgba(11, 42, 68, 0.14);
|
||||
--shadow-md: 0 12px 28px rgba(11, 42, 68, 0.08);
|
||||
--radius-lg: 18px;
|
||||
}
|
||||
|
||||
html,
|
||||
body,
|
||||
#app {
|
||||
min-height: 100%;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: 'Plus Jakarta Sans', 'Noto Sans SC', 'PingFang SC', 'Microsoft YaHei', sans-serif;
|
||||
color: var(--text);
|
||||
background:
|
||||
radial-gradient(880px 520px at -8% -12%, rgba(13, 108, 184, 0.2), transparent 58%),
|
||||
radial-gradient(760px 440px at 110% -10%, rgba(15, 157, 140, 0.18), transparent 52%),
|
||||
radial-gradient(900px 520px at 60% 118%, rgba(215, 139, 30, 0.12), transparent 56%),
|
||||
linear-gradient(160deg, var(--bg-top) 0%, var(--bg-mid) 52%, var(--bg-bottom) 100%);
|
||||
--el-color-primary: #0d6cb8;
|
||||
--el-border-radius-base: 12px;
|
||||
--el-font-family: 'Plus Jakarta Sans', 'Noto Sans SC', 'PingFang SC', 'Microsoft YaHei', sans-serif;
|
||||
}
|
||||
|
||||
a {
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.app-shell {
|
||||
min-height: 100vh;
|
||||
padding: 24px;
|
||||
}
|
||||
|
||||
.app-frame {
|
||||
max-width: 1480px;
|
||||
margin: 0 auto;
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 28px;
|
||||
overflow: hidden;
|
||||
background: rgba(255, 255, 255, 0.68);
|
||||
box-shadow: var(--shadow-lg);
|
||||
backdrop-filter: blur(22px);
|
||||
}
|
||||
|
||||
.app-topbar {
|
||||
padding: 24px 28px;
|
||||
color: #eff8ff;
|
||||
background:
|
||||
radial-gradient(circle at top right, rgba(255, 255, 255, 0.2), transparent 26%),
|
||||
linear-gradient(125deg, #0c639f 0%, #0f85a2 55%, #17a86b 100%);
|
||||
}
|
||||
|
||||
.app-brand {
|
||||
display: flex;
|
||||
gap: 16px;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.app-brand__mark {
|
||||
width: 54px;
|
||||
height: 54px;
|
||||
border-radius: 16px;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
color: #0d4c6e;
|
||||
font-size: 24px;
|
||||
font-weight: 800;
|
||||
background: linear-gradient(145deg, rgba(255, 255, 255, 0.96), rgba(232, 248, 255, 0.84));
|
||||
}
|
||||
|
||||
.app-brand__text h1 {
|
||||
margin: 0;
|
||||
font-size: 26px;
|
||||
}
|
||||
|
||||
.app-brand__text p {
|
||||
margin: 6px 0 0;
|
||||
font-size: 13px;
|
||||
opacity: 0.92;
|
||||
}
|
||||
|
||||
.app-menu.el-menu {
|
||||
padding: 10px 10px 0;
|
||||
border-bottom: 1px solid var(--line);
|
||||
background: rgba(255, 255, 255, 0.26);
|
||||
}
|
||||
|
||||
.app-menu.el-menu--horizontal > .el-menu-item {
|
||||
height: 48px;
|
||||
line-height: 48px;
|
||||
border-bottom: 0;
|
||||
margin: 0 8px 10px 0;
|
||||
border-radius: 14px;
|
||||
color: var(--muted);
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.app-menu.el-menu--horizontal > .el-menu-item.is-active {
|
||||
color: var(--el-color-primary);
|
||||
background: rgba(255, 255, 255, 0.9);
|
||||
}
|
||||
|
||||
.app-main {
|
||||
padding: 28px;
|
||||
}
|
||||
|
||||
.page-grid {
|
||||
display: grid;
|
||||
gap: 18px;
|
||||
}
|
||||
|
||||
.page-head {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
gap: 20px;
|
||||
align-items: flex-end;
|
||||
}
|
||||
|
||||
.page-head h2 {
|
||||
margin: 0;
|
||||
font-size: 28px;
|
||||
}
|
||||
|
||||
.page-head p {
|
||||
margin: 8px 0 0;
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.toolbar {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.toolbar__group {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.summary-text {
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.hero-panel.el-card {
|
||||
border: 0;
|
||||
color: #f7fcff;
|
||||
background: linear-gradient(140deg, rgba(13, 108, 184, 0.95), rgba(15, 157, 140, 0.92));
|
||||
box-shadow: var(--shadow-md);
|
||||
}
|
||||
|
||||
.hero-panel .el-card__body {
|
||||
position: relative;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.hero-panel .el-card__body::after {
|
||||
content: '';
|
||||
position: absolute;
|
||||
right: -70px;
|
||||
top: -80px;
|
||||
width: 240px;
|
||||
height: 240px;
|
||||
border-radius: 50%;
|
||||
background: rgba(255, 255, 255, 0.14);
|
||||
}
|
||||
|
||||
.hero-panel h3,
|
||||
.hero-panel p {
|
||||
position: relative;
|
||||
z-index: 1;
|
||||
}
|
||||
|
||||
.hero-panel h3 {
|
||||
margin: 0;
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
.hero-panel p {
|
||||
margin: 8px 0 0;
|
||||
max-width: 760px;
|
||||
opacity: 0.92;
|
||||
}
|
||||
|
||||
.filter-form {
|
||||
margin-top: 18px;
|
||||
}
|
||||
|
||||
.table-prompt {
|
||||
display: -webkit-box;
|
||||
overflow: hidden;
|
||||
-webkit-box-orient: vertical;
|
||||
-webkit-line-clamp: 2;
|
||||
line-height: 1.6;
|
||||
}
|
||||
|
||||
.table-actions {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.pagination-wrap {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
margin-top: 18px;
|
||||
}
|
||||
|
||||
.dialog-footer {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.result-card-plus {
|
||||
border-radius: var(--radius-lg);
|
||||
box-shadow: var(--shadow-md);
|
||||
}
|
||||
|
||||
.result-card-plus__header {
|
||||
display: flex;
|
||||
align-items: flex-start;
|
||||
justify-content: space-between;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.result-card-plus__title {
|
||||
font-size: 18px;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.result-card-plus__meta {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
margin-top: 8px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.result-card-plus__time {
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.result-card-plus__list {
|
||||
display: grid;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.result-card-plus__item {
|
||||
display: grid;
|
||||
grid-template-columns: 110px 1.6fr 150px 120px auto;
|
||||
gap: 12px;
|
||||
align-items: center;
|
||||
padding-bottom: 12px;
|
||||
border-bottom: 1px solid rgba(15, 34, 52, 0.08);
|
||||
}
|
||||
|
||||
.result-card-plus__item:last-child {
|
||||
border-bottom: 0;
|
||||
padding-bottom: 0;
|
||||
}
|
||||
|
||||
.result-card-plus__type {
|
||||
color: var(--el-color-primary);
|
||||
font-size: 13px;
|
||||
font-weight: 800;
|
||||
}
|
||||
|
||||
.result-card-plus__name {
|
||||
min-width: 0;
|
||||
word-break: break-word;
|
||||
}
|
||||
|
||||
.result-card-plus__amount {
|
||||
color: #99591a;
|
||||
font-weight: 700;
|
||||
text-align: right;
|
||||
}
|
||||
|
||||
.result-card-plus__date {
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.result-card-plus__action {
|
||||
text-align: right;
|
||||
}
|
||||
|
||||
.result-card-plus__footer {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.el-card {
|
||||
--el-card-border-color: rgba(15, 34, 52, 0.08);
|
||||
border-radius: var(--radius-lg);
|
||||
box-shadow: var(--shadow-md);
|
||||
}
|
||||
|
||||
.el-statistic {
|
||||
padding: 18px;
|
||||
border-radius: var(--radius-lg);
|
||||
background: rgba(255, 255, 255, 0.76);
|
||||
border: 1px solid rgba(15, 34, 52, 0.08);
|
||||
box-shadow: var(--shadow-md);
|
||||
}
|
||||
|
||||
.el-statistic__head {
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.el-table {
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
.el-table,
|
||||
.el-table tr,
|
||||
.el-table th.el-table__cell,
|
||||
.el-table td.el-table__cell {
|
||||
background: transparent;
|
||||
}
|
||||
|
||||
.el-empty {
|
||||
padding: 36px 0;
|
||||
}
|
||||
|
||||
@media (max-width: 1120px) {
|
||||
.result-card-plus__item {
|
||||
grid-template-columns: 90px 1fr;
|
||||
}
|
||||
|
||||
.result-card-plus__amount,
|
||||
.result-card-plus__date,
|
||||
.result-card-plus__action {
|
||||
text-align: left;
|
||||
}
|
||||
}
|
||||
|
||||
@media (max-width: 840px) {
|
||||
.app-shell {
|
||||
padding: 12px;
|
||||
}
|
||||
|
||||
.app-main,
|
||||
.app-topbar {
|
||||
padding: 18px;
|
||||
}
|
||||
|
||||
.page-head {
|
||||
align-items: flex-start;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.app-menu.el-menu--horizontal {
|
||||
display: block;
|
||||
height: auto;
|
||||
}
|
||||
|
||||
.app-menu.el-menu--horizontal > .el-menu-item {
|
||||
display: block;
|
||||
}
|
||||
}
|
||||
26
client/src/utils/format.js
Normal file
26
client/src/utils/format.js
Normal file
@@ -0,0 +1,26 @@
|
||||
export function formatDateTime(value) {
|
||||
if (!value) return '-';
|
||||
const date = new Date(value);
|
||||
if (Number.isNaN(date.getTime())) return value;
|
||||
return date.toLocaleString('zh-CN');
|
||||
}
|
||||
|
||||
export function formatYuan(value) {
|
||||
if (typeof value !== 'number' || !Number.isFinite(value)) return '-';
|
||||
return `${value.toLocaleString('zh-CN', { maximumFractionDigits: 2 })} 元`;
|
||||
}
|
||||
|
||||
export function formatElapsed(totalSeconds) {
|
||||
const seconds = Number(totalSeconds) || 0;
|
||||
const minutes = Math.floor(seconds / 60);
|
||||
const remainder = seconds % 60;
|
||||
return minutes > 0 ? `${minutes} 分 ${remainder} 秒` : `${remainder} 秒`;
|
||||
}
|
||||
|
||||
export function pickResultName(item = {}) {
|
||||
return item.project_name || item.projectName || item.title || item.name || item.bidName || '-';
|
||||
}
|
||||
|
||||
export function pickResultLink(item = {}) {
|
||||
return item.detail_link || item.target_link || item.url || item.href || '';
|
||||
}
|
||||
13
config.json
13
config.json
@@ -1,16 +1,21 @@
|
||||
{
|
||||
"agent": {
|
||||
"baseUrl": "http://192.168.3.65:18777",
|
||||
"pollInterval": 3000,
|
||||
"timeout": 3600000
|
||||
},
|
||||
"scheduler": {
|
||||
"enabled": false,
|
||||
"enabled": true,
|
||||
"cronTime": "0 9 * * *",
|
||||
"threshold": 100000,
|
||||
"description": "每天9点采集当日大于10亿的项目",
|
||||
"timeRange": "today"
|
||||
"description": "每天9点采集当日项目",
|
||||
"timeRange": "thisMonth"
|
||||
},
|
||||
"email": {
|
||||
"smtpHost": "smtp.qq.com",
|
||||
"smtpPort": 587,
|
||||
"smtpUser": "1076597680@qq.com",
|
||||
"smtpPass": "nfrjdiraqddsjeeh",
|
||||
"recipients": "5482498@qq.com"
|
||||
"recipients": "1650243281@qq.com"
|
||||
}
|
||||
}
|
||||
3070
package-lock.json
generated
3070
package-lock.json
generated
File diff suppressed because it is too large
Load Diff
26
package.json
26
package.json
@@ -2,19 +2,35 @@
|
||||
"name": "njggzy-scraper",
|
||||
"version": "2.0.0",
|
||||
"type": "module",
|
||||
"description": "南京公共资源交易平台 - 合同估算价采集工具",
|
||||
"description": "公告抓取与分析工具",
|
||||
"main": "src/server.js",
|
||||
"scripts": {
|
||||
"dev": "concurrently \"npm:dev:server\" \"npm:dev:client\"",
|
||||
"dev:server": "node src/server.js",
|
||||
"dev:client": "vite --config vite.config.js",
|
||||
"build": "vite build --config vite.config.js",
|
||||
"preview": "vite preview --config vite.config.js",
|
||||
"start": "node src/server.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"axios": "^1.6.8",
|
||||
"cheerio": "^1.0.0-rc.12",
|
||||
"@mendable/firecrawl-js": "latest",
|
||||
"axios": "^1.11.0",
|
||||
"better-sqlite3": "^12.8.0",
|
||||
"cors": "^2.8.5",
|
||||
"dotenv": "^17.2.3",
|
||||
"element-plus": "^2.11.4",
|
||||
"express": "^5.2.1",
|
||||
"iconv-lite": "^0.6.3",
|
||||
"node-cron": "^4.2.1",
|
||||
"nodemailer": "^7.0.11"
|
||||
"nodemailer": "^7.0.11",
|
||||
"vue": "^3.5.18",
|
||||
"vue-router": "^4.5.1",
|
||||
"zod": "^3.24.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@vitejs/plugin-vue": "^6.0.1",
|
||||
"concurrently": "^9.2.1",
|
||||
"unplugin-auto-import": "^20.2.0",
|
||||
"unplugin-vue-components": "^29.0.0",
|
||||
"vite": "^7.1.3"
|
||||
}
|
||||
}
|
||||
|
||||
529
public/app.js
529
public/app.js
@@ -90,7 +90,7 @@ function displayList(items, container) {
|
||||
<div class="meta">标段编号: ${item.bidNo}</div>
|
||||
<div class="meta">标段名称: ${item.bidName}</div>
|
||||
<div class="meta">发布日期: ${item.date}</div>
|
||||
<span class="budget">${item.budget.amount}${item.budget.unit}</span>
|
||||
<span class="budget">${item.winningBid.amount}${item.winningBid.unit}</span>
|
||||
${item.href ? `<br><a href="${item.href}" target="_blank">查看详情 →</a>` : ''}
|
||||
</div>
|
||||
`).join('')}
|
||||
@@ -189,7 +189,7 @@ function displayReport(report, container) {
|
||||
<div class="meta">标段编号: ${project.bidNo || '-'}</div>
|
||||
<div class="meta">标段名称: ${project.bidName || '-'}</div>
|
||||
<div class="meta">发布日期: ${project.date}</div>
|
||||
<span class="budget">${project.budget.amount}${project.budget.unit}</span>
|
||||
<span class="budget">${project.winningBid.amount}${project.winningBid.unit}</span>
|
||||
${project.url ? `<br><a href="${project.url}" target="_blank">查看详情 →</a>` : ''}
|
||||
</div>
|
||||
`).join('')}
|
||||
@@ -202,10 +202,34 @@ function displayReport(report, container) {
|
||||
async function exportReport() {
|
||||
if (!currentReport) return;
|
||||
|
||||
// 检查docx库是否加载
|
||||
// 按需动态加载docx库
|
||||
if (!window.docx) {
|
||||
alert('Word导出库正在加载中,请稍后再试...');
|
||||
return;
|
||||
try {
|
||||
// 显示加载提示
|
||||
const loadingMsg = document.createElement('div');
|
||||
loadingMsg.textContent = '正在加载导出库...';
|
||||
loadingMsg.style.cssText = 'position:fixed;top:50%;left:50%;transform:translate(-50%,-50%);background:#fff;padding:20px;border-radius:8px;box-shadow:0 4px 12px rgba(0,0,0,0.15);z-index:9999;';
|
||||
document.body.appendChild(loadingMsg);
|
||||
|
||||
// 动态加载docx库
|
||||
await new Promise((resolve, reject) => {
|
||||
const script = document.createElement('script');
|
||||
script.src = 'https://cdn.jsdelivr.net/npm/docx@7.8.2/build/index.js';
|
||||
script.onload = resolve;
|
||||
script.onerror = () => {
|
||||
// 降级到unpkg
|
||||
script.src = 'https://unpkg.com/docx@7.8.2/build/index.js';
|
||||
script.onload = resolve;
|
||||
script.onerror = reject;
|
||||
};
|
||||
document.head.appendChild(script);
|
||||
});
|
||||
|
||||
document.body.removeChild(loadingMsg);
|
||||
} catch (error) {
|
||||
alert('导出库加载失败,请检查网络连接后重试');
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const report = currentReport;
|
||||
@@ -319,7 +343,7 @@ async function exportReport() {
|
||||
new Paragraph({
|
||||
children: [
|
||||
new TextRun({ text: '合同估算价: ', bold: true }),
|
||||
new TextRun({ text: `${project.budget.amount}${project.budget.unit}` })
|
||||
new TextRun({ text: `${project.winningBid.amount}${project.winningBid.unit}` })
|
||||
],
|
||||
spacing: { after: 50 }
|
||||
}),
|
||||
@@ -356,8 +380,8 @@ async function exportReport() {
|
||||
|
||||
// ========== 邮件功能 ==========
|
||||
|
||||
// 保存邮件配置到localStorage
|
||||
function saveEmailConfig() {
|
||||
// 保存邮件配置到服务器
|
||||
async function saveEmailConfig() {
|
||||
const config = {
|
||||
smtpHost: document.getElementById('smtpHost').value,
|
||||
smtpPort: parseInt(document.getElementById('smtpPort').value) || 587,
|
||||
@@ -367,29 +391,107 @@ function saveEmailConfig() {
|
||||
};
|
||||
|
||||
// 验证配置
|
||||
if (!config.smtpHost || !config.smtpUser || !config.smtpPass || !config.recipients) {
|
||||
showEmailStatus('请填写所有必填项', 'error');
|
||||
if (!config.smtpHost || !config.smtpUser || !config.recipients) {
|
||||
showEmailStatus('请填写SMTP服务器、发件人邮箱和收件人', 'error');
|
||||
return;
|
||||
}
|
||||
|
||||
// 保存到localStorage
|
||||
localStorage.setItem('emailConfig', JSON.stringify(config));
|
||||
showEmailStatus('邮件配置已保存', 'success');
|
||||
// 如果密码为空,可能是要保持原密码不变
|
||||
const smtpPassInput = document.getElementById('smtpPass');
|
||||
if (!config.smtpPass && smtpPassInput.placeholder.includes('已配置')) {
|
||||
// 使用占位符表示保持原密码
|
||||
config.smtpPass = '***已配置***';
|
||||
} else if (!config.smtpPass) {
|
||||
showEmailStatus('请填写SMTP密码', 'error');
|
||||
return;
|
||||
}
|
||||
|
||||
showEmailStatus('正在保存配置...', 'info');
|
||||
|
||||
try {
|
||||
// 先获取当前服务器配置
|
||||
const getResponse = await fetch(`${API_BASE}/config`);
|
||||
const getData = await getResponse.json();
|
||||
|
||||
let fullConfig = { email: config };
|
||||
|
||||
// 如果服务器有其他配置(如scheduler),保留它们
|
||||
if (getData.success && getData.data) {
|
||||
fullConfig = {
|
||||
...getData.data,
|
||||
email: config
|
||||
};
|
||||
}
|
||||
|
||||
// 保存到服务器
|
||||
const saveResponse = await fetch(`${API_BASE}/config`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(fullConfig)
|
||||
});
|
||||
|
||||
const saveData = await saveResponse.json();
|
||||
|
||||
if (saveData.success) {
|
||||
// 同时保存到localStorage作为备份
|
||||
localStorage.setItem('emailConfig', JSON.stringify(config));
|
||||
showEmailStatus('邮件配置已保存到服务器', 'success');
|
||||
} else {
|
||||
showEmailStatus(`保存失败: ${saveData.error}`, 'error');
|
||||
}
|
||||
} catch (error) {
|
||||
showEmailStatus(`保存失败: ${error.message}`, 'error');
|
||||
}
|
||||
}
|
||||
|
||||
// 从localStorage加载邮件配置
|
||||
function loadEmailConfig() {
|
||||
const configStr = localStorage.getItem('emailConfig');
|
||||
if (configStr) {
|
||||
try {
|
||||
const config = JSON.parse(configStr);
|
||||
// 从服务器加载邮件配置
|
||||
async function loadEmailConfig() {
|
||||
try {
|
||||
// 从服务器获取配置
|
||||
const response = await fetch(`${API_BASE}/config`);
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success && data.data && data.data.email) {
|
||||
const config = data.data.email;
|
||||
document.getElementById('smtpHost').value = config.smtpHost || '';
|
||||
document.getElementById('smtpPort').value = config.smtpPort || 587;
|
||||
document.getElementById('smtpUser').value = config.smtpUser || '';
|
||||
document.getElementById('smtpPass').value = config.smtpPass || '';
|
||||
// 如果密码是占位符,保持输入框为空或显示占位符
|
||||
document.getElementById('smtpPass').value = config.smtpPass === '***已配置***' ? '' : (config.smtpPass || '');
|
||||
if (config.smtpPass === '***已配置***') {
|
||||
document.getElementById('smtpPass').placeholder = '***已配置*** (留空保持不变)';
|
||||
}
|
||||
document.getElementById('recipients').value = config.recipients || '';
|
||||
} catch (e) {
|
||||
console.error('加载邮件配置失败:', e);
|
||||
|
||||
// 同时保存到localStorage作为备份
|
||||
localStorage.setItem('emailConfig', JSON.stringify(config));
|
||||
} else {
|
||||
// 如果服务器没有配置,尝试从localStorage加载
|
||||
const configStr = localStorage.getItem('emailConfig');
|
||||
if (configStr) {
|
||||
const config = JSON.parse(configStr);
|
||||
document.getElementById('smtpHost').value = config.smtpHost || '';
|
||||
document.getElementById('smtpPort').value = config.smtpPort || 587;
|
||||
document.getElementById('smtpUser').value = config.smtpUser || '';
|
||||
document.getElementById('smtpPass').value = config.smtpPass || '';
|
||||
document.getElementById('recipients').value = config.recipients || '';
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('从服务器加载邮件配置失败:', error);
|
||||
// 失败时尝试从localStorage加载
|
||||
const configStr = localStorage.getItem('emailConfig');
|
||||
if (configStr) {
|
||||
try {
|
||||
const config = JSON.parse(configStr);
|
||||
document.getElementById('smtpHost').value = config.smtpHost || '';
|
||||
document.getElementById('smtpPort').value = config.smtpPort || 587;
|
||||
document.getElementById('smtpUser').value = config.smtpUser || '';
|
||||
document.getElementById('smtpPass').value = config.smtpPass || '';
|
||||
document.getElementById('recipients').value = config.recipients || '';
|
||||
} catch (e) {
|
||||
console.error('从localStorage加载邮件配置失败:', e);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -585,50 +687,34 @@ function cronToFriendlyText(cronTime) {
|
||||
// 加载定时任务配置
|
||||
async function loadSchedulerConfig() {
|
||||
try {
|
||||
// 从服务器获取配置
|
||||
const response = await fetch(`${API_BASE}/config`);
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success && data.data) {
|
||||
const config = data.data;
|
||||
|
||||
// 填充表单
|
||||
if (config.scheduler) {
|
||||
document.getElementById('schedulerEnabled').checked = config.scheduler.enabled || false;
|
||||
const cronTime = config.scheduler.cronTime || '0 9 * * *';
|
||||
document.getElementById('schedulerCronInput').value = cronTime;
|
||||
document.getElementById('schedulerThresholdInput').value = config.scheduler.threshold || 10000;
|
||||
document.getElementById('schedulerThresholdInput').value = config.scheduler.threshold ?? 0;
|
||||
document.getElementById('schedulerDescription').value = config.scheduler.description || '';
|
||||
|
||||
// 时间段配置
|
||||
document.getElementById('schedulerTimeRange').value = config.scheduler.timeRange || 'thisMonth';
|
||||
|
||||
// 反向映射Cron表达式到预设选择器
|
||||
const presetSelector = document.getElementById('schedulerCronPreset');
|
||||
const customGroup = document.getElementById('customCronGroup');
|
||||
|
||||
// 预设值列表
|
||||
const presets = [
|
||||
'0 9 * * *',
|
||||
'0 6 * * *',
|
||||
'0 12 * * *',
|
||||
'0 18 * * *',
|
||||
'0 9,18 * * *',
|
||||
'0 */6 * * *',
|
||||
'0 */12 * * *',
|
||||
'0 9 * * 1',
|
||||
'0 9 1 * *'
|
||||
'0 9 * * *', '0 6 * * *', '0 12 * * *', '0 18 * * *',
|
||||
'0 9,18 * * *', '0 */6 * * *', '0 */12 * * *', '0 9 * * 1', '0 9 1 * *'
|
||||
];
|
||||
|
||||
// 检查是否匹配预设值
|
||||
if (presets.includes(cronTime)) {
|
||||
presetSelector.value = cronTime;
|
||||
customGroup.style.display = 'none';
|
||||
} else {
|
||||
// 自定义时间 - 尝试解析为 "分 时 * * *" 格式
|
||||
presetSelector.value = 'custom';
|
||||
customGroup.style.display = 'block';
|
||||
|
||||
const cronParts = cronTime.split(/\s+/);
|
||||
if (cronParts.length >= 2) {
|
||||
document.getElementById('customMinute').value = cronParts[0];
|
||||
@@ -637,7 +723,6 @@ async function loadSchedulerConfig() {
|
||||
}
|
||||
}
|
||||
|
||||
// 更新状态显示
|
||||
await updateSchedulerStatus();
|
||||
}
|
||||
} catch (error) {
|
||||
@@ -673,9 +758,14 @@ function updateCustomCron() {
|
||||
cronInput.value = `${minute} ${hour} * * *`;
|
||||
}
|
||||
|
||||
document.addEventListener('DOMContentLoaded', function() {
|
||||
loadEmailConfig();
|
||||
loadSchedulerConfig();
|
||||
document.addEventListener('DOMContentLoaded', function () {
|
||||
// 并行加载配置,提高加载速度
|
||||
Promise.all([
|
||||
loadEmailConfig().catch(err => console.error('加载邮件配置失败:', err)),
|
||||
loadSchedulerConfig().catch(err => console.error('加载定时任务配置失败:', err))
|
||||
]).then(() => {
|
||||
console.log('配置加载完成');
|
||||
});
|
||||
|
||||
// 添加自定义时间输入框的事件监听
|
||||
const customHour = document.getElementById('customHour');
|
||||
@@ -705,8 +795,12 @@ async function updateSchedulerStatus() {
|
||||
// 更新执行计划
|
||||
if (status.config) {
|
||||
document.getElementById('schedulerCronTime').textContent = cronToFriendlyText(status.config.cronTime);
|
||||
const thresholdBillion = (status.config.threshold / 10000).toFixed(1);
|
||||
document.getElementById('schedulerThreshold').textContent = `${status.config.threshold}万元 (${thresholdBillion}亿)`;
|
||||
}
|
||||
|
||||
// 更新已启用来源数
|
||||
const enabledCountEl = document.getElementById('schedulerEnabledCount');
|
||||
if (enabledCountEl) {
|
||||
enabledCountEl.textContent = `${status.enabledScrapers ?? '-'} 个`;
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
@@ -719,9 +813,8 @@ async function saveSchedulerConfig() {
|
||||
const schedulerConfig = {
|
||||
enabled: document.getElementById('schedulerEnabled').checked,
|
||||
cronTime: document.getElementById('schedulerCronInput').value,
|
||||
threshold: parseInt(document.getElementById('schedulerThresholdInput').value),
|
||||
threshold: parseInt(document.getElementById('schedulerThresholdInput').value) || 0,
|
||||
description: document.getElementById('schedulerDescription').value,
|
||||
timeRange: document.getElementById('schedulerTimeRange').value
|
||||
};
|
||||
|
||||
// 验证Cron表达式格式(简单验证)
|
||||
@@ -731,36 +824,16 @@ async function saveSchedulerConfig() {
|
||||
return;
|
||||
}
|
||||
|
||||
// 从localStorage获取邮件配置
|
||||
const emailConfigStr = localStorage.getItem('emailConfig');
|
||||
let emailConfig = {};
|
||||
|
||||
if (emailConfigStr) {
|
||||
try {
|
||||
emailConfig = JSON.parse(emailConfigStr);
|
||||
} catch (e) {
|
||||
console.error('解析邮件配置失败:', e);
|
||||
}
|
||||
}
|
||||
|
||||
// 如果邮件配置为空,提示用户
|
||||
if (!emailConfig.smtpHost || !emailConfig.smtpUser) {
|
||||
if (confirm('检测到邮件配置未完成,定时任务需要邮件配置才能发送报告。\n\n是否继续保存定时任务配置(不保存邮件配置)?')) {
|
||||
// 继续保存,但不包含邮件配置
|
||||
} else {
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// 构建完整配置对象
|
||||
const fullConfig = {
|
||||
scheduler: schedulerConfig,
|
||||
email: emailConfig
|
||||
};
|
||||
|
||||
showSchedulerStatus('正在保存配置...', 'info');
|
||||
|
||||
try {
|
||||
// 先获取当前服务器配置(保留 email/scrapers 等字段)
|
||||
const getResponse = await fetch(`${API_BASE}/config`);
|
||||
const getData = await getResponse.json();
|
||||
const currentCfg = (getData.success && getData.data) ? getData.data : {};
|
||||
|
||||
const fullConfig = { ...currentCfg, scheduler: schedulerConfig };
|
||||
|
||||
const response = await fetch(`${API_BASE}/config`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
@@ -771,7 +844,6 @@ async function saveSchedulerConfig() {
|
||||
|
||||
if (data.success) {
|
||||
showSchedulerStatus('配置已保存,定时任务已重新加载!', 'success');
|
||||
// 刷新状态显示
|
||||
await updateSchedulerStatus();
|
||||
} else {
|
||||
showSchedulerStatus(`保存失败: ${data.error}`, 'error');
|
||||
@@ -834,3 +906,306 @@ function showSchedulerStatus(message, type) {
|
||||
}, 3000);
|
||||
}
|
||||
}
|
||||
|
||||
// ========== 招标公告列表功能 ==========
|
||||
let currentBidListPage = 1;
|
||||
|
||||
// 获取招标公告列表
|
||||
async function fetchBidList(pageNum) {
|
||||
const page = pageNum || parseInt(document.getElementById('bidListPage').value) || 1;
|
||||
const loading = document.getElementById('bidListLoading');
|
||||
const results = document.getElementById('bidListResults');
|
||||
const pagination = document.getElementById('bidListPagination');
|
||||
|
||||
currentBidListPage = page;
|
||||
document.getElementById('bidListPage').value = page;
|
||||
|
||||
loading.classList.add('active');
|
||||
results.innerHTML = '';
|
||||
pagination.style.display = 'none';
|
||||
|
||||
try {
|
||||
const response = await fetch(`${API_BASE}/bid-announce/list?page=${page}`);
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
displayBidList(data.data, results);
|
||||
updateBidListPagination(page, data.data.length > 0);
|
||||
} else {
|
||||
results.innerHTML = `<div class="error">错误: ${data.error}</div>`;
|
||||
}
|
||||
} catch (error) {
|
||||
results.innerHTML = `<div class="error">请求失败: ${error.message}</div>`;
|
||||
} finally {
|
||||
loading.classList.remove('active');
|
||||
}
|
||||
}
|
||||
|
||||
function goToBidListPage(page) {
|
||||
if (page < 1) return;
|
||||
fetchBidList(page);
|
||||
}
|
||||
|
||||
function updateBidListPagination(page, hasData) {
|
||||
const pagination = document.getElementById('bidListPagination');
|
||||
const currentPageSpan = document.getElementById('bidCurrentPage');
|
||||
const prevBtn = document.getElementById('bidPrevPage');
|
||||
const firstBtn = document.getElementById('bidFirstPage');
|
||||
const nextBtn = document.getElementById('bidNextPage');
|
||||
|
||||
if (hasData) {
|
||||
pagination.style.display = 'flex';
|
||||
currentPageSpan.textContent = page;
|
||||
|
||||
prevBtn.disabled = page <= 1;
|
||||
firstBtn.disabled = page <= 1;
|
||||
nextBtn.disabled = !hasData;
|
||||
}
|
||||
}
|
||||
|
||||
function displayBidList(items, container) {
|
||||
if (items.length === 0) {
|
||||
container.innerHTML = '<p>没有找到招标公告</p>';
|
||||
return;
|
||||
}
|
||||
|
||||
const html = `
|
||||
<div class="simple-list">
|
||||
<h3 style="margin-bottom: 15px; color: #e67e22;">找到 ${items.length} 条招标公告</h3>
|
||||
${items.map((item, index) => `
|
||||
<div class="list-item" style="border-left-color: #e67e22;">
|
||||
<h3>${index + 1}. ${item.title}</h3>
|
||||
<div class="meta">发布日期: ${item.date}</div>
|
||||
${item.href ? `<a href="${item.href}" target="_blank" style="color: #e67e22;">查看详情 →</a>` : ''}
|
||||
</div>
|
||||
`).join('')}
|
||||
</div>
|
||||
`;
|
||||
|
||||
container.innerHTML = html;
|
||||
}
|
||||
|
||||
// ========== 综合报告功能 ==========
|
||||
let currentWinningReport = null;
|
||||
let currentBidReport = null;
|
||||
|
||||
// 初始化报告日期
|
||||
function initReportDates() {
|
||||
const today = new Date();
|
||||
const firstDayOfMonth = new Date(today.getFullYear(), today.getMonth(), 1);
|
||||
|
||||
document.getElementById('startDate').value = firstDayOfMonth.toISOString().split('T')[0];
|
||||
document.getElementById('endDate').value = today.toISOString().split('T')[0];
|
||||
}
|
||||
|
||||
// 生成综合报告（同时包含中标和招标）
|
||||
async function generateCombinedReport() {
|
||||
const startDate = document.getElementById('startDate').value;
|
||||
const endDate = document.getElementById('endDate').value;
|
||||
const maxPages = parseInt(document.getElementById('maxPages').value) || 10;
|
||||
const winningThreshold = parseFloat(document.getElementById('reportThreshold').value) * 10000 || 0; // 转换为元
|
||||
const bidThreshold = parseFloat(document.getElementById('bidReportThreshold').value) * 10000 || 0;
|
||||
|
||||
if (!startDate && !endDate) {
|
||||
alert('请至少填写开始日期或结束日期');
|
||||
return;
|
||||
}
|
||||
|
||||
const loading = document.getElementById('reportLoading');
|
||||
const loadingText = document.getElementById('reportLoadingText');
|
||||
const results = document.getElementById('reportResults');
|
||||
const sendBtn = document.getElementById('sendEmailBtn');
|
||||
|
||||
loading.classList.add('active');
|
||||
results.innerHTML = '';
|
||||
sendBtn.style.display = 'none';
|
||||
|
||||
try {
|
||||
// 1. 先获取中标报告
|
||||
loadingText.textContent = '正在采集中标公示...';
|
||||
const winningResponse = await fetch(`${API_BASE}/report-daterange`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ startDate, endDate, threshold: winningThreshold, maxPages })
|
||||
});
|
||||
const winningData = await winningResponse.json();
|
||||
|
||||
// 2. 再获取招标报告
|
||||
loadingText.textContent = '正在采集招标公告...';
|
||||
const bidResponse = await fetch(`${API_BASE}/bid-announce/report`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ startDate, endDate, maxPages, threshold: bidThreshold })
|
||||
});
|
||||
const bidData = await bidResponse.json();
|
||||
|
||||
// 3. 显示综合报告
|
||||
if (winningData.success && bidData.success) {
|
||||
currentWinningReport = winningData.data;
|
||||
currentBidReport = bidData.data;
|
||||
displayCombinedReport(winningData.data, bidData.data, results);
|
||||
sendBtn.style.display = 'inline-block';
|
||||
} else {
|
||||
let errorMsg = '';
|
||||
if (!winningData.success) errorMsg += `中标报告错误: ${winningData.error}\n`;
|
||||
if (!bidData.success) errorMsg += `招标报告错误: ${bidData.error}`;
|
||||
results.innerHTML = `<div class="error">${errorMsg}</div>`;
|
||||
}
|
||||
} catch (error) {
|
||||
results.innerHTML = `<div class="error">请求失败: ${error.message}</div>`;
|
||||
} finally {
|
||||
loading.classList.remove('active');
|
||||
loadingText.textContent = '正在生成报告...';
|
||||
}
|
||||
}
|
||||
|
||||
// 显示综合报告
|
||||
function displayCombinedReport(winningReport, bidReport, container) {
|
||||
const html = `
|
||||
<!-- 中标公示部分 -->
|
||||
<div class="summary" style="background: linear-gradient(135deg, #0f6ecd 0%, #0ea5a4 100%);">
|
||||
<h2>中标公示报告</h2>
|
||||
<div class="stat">
|
||||
<div class="stat-label">总项目数</div>
|
||||
<div class="stat-value">${winningReport.summary.total_count}</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">符合条件</div>
|
||||
<div class="stat-value">${winningReport.summary.filtered_count}</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">总金额</div>
|
||||
<div class="stat-value">${winningReport.summary.total_amount}</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">阈值</div>
|
||||
<div class="stat-value">${winningReport.summary.threshold}</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
${winningReport.projects.length === 0 ? '<p style="color: #999; margin-bottom: 20px;">暂无符合条件的中标项目</p>' : `
|
||||
<h3 style="margin-bottom: 15px;">中标项目列表 (${winningReport.projects.length} 条)</h3>
|
||||
<div class="simple-list" style="margin-bottom: 30px;">
|
||||
${winningReport.projects.map((project, index) => `
|
||||
<div class="list-item">
|
||||
<h3>${index + 1}. ${project.title}</h3>
|
||||
<div class="meta">标段编号: ${project.bidNo || '-'}</div>
|
||||
<div class="meta">标段名称: ${project.bidName || '-'}</div>
|
||||
<div class="meta">发布日期: ${project.date}</div>
|
||||
<span class="budget">${project.winningBid.amount}${project.winningBid.unit}</span>
|
||||
${project.url ? `<br><a href="${project.url}" target="_blank">查看详情 →</a>` : ''}
|
||||
</div>
|
||||
`).join('')}
|
||||
</div>
|
||||
`}
|
||||
|
||||
<!-- 招标公告部分 -->
|
||||
<div class="summary" style="background: linear-gradient(135deg, #e67e22 0%, #d35400 100%); margin-top: 30px;">
|
||||
<h2>招标公告报告</h2>
|
||||
<div class="stat">
|
||||
<div class="stat-label">总公告数量</div>
|
||||
<div class="stat-value">${bidReport.summary.total_count} 条</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">有金额信息</div>
|
||||
<div class="stat-value">${bidReport.summary.has_amount_count || bidReport.summary.filtered_count} 条</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">金额阈值</div>
|
||||
<div class="stat-value">${bidReport.summary.threshold || '无'}</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">合同估算总额</div>
|
||||
<div class="stat-value">${bidReport.summary.total_amount}</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
${bidReport.projects.length === 0 ? '<p style="color: #999;">暂无符合条件的招标项目</p>' : `
|
||||
<h3 style="margin-bottom: 15px; color: #e67e22;">招标项目详情 (${bidReport.projects.length} 条)</h3>
|
||||
<div class="simple-list">
|
||||
${bidReport.projects.map((item, index) => `
|
||||
<div class="list-item" style="border-left-color: #e67e22;">
|
||||
<h3>${index + 1}. ${item.title}</h3>
|
||||
<div class="meta">发布日期: ${item.date}</div>
|
||||
${item.bidCode ? `<div class="meta">标段编码: ${item.bidCode}</div>` : ''}
|
||||
${item.tenderee ? `<div class="meta">招标人: ${item.tenderee}</div>` : ''}
|
||||
${item.duration ? `<div class="meta">工期: ${item.duration} 日历天</div>` : ''}
|
||||
${item.estimatedAmount ? `
|
||||
<span class="budget" style="background: #e67e22;">合同估算价: ${item.estimatedAmount.amountWan} 万元</span>
|
||||
` : ''}
|
||||
<br><a href="${item.url}" target="_blank" style="color: #e67e22;">查看详情 →</a>
|
||||
</div>
|
||||
`).join('')}
|
||||
</div>
|
||||
`}
|
||||
`;
|
||||
|
||||
container.innerHTML = html;
|
||||
}
|
||||
|
||||
// 发送综合报告邮件
|
||||
async function sendCombinedReportByEmail() {
|
||||
if (!currentWinningReport && !currentBidReport) {
|
||||
alert('请先生成报告');
|
||||
return;
|
||||
}
|
||||
|
||||
// 从localStorage获取邮件配置
|
||||
const emailConfigStr = localStorage.getItem('emailConfig');
|
||||
if (!emailConfigStr) {
|
||||
alert('请先在"邮件配置"页面配置邮件服务器信息');
|
||||
return;
|
||||
}
|
||||
|
||||
let emailConfig;
|
||||
try {
|
||||
emailConfig = JSON.parse(emailConfigStr);
|
||||
} catch (e) {
|
||||
alert('邮件配置解析失败,请重新配置');
|
||||
return;
|
||||
}
|
||||
|
||||
if (!emailConfig.smtpHost || !emailConfig.smtpUser || !emailConfig.smtpPass || !emailConfig.recipients) {
|
||||
alert('邮件配置不完整,请检查SMTP服务器、用户名、密码和收件人');
|
||||
return;
|
||||
}
|
||||
|
||||
if (!confirm(`确定要将综合报告发送到以下邮箱吗?\n\n${emailConfig.recipients}`)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const sendBtn = document.getElementById('sendEmailBtn');
|
||||
const originalText = sendBtn.textContent;
|
||||
sendBtn.disabled = true;
|
||||
sendBtn.textContent = '正在发送...';
|
||||
|
||||
try {
|
||||
const response = await fetch(`${API_BASE}/send-combined-email`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
emailConfig,
|
||||
winningReport: currentWinningReport,
|
||||
bidReport: currentBidReport
|
||||
})
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (data.success) {
|
||||
alert('综合报告邮件发送成功!');
|
||||
} else {
|
||||
alert(`邮件发送失败: ${data.error}`);
|
||||
}
|
||||
} catch (error) {
|
||||
alert(`请求失败: ${error.message}`);
|
||||
} finally {
|
||||
sendBtn.disabled = false;
|
||||
sendBtn.textContent = originalText;
|
||||
}
|
||||
}
|
||||
|
||||
// 页面加载时初始化报告日期
|
||||
document.addEventListener('DOMContentLoaded', function () {
|
||||
initReportDates();
|
||||
});
|
||||
|
||||
1759
public/index.html
1759
public/index.html
File diff suppressed because it is too large
Load Diff
1579
public/results.html
Normal file
1579
public/results.html
Normal file
File diff suppressed because it is too large
Load Diff
1
scrapegraph-service/.env
Normal file
1
scrapegraph-service/.env
Normal file
@@ -0,0 +1 @@
|
||||
|
||||
151
src/agentService.js
Normal file
151
src/agentService.js
Normal file
@@ -0,0 +1,151 @@
|
||||
/**
|
||||
* Agent API 服务封装
|
||||
* 调用本地部署的 agent 进行公告抓取
|
||||
*/
|
||||
|
||||
const DEFAULT_BASE_URL = 'http://192.168.3.65:18625';
|
||||
const DEFAULT_POLL_INTERVAL = 3000; // 3秒轮询
|
||||
const DEFAULT_TIMEOUT = 3600000; // 1小时超时
|
||||
const FETCH_TIMEOUT = 30000; // 单次 fetch 30秒超时
|
||||
const MAX_FETCH_RETRIES = 5; // 网络错误最多重试5次
|
||||
const DEFAULT_MODE = 'qwen3.5-plus';
|
||||
|
||||
function normalizeMode(value) {
|
||||
if (typeof value === 'string' && value.trim()) return value.trim();
|
||||
return DEFAULT_MODE;
|
||||
}
|
||||
|
||||
|
||||
function generateTaskId() {
|
||||
return `task-${Date.now()}-${Math.random().toString(36).slice(2, 7)}`;
|
||||
}
|
||||
|
||||
function sleep(ms) {
|
||||
return new Promise(resolve => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
/**
|
||||
* 带超时和重试的 fetch
|
||||
*/
|
||||
async function fetchWithRetry(url, fetchOptions, retries = MAX_FETCH_RETRIES, logPrefix = '[Agent]') {
|
||||
for (let attempt = 1; attempt <= retries; attempt++) {
|
||||
try {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), FETCH_TIMEOUT);
|
||||
const res = await fetch(url, { ...fetchOptions, signal: controller.signal });
|
||||
clearTimeout(timer);
|
||||
return res;
|
||||
} catch (err) {
|
||||
const isLast = attempt === retries;
|
||||
console.warn(`${logPrefix} fetch 失败 (${attempt}/${retries}): ${err.message}`);
|
||||
if (isLast) throw err;
|
||||
await sleep(3000 * attempt); // 递增等待
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 创建 agent 任务
|
||||
*/
|
||||
async function createTask(prompt, options = {}) {
|
||||
const baseUrl = options.baseUrl || DEFAULT_BASE_URL;
|
||||
const mode = normalizeMode(options.mode);
|
||||
const taskId = generateTaskId();
|
||||
const logPrefix = options.logPrefix || '[Agent]';
|
||||
|
||||
const res = await fetchWithRetry(`${baseUrl}/agent/createTask`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ taskId, prompt, mode }),
|
||||
}, MAX_FETCH_RETRIES, logPrefix);
|
||||
|
||||
if (!res.ok) {
|
||||
throw new Error(`创建任务失败: HTTP ${res.status}`);
|
||||
}
|
||||
|
||||
return { taskId };
|
||||
}
|
||||
|
||||
/**
|
||||
* 检查任务状态
|
||||
* 返回空/null 表示任务还在运行，返回 { success, message, data } 表示完成
|
||||
*/
|
||||
async function checkTask(taskId, options = {}) {
|
||||
const baseUrl = options.baseUrl || DEFAULT_BASE_URL;
|
||||
const logPrefix = options.logPrefix || '[Agent]';
|
||||
|
||||
const res = await fetchWithRetry(`${baseUrl}/agent/checkTask`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ taskId }),
|
||||
}, MAX_FETCH_RETRIES, logPrefix);
|
||||
|
||||
if (!res.ok) {
|
||||
throw new Error(`检查任务失败: HTTP ${res.status}`);
|
||||
}
|
||||
|
||||
const text = await res.text();
|
||||
console.log(`${logPrefix} checkTask(${taskId}) 返回:`, text ? text.substring(0, 500) : '(空)');
|
||||
if (!text || text.trim() === '' || text.trim() === 'null') {
|
||||
return null; // 任务还在运行
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 运行 agent 任务：创建 + 轮询直到完成
|
||||
* 返回 { results: [{ type, project_name, amount_yuan, date, detail_link }] }
|
||||
*/
|
||||
export async function runAgentTask(prompt, options = {}) {
|
||||
const baseUrl = options.baseUrl || DEFAULT_BASE_URL;
|
||||
const mode = normalizeMode(options.mode);
|
||||
const pollInterval = options.pollInterval || DEFAULT_POLL_INTERVAL;
|
||||
const timeout = options.timeout || DEFAULT_TIMEOUT;
|
||||
const logPrefix = options.logPrefix || '[Agent]';
|
||||
|
||||
console.log(`${logPrefix} 创建任务...`);
|
||||
console.log(`${logPrefix} 使用 mode: ${mode}`);
|
||||
const { taskId } = await createTask(prompt, { baseUrl, mode, logPrefix });
|
||||
console.log(`${logPrefix} 任务已创建: ${taskId}`);
|
||||
|
||||
const startTime = Date.now();
|
||||
|
||||
while (true) {
|
||||
if (Date.now() - startTime > timeout) {
|
||||
throw new Error(`任务超时 (${timeout / 1000}秒): ${taskId}`);
|
||||
}
|
||||
|
||||
await sleep(pollInterval);
|
||||
|
||||
const result = await checkTask(taskId, { baseUrl, logPrefix });
|
||||
|
||||
if (result === null) {
|
||||
const elapsed = Math.round((Date.now() - startTime) / 1000);
|
||||
console.log(`${logPrefix} 任务进行中... (${elapsed}秒)`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (result.success) {
|
||||
console.log(`${logPrefix} 任务完成: ${result.message}`);
|
||||
const data = result.data || {};
|
||||
const rawResults = Array.isArray(data.results) ? data.results : [];
|
||||
const results = rawResults.map(item => {
|
||||
if (!item || typeof item !== 'object') return item;
|
||||
const detailLink = item.detail_link || item.target_link;
|
||||
const { target_link, ...rest } = item;
|
||||
return detailLink ? { ...rest, detail_link: detailLink } : rest;
|
||||
});
|
||||
console.log(`${logPrefix} 获取到 ${results.length} 条结果`);
|
||||
return { results };
|
||||
} else {
|
||||
throw new Error(`任务失败: ${result.message || '未知错误'}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export { generateTaskId, createTask, checkTask };
|
||||
@@ -21,7 +21,7 @@ export async function sendReportEmail(emailConfig, report) {
|
||||
const info = await transporter.sendMail({
|
||||
from: `"公告采集系统" <${emailConfig.smtpUser}>`,
|
||||
to: emailConfig.recipients,
|
||||
subject: `采购公告分析报告 - ${new Date().toLocaleDateString('zh-CN')}`,
|
||||
subject: `交通水务中标结果报告 - ${new Date().toLocaleDateString('zh-CN')}`,
|
||||
html: htmlContent,
|
||||
});
|
||||
|
||||
@@ -35,6 +35,523 @@ export async function sendReportEmail(emailConfig, report) {
|
||||
}
|
||||
}
|
||||
|
||||
// 发送招标公告报告邮件
|
||||
export async function sendBidAnnounceReportEmail(emailConfig, report) {
|
||||
try {
|
||||
const transporter = nodemailer.createTransport({
|
||||
host: emailConfig.smtpHost,
|
||||
port: emailConfig.smtpPort || 587,
|
||||
secure: emailConfig.smtpPort === 465,
|
||||
auth: {
|
||||
user: emailConfig.smtpUser,
|
||||
pass: emailConfig.smtpPass,
|
||||
},
|
||||
});
|
||||
|
||||
const htmlContent = generateBidAnnounceReportHtml(report);
|
||||
|
||||
const info = await transporter.sendMail({
|
||||
from: `"公告采集系统" <${emailConfig.smtpUser}>`,
|
||||
to: emailConfig.recipients,
|
||||
subject: `交通水务招标公告报告 - ${new Date().toLocaleDateString('zh-CN')}`,
|
||||
html: htmlContent,
|
||||
});
|
||||
|
||||
return {
|
||||
success: true,
|
||||
messageId: info.messageId,
|
||||
};
|
||||
} catch (error) {
|
||||
console.error('发送招标公告邮件失败:', error);
|
||||
throw new Error(`邮件发送失败: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// 生成招标公告报告HTML
|
||||
function generateBidAnnounceReportHtml(report) {
|
||||
const { summary, projects } = report;
|
||||
|
||||
return `
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>交通水务招标公告报告</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: 'PingFang SC', 'Microsoft YaHei', Arial, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
max-width: 900px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
background-color: #f5f5f5;
|
||||
}
|
||||
.container {
|
||||
background: white;
|
||||
border-radius: 8px;
|
||||
padding: 30px;
|
||||
box-shadow: 0 2px 10px rgba(0,0,0,0.1);
|
||||
}
|
||||
h1 {
|
||||
color: #e67e22;
|
||||
border-bottom: 3px solid #e67e22;
|
||||
padding-bottom: 10px;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
.summary {
|
||||
background: linear-gradient(135deg, #e67e22 0%, #d35400 100%);
|
||||
color: white;
|
||||
padding: 20px;
|
||||
border-radius: 8px;
|
||||
margin-bottom: 30px;
|
||||
}
|
||||
.summary h2 {
|
||||
margin-top: 0;
|
||||
margin-bottom: 15px;
|
||||
font-size: 18px;
|
||||
}
|
||||
.stat-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(180px, 1fr));
|
||||
gap: 15px;
|
||||
}
|
||||
.stat {
|
||||
background: rgba(255,255,255,0.15);
|
||||
padding: 12px;
|
||||
border-radius: 6px;
|
||||
}
|
||||
.stat-label {
|
||||
font-size: 13px;
|
||||
opacity: 0.9;
|
||||
margin-bottom: 5px;
|
||||
}
|
||||
.stat-value {
|
||||
font-size: 20px;
|
||||
font-weight: bold;
|
||||
}
|
||||
.project-list {
|
||||
margin-top: 20px;
|
||||
}
|
||||
.project-item {
|
||||
background: #f9f9f9;
|
||||
border-left: 4px solid #e67e22;
|
||||
padding: 15px;
|
||||
margin-bottom: 15px;
|
||||
border-radius: 4px;
|
||||
}
|
||||
.project-item h3 {
|
||||
color: #333;
|
||||
margin: 0 0 10px 0;
|
||||
font-size: 15px;
|
||||
line-height: 1.4;
|
||||
}
|
||||
.project-meta {
|
||||
color: #666;
|
||||
font-size: 13px;
|
||||
margin: 5px 0;
|
||||
}
|
||||
.amount {
|
||||
display: inline-block;
|
||||
background: #e67e22;
|
||||
color: white;
|
||||
padding: 4px 12px;
|
||||
border-radius: 4px;
|
||||
font-weight: bold;
|
||||
margin-top: 8px;
|
||||
font-size: 14px;
|
||||
}
|
||||
.project-link {
|
||||
color: #e67e22;
|
||||
text-decoration: none;
|
||||
font-size: 12px;
|
||||
word-break: break-all;
|
||||
}
|
||||
.footer {
|
||||
margin-top: 30px;
|
||||
padding-top: 20px;
|
||||
border-top: 1px solid #e0e0e0;
|
||||
color: #999;
|
||||
font-size: 12px;
|
||||
text-align: center;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<h1>南京公共资源交易平台 - 交通水务招标公告报告</h1>
|
||||
|
||||
<div class="summary">
|
||||
<h2>报告摘要</h2>
|
||||
<div class="stat-grid">
|
||||
<div class="stat">
|
||||
<div class="stat-label">总公告数量</div>
|
||||
<div class="stat-value">${summary.total_count} 条</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">有金额信息</div>
|
||||
<div class="stat-value">${summary.has_amount_count || summary.filtered_count} 条</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">符合筛选</div>
|
||||
<div class="stat-value">${summary.filtered_count} 条</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">金额阈值</div>
|
||||
<div class="stat-value">${summary.threshold || '无'}</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">合同估算总额</div>
|
||||
<div class="stat-value">${summary.total_amount}</div>
|
||||
</div>
|
||||
</div>
|
||||
${summary.date_range ? `
|
||||
<div style="margin-top: 15px; padding-top: 15px; border-top: 1px solid rgba(255,255,255,0.2);">
|
||||
<div class="stat-label">时间范围</div>
|
||||
<div style="font-size: 14px; margin-top: 5px;">
|
||||
${summary.date_range.startDate || '不限'} 至 ${summary.date_range.endDate || '不限'}
|
||||
</div>
|
||||
</div>
|
||||
` : ''}
|
||||
</div>
|
||||
|
||||
<h2>招标项目详情</h2>
|
||||
<div class="project-list">
|
||||
${projects.length === 0 ? '<p style="color: #999; text-align: center; padding: 20px;">暂无符合条件的项目</p>' : ''}
|
||||
${projects.map((project, index) => `
|
||||
<div class="project-item">
|
||||
<h3>${index + 1}. ${project.title}</h3>
|
||||
<div class="project-meta">
|
||||
<strong>发布日期:</strong> ${project.date}
|
||||
${project.bidCode ? ` | <strong>标段编码:</strong> ${project.bidCode}` : ''}
|
||||
</div>
|
||||
${project.tenderee ? `
|
||||
<div class="project-meta">
|
||||
<strong>招标人:</strong> ${project.tenderee}
|
||||
${project.duration ? ` | <strong>工期:</strong> ${project.duration}日历天` : ''}
|
||||
</div>
|
||||
` : ''}
|
||||
${project.estimatedAmount ? `
|
||||
<div class="amount">
|
||||
合同估算价: ${project.estimatedAmount.amountWan} 万元
|
||||
</div>
|
||||
` : ''}
|
||||
<div style="margin-top: 10px;">
|
||||
<a href="${project.url}" class="project-link" target="_blank">${project.url}</a>
|
||||
</div>
|
||||
</div>
|
||||
`).join('')}
|
||||
</div>
|
||||
|
||||
<div class="footer">
|
||||
<p>报告生成时间: ${new Date(summary.generated_at).toLocaleString('zh-CN')}</p>
|
||||
<p>本报告由公告采集系统自动生成</p>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
`;
|
||||
}
|
||||
|
||||
// 发送综合报告邮件（中标+招标）
|
||||
export async function sendCombinedReportEmail(emailConfig, winningReport, bidReport) {
|
||||
try {
|
||||
const transporter = nodemailer.createTransport({
|
||||
host: emailConfig.smtpHost,
|
||||
port: emailConfig.smtpPort || 587,
|
||||
secure: emailConfig.smtpPort === 465,
|
||||
auth: {
|
||||
user: emailConfig.smtpUser,
|
||||
pass: emailConfig.smtpPass,
|
||||
},
|
||||
});
|
||||
|
||||
const htmlContent = generateCombinedReportHtml(winningReport, bidReport);
|
||||
|
||||
const info = await transporter.sendMail({
|
||||
from: `"公告采集系统" <${emailConfig.smtpUser}>`,
|
||||
to: emailConfig.recipients,
|
||||
subject: `交通水务综合报告 - ${new Date().toLocaleDateString('zh-CN')}`,
|
||||
html: htmlContent,
|
||||
});
|
||||
|
||||
return {
|
||||
success: true,
|
||||
messageId: info.messageId,
|
||||
};
|
||||
} catch (error) {
|
||||
console.error('发送综合邮件失败:', error);
|
||||
throw new Error(`邮件发送失败: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// 生成综合报告HTML（中标+招标）
|
||||
function generateCombinedReportHtml(winningReport, bidReport) {
|
||||
return `
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>交通水务综合报告</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: 'PingFang SC', 'Microsoft YaHei', Arial, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
max-width: 900px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
background-color: #f5f5f5;
|
||||
}
|
||||
.container {
|
||||
background: white;
|
||||
border-radius: 8px;
|
||||
padding: 30px;
|
||||
box-shadow: 0 2px 10px rgba(0,0,0,0.1);
|
||||
}
|
||||
h1 {
|
||||
color: #333;
|
||||
border-bottom: 3px solid #667eea;
|
||||
padding-bottom: 10px;
|
||||
margin-bottom: 30px;
|
||||
text-align: center;
|
||||
}
|
||||
.section {
|
||||
margin-bottom: 40px;
|
||||
}
|
||||
.section-title {
|
||||
font-size: 20px;
|
||||
font-weight: bold;
|
||||
margin-bottom: 15px;
|
||||
padding: 10px 15px;
|
||||
border-radius: 6px;
|
||||
}
|
||||
.section-title.winning {
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
color: white;
|
||||
}
|
||||
.section-title.bid {
|
||||
background: linear-gradient(135deg, #e67e22 0%, #d35400 100%);
|
||||
color: white;
|
||||
}
|
||||
.summary {
|
||||
padding: 15px;
|
||||
border-radius: 8px;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
.summary.winning {
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
color: white;
|
||||
}
|
||||
.summary.bid {
|
||||
background: linear-gradient(135deg, #e67e22 0%, #d35400 100%);
|
||||
color: white;
|
||||
}
|
||||
.stat-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
|
||||
gap: 15px;
|
||||
}
|
||||
.stat {
|
||||
background: rgba(255,255,255,0.15);
|
||||
padding: 10px;
|
||||
border-radius: 6px;
|
||||
}
|
||||
.stat-label {
|
||||
font-size: 12px;
|
||||
opacity: 0.9;
|
||||
margin-bottom: 5px;
|
||||
}
|
||||
.stat-value {
|
||||
font-size: 18px;
|
||||
font-weight: bold;
|
||||
}
|
||||
.project-list {
|
||||
margin-top: 15px;
|
||||
}
|
||||
.project-item {
|
||||
background: #f9f9f9;
|
||||
padding: 12px;
|
||||
margin-bottom: 10px;
|
||||
border-radius: 4px;
|
||||
}
|
||||
.project-item.winning {
|
||||
border-left: 4px solid #667eea;
|
||||
}
|
||||
.project-item.bid {
|
||||
border-left: 4px solid #e67e22;
|
||||
}
|
||||
.project-item h3 {
|
||||
color: #333;
|
||||
margin: 0 0 8px 0;
|
||||
font-size: 14px;
|
||||
line-height: 1.4;
|
||||
}
|
||||
.project-meta {
|
||||
color: #666;
|
||||
font-size: 12px;
|
||||
margin: 3px 0;
|
||||
}
|
||||
.amount {
|
||||
display: inline-block;
|
||||
color: white;
|
||||
padding: 3px 10px;
|
||||
border-radius: 4px;
|
||||
font-weight: bold;
|
||||
margin-top: 6px;
|
||||
font-size: 13px;
|
||||
}
|
||||
.amount.winning {
|
||||
background: #667eea;
|
||||
}
|
||||
.amount.bid {
|
||||
background: #e67e22;
|
||||
}
|
||||
.project-link {
|
||||
text-decoration: none;
|
||||
font-size: 11px;
|
||||
word-break: break-all;
|
||||
}
|
||||
.project-link.winning {
|
||||
color: #667eea;
|
||||
}
|
||||
.project-link.bid {
|
||||
color: #e67e22;
|
||||
}
|
||||
.footer {
|
||||
margin-top: 30px;
|
||||
padding-top: 20px;
|
||||
border-top: 1px solid #e0e0e0;
|
||||
color: #999;
|
||||
font-size: 12px;
|
||||
text-align: center;
|
||||
}
|
||||
.no-data {
|
||||
color: #999;
|
||||
text-align: center;
|
||||
padding: 20px;
|
||||
font-style: italic;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<h1>南京公共资源交易平台 - 交通水务综合报告</h1>
|
||||
|
||||
${bidReport ? `
|
||||
<!-- 招标公告部分 -->
|
||||
<div class="section">
|
||||
<div class="section-title bid">招标公告</div>
|
||||
<div class="summary bid">
|
||||
<div class="stat-grid">
|
||||
<div class="stat">
|
||||
<div class="stat-label">总公告数量</div>
|
||||
<div class="stat-value">${bidReport.summary.total_count} 条</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">有金额信息</div>
|
||||
<div class="stat-value">${bidReport.summary.has_amount_count || bidReport.summary.filtered_count} 条</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">符合筛选</div>
|
||||
<div class="stat-value">${bidReport.summary.filtered_count} 条</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">金额阈值</div>
|
||||
<div class="stat-value">${bidReport.summary.threshold || '无'}</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">合同估算总额</div>
|
||||
<div class="stat-value">${bidReport.summary.total_amount}</div>
|
||||
</div>
|
||||
</div>
|
||||
${bidReport.summary.date_range ? `
|
||||
<div style="margin-top: 10px; padding-top: 10px; border-top: 1px solid rgba(255,255,255,0.2); font-size: 13px;">
|
||||
时间范围: ${bidReport.summary.date_range.startDate || '不限'} 至 ${bidReport.summary.date_range.endDate || '不限'}
|
||||
</div>
|
||||
` : ''}
|
||||
</div>
|
||||
<div class="project-list">
|
||||
${bidReport.projects.length === 0 ? '<p class="no-data">暂无符合条件的招标项目</p>' : ''}
|
||||
${bidReport.projects.map((project, index) => `
|
||||
<div class="project-item bid">
|
||||
<h3>${index + 1}. ${project.title}</h3>
|
||||
<div class="project-meta"><strong>发布日期:</strong> ${project.date}</div>
|
||||
${project.bidCode ? `<div class="project-meta"><strong>标段编码:</strong> ${project.bidCode}</div>` : ''}
|
||||
${project.tenderee ? `<div class="project-meta"><strong>招标人:</strong> ${project.tenderee}</div>` : ''}
|
||||
${project.duration ? `<div class="project-meta"><strong>工期:</strong> ${project.duration}日历天</div>` : ''}
|
||||
${project.estimatedAmount ? `
|
||||
<div class="amount bid">合同估算价: ${project.estimatedAmount.amountWan} 万元</div>
|
||||
` : ''}
|
||||
<div style="margin-top: 8px;">
|
||||
<a href="${project.url}" class="project-link bid" target="_blank">${project.url}</a>
|
||||
</div>
|
||||
</div>
|
||||
`).join('')}
|
||||
</div>
|
||||
</div>
|
||||
` : ''}
|
||||
|
||||
${winningReport ? `
|
||||
<!-- 中标公示部分 -->
|
||||
<div class="section">
|
||||
<div class="section-title winning">中标结果公示</div>
|
||||
<div class="summary winning">
|
||||
<div class="stat-grid">
|
||||
<div class="stat">
|
||||
<div class="stat-label">总项目数</div>
|
||||
<div class="stat-value">${winningReport.summary.total_count} 条</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">符合条件</div>
|
||||
<div class="stat-value">${winningReport.summary.filtered_count} 条</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">金额阈值</div>
|
||||
<div class="stat-value">${winningReport.summary.threshold}</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<div class="stat-label">总金额</div>
|
||||
<div class="stat-value">${winningReport.summary.total_amount}</div>
|
||||
</div>
|
||||
</div>
|
||||
${winningReport.summary.date_range ? `
|
||||
<div style="margin-top: 10px; padding-top: 10px; border-top: 1px solid rgba(255,255,255,0.2); font-size: 13px;">
|
||||
时间范围: ${winningReport.summary.date_range.startDate || '不限'} 至 ${winningReport.summary.date_range.endDate || '不限'}
|
||||
</div>
|
||||
` : ''}
|
||||
</div>
|
||||
<div class="project-list">
|
||||
${winningReport.projects.length === 0 ? '<p class="no-data">暂无符合条件的中标项目</p>' : ''}
|
||||
${winningReport.projects.map((project, index) => `
|
||||
<div class="project-item winning">
|
||||
<h3>${index + 1}. ${project.title}</h3>
|
||||
<div class="project-meta"><strong>中标日期:</strong> ${project.date}</div>
|
||||
${project.bidNo ? `<div class="project-meta"><strong>标段编号:</strong> ${project.bidNo}</div>` : ''}
|
||||
${project.winningBid ? `
|
||||
<div class="amount winning">中标金额: ${project.winningBid.amount.toFixed(2)} ${project.winningBid.unit}</div>
|
||||
` : ''}
|
||||
<div style="margin-top: 8px;">
|
||||
<a href="${project.url}" class="project-link winning" target="_blank">${project.url}</a>
|
||||
</div>
|
||||
</div>
|
||||
`).join('')}
|
||||
</div>
|
||||
</div>
|
||||
` : ''}
|
||||
|
||||
<div class="footer">
|
||||
<p>报告生成时间: ${new Date().toLocaleString('zh-CN')}</p>
|
||||
<p>本报告由公告采集系统自动生成</p>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
`;
|
||||
}
|
||||
|
||||
// 生成HTML格式的报告
|
||||
function generateReportHtml(report) {
|
||||
const { summary, projects } = report;
|
||||
@@ -45,7 +562,7 @@ function generateReportHtml(report) {
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>采购公告分析报告</title>
|
||||
<title>交通水务中标结果报告</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: 'PingFang SC', 'Microsoft YaHei', Arial, sans-serif;
|
||||
@@ -147,7 +664,7 @@ function generateReportHtml(report) {
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<h1>南京公共工程建设中心 - 采购公告分析报告</h1>
|
||||
<h1>南京公共资源交易平台 - 交通水务中标结果报告</h1>
|
||||
|
||||
<div class="summary">
|
||||
<h2>报告摘要</h2>
|
||||
@@ -186,15 +703,17 @@ function generateReportHtml(report) {
|
||||
<div class="project-item">
|
||||
<h3>${index + 1}. ${project.title}</h3>
|
||||
<div class="project-meta">
|
||||
<strong>发布日期:</strong> ${project.date}
|
||||
${project.publish_time ? ` | <strong>发布时间:</strong> ${project.publish_time}` : ''}
|
||||
<strong>中标日期:</strong> ${project.date}
|
||||
</div>
|
||||
${project.budget ? `
|
||||
${project.winningBid ? `
|
||||
<div class="budget">
|
||||
中标金额: ${project.winningBid.amount.toFixed(2)} ${project.winningBid.unit}
|
||||
</div>
|
||||
` : (project.budget ? `
|
||||
<div class="budget">
|
||||
预算金额: ${project.budget.amount.toFixed(2)} ${project.budget.unit}
|
||||
${project.budget.originalUnit !== project.budget.unit ? ` (原始: ${project.budget.originalUnit})` : ''}
|
||||
</div>
|
||||
` : ''}
|
||||
` : '')}
|
||||
<div style="margin-top: 10px;">
|
||||
<a href="${project.url}" class="project-link" target="_blank">${project.url}</a>
|
||||
</div>
|
||||
@@ -211,3 +730,172 @@ function generateReportHtml(report) {
|
||||
</html>
|
||||
`;
|
||||
}
|
||||
|
||||
// ========== 通用抓取结果邮件（定时任务使用） ==========
|
||||
|
||||
export async function sendScraperResultsEmail(emailConfig, results) {
|
||||
try {
|
||||
const transporter = nodemailer.createTransport({
|
||||
host: emailConfig.smtpHost,
|
||||
port: emailConfig.smtpPort || 587,
|
||||
secure: emailConfig.smtpPort === 465,
|
||||
auth: {
|
||||
user: emailConfig.smtpUser,
|
||||
pass: emailConfig.smtpPass,
|
||||
},
|
||||
});
|
||||
|
||||
const htmlContent = generateScraperResultsHtml(results);
|
||||
const successCount = results.filter(r => !r.error).length;
|
||||
|
||||
const info = await transporter.sendMail({
|
||||
from: `"公告采集系统" <${emailConfig.smtpUser}>`,
|
||||
to: emailConfig.recipients,
|
||||
subject: `公告采集结果报告(${successCount}条) - ${new Date().toLocaleDateString('zh-CN')}`,
|
||||
html: htmlContent,
|
||||
});
|
||||
|
||||
return { success: true, messageId: info.messageId };
|
||||
} catch (error) {
|
||||
console.error('发送抓取结果邮件失败:', error);
|
||||
throw new Error(`邮件发送失败: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
function generateScraperResultsHtml(results) {
|
||||
const successResults = results.filter(r => !r.error);
|
||||
const failResults = results.filter(r => r.error);
|
||||
const generatedAt = new Date().toLocaleString('zh-CN');
|
||||
|
||||
// Flatten all successful source items into one table.
|
||||
const allRows = [];
|
||||
for (const r of successResults) {
|
||||
const items = r.data?.results || r.data?.result || [];
|
||||
for (const item of items) {
|
||||
const hasAmount = typeof item.amount_yuan === 'number' || !!item.amount;
|
||||
const amountText =
|
||||
typeof item.amount_yuan === 'number'
|
||||
? `${item.amount_yuan.toLocaleString('zh-CN')} CNY`
|
||||
: (item.amount || 'N/A');
|
||||
|
||||
allRows.push({
|
||||
section: [r.section, r.subsection].filter(Boolean).join(' / ') || r.city || '-',
|
||||
type: item.type || r.type || '-',
|
||||
title: item.project_name || item.title || '-',
|
||||
date: item.date || '-',
|
||||
amount: amountText,
|
||||
hasAmount,
|
||||
url: item.detail_link || item.target_link || item.url || '',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
allRows.sort((a, b) => {
|
||||
if (a.date === b.date) return 0;
|
||||
return a.date > b.date ? -1 : 1;
|
||||
});
|
||||
|
||||
const totalItems = allRows.length;
|
||||
|
||||
// 行颜色交替
|
||||
const rowHtml = allRows.length === 0
|
||||
? `<tr><td colspan="6" style="text-align:center;color:#999;padding:30px;font-size:14px;">暂无数据</td></tr>`
|
||||
: allRows.map((row, i) => `
|
||||
<tr style="background:${i % 2 === 0 ? '#fff' : '#f7f8ff'};">
|
||||
<td style="padding:9px 12px;border-bottom:1px solid #eaecf5;white-space:nowrap;color:#555;font-size:13px;">${row.section}</td>
|
||||
<td style="padding:9px 12px;border-bottom:1px solid #eaecf5;white-space:nowrap;">
|
||||
<span style="display:inline-block;padding:2px 8px;background:#e8f4fd;color:#1a73c8;border-radius:10px;font-size:11px;font-weight:600;">${row.type}</span>
|
||||
</td>
|
||||
<td style="padding:9px 12px;border-bottom:1px solid #eaecf5;font-size:13px;max-width:320px;">${row.title}</td>
|
||||
<td style="padding:9px 12px;border-bottom:1px solid #eaecf5;white-space:nowrap;font-size:13px;color:#555;">${row.date}</td>
|
||||
<td style="padding:9px 12px;border-bottom:1px solid #eaecf5;white-space:nowrap;font-size:13px;font-weight:600;color:${row.hasAmount ? '#e67e22' : '#aaa'};">${row.amount}</td>
|
||||
<td style="padding:9px 12px;border-bottom:1px solid #eaecf5;text-align:center;">
|
||||
${row.url
|
||||
? `<a href="${row.url}" target="_blank" style="color:#667eea;font-size:12px;text-decoration:none;white-space:nowrap;">查看 →</a>`
|
||||
: '<span style="color:#ccc;font-size:12px;">-</span>'
|
||||
}
|
||||
</td>
|
||||
</tr>`).join('');
|
||||
|
||||
// 失败来源列表
|
||||
const failHtml = failResults.length === 0 ? '' : `
|
||||
<div style="margin-top:24px;">
|
||||
<div style="font-size:14px;font-weight:600;color:#c0392b;margin-bottom:10px;">⚠️ 抓取失败的来源（${failResults.length} 个）</div>
|
||||
${failResults.map(r => `
|
||||
<div style="background:#fdeaea;border-left:3px solid #e74c3c;padding:10px 14px;border-radius:4px;margin-bottom:8px;font-size:13px;">
|
||||
<strong>${r.city || ''}${r.section ? ' · ' + r.section : ''}${r.type ? ' · ' + r.type : ''}</strong>
|
||||
<div style="color:#999;font-size:12px;margin-top:4px;">${r.url}</div>
|
||||
<div style="color:#c0392b;margin-top:4px;">❌ ${r.error}</div>
|
||||
</div>`).join('')}
|
||||
</div>`;
|
||||
|
||||
return `
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>公告采集结果报告</title>
|
||||
</head>
|
||||
<body style="font-family:'PingFang SC','Microsoft YaHei',Arial,sans-serif;line-height:1.6;color:#333;margin:0;padding:20px;background:#f0f2f8;">
|
||||
<div style="max-width:960px;margin:0 auto;background:white;border-radius:10px;overflow:hidden;box-shadow:0 4px 20px rgba(0,0,0,.1);">
|
||||
|
||||
<!-- 标题栏 -->
|
||||
<div style="background:linear-gradient(135deg,#667eea 0%,#764ba2 100%);padding:24px 30px;color:white;">
|
||||
<h1 style="margin:0;font-size:20px;font-weight:700;">📋 公告采集结果报告</h1>
|
||||
<div style="margin-top:6px;opacity:.85;font-size:13px;">生成时间：${generatedAt}</div>
|
||||
</div>
|
||||
|
||||
<!-- 统计栏 -->
|
||||
<div style="display:flex;gap:0;border-bottom:1px solid #eaecf5;">
|
||||
<div style="flex:1;padding:16px 24px;text-align:center;border-right:1px solid #eaecf5;">
|
||||
<div style="font-size:28px;font-weight:700;color:#667eea;">${totalItems}</div>
|
||||
<div style="font-size:12px;color:#888;margin-top:2px;">公告总数</div>
|
||||
</div>
|
||||
<div style="flex:1;padding:16px 24px;text-align:center;border-right:1px solid #eaecf5;">
|
||||
<div style="font-size:28px;font-weight:700;color:#1a8a4a;">${successResults.length}</div>
|
||||
<div style="font-size:12px;color:#888;margin-top:2px;">成功来源</div>
|
||||
</div>
|
||||
<div style="flex:1;padding:16px 24px;text-align:center;border-right:1px solid #eaecf5;">
|
||||
<div style="font-size:28px;font-weight:700;color:#e67e22;">${allRows.filter(r => r.hasAmount).length}</div>
|
||||
<div style="font-size:12px;color:#888;margin-top:2px;">有金额</div>
|
||||
</div>
|
||||
<div style="flex:1;padding:16px 24px;text-align:center;">
|
||||
<div style="font-size:28px;font-weight:700;color:${failResults.length > 0 ? '#c0392b' : '#aaa'};">${failResults.length}</div>
|
||||
<div style="font-size:12px;color:#888;margin-top:2px;">失败来源</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- 公告汇总表格 -->
|
||||
<div style="padding:24px 30px;">
|
||||
<div style="font-size:15px;font-weight:600;color:#333;margin-bottom:14px;">公告汇总（共 ${totalItems} 条）</div>
|
||||
<div style="overflow-x:auto;">
|
||||
<table style="width:100%;border-collapse:collapse;font-size:13px;">
|
||||
<thead>
|
||||
<tr style="background:linear-gradient(135deg,#667eea 0%,#764ba2 100%);color:white;">
|
||||
<th style="padding:10px 12px;text-align:left;font-weight:600;white-space:nowrap;">板块</th>
|
||||
<th style="padding:10px 12px;text-align:left;font-weight:600;white-space:nowrap;">类型</th>
|
||||
<th style="padding:10px 12px;text-align:left;font-weight:600;">公告标题</th>
|
||||
<th style="padding:10px 12px;text-align:left;font-weight:600;white-space:nowrap;">发布日期</th>
|
||||
<th style="padding:10px 12px;text-align:left;font-weight:600;white-space:nowrap;">项目金额</th>
|
||||
<th style="padding:10px 12px;text-align:center;font-weight:600;white-space:nowrap;">详情</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
${rowHtml}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
${failHtml}
|
||||
|
||||
<div style="margin-top:24px;padding-top:16px;border-top:1px solid #eaecf5;color:#aaa;font-size:12px;text-align:center;">
|
||||
本报告由公告采集系统自动生成 · ${generatedAt}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
`;
|
||||
}
|
||||
|
||||
275
src/firecrawlBrowserScraper.js
Normal file
275
src/firecrawlBrowserScraper.js
Normal file
@@ -0,0 +1,275 @@
|
||||
const DEFAULT_SCRAPER_PROMPT = '提取页面上今日的招标公告信息，包括：标题、项目金额（可能为合同预估价/最高投标限价等）、发布日期（YYYY-MM-DD格式）、详情页完整URL';
|
||||
const PAYLOAD_MARKER = '__FC_PAYLOAD__';
|
||||
|
||||
function pad2(value) {
|
||||
return String(value).padStart(2, '0');
|
||||
}
|
||||
|
||||
function formatDate(year, month, day) {
|
||||
return `${year}-${pad2(month)}-${pad2(day)}`;
|
||||
}
|
||||
|
||||
function getTodayInShanghai() {
|
||||
return new Intl.DateTimeFormat('en-CA', {
|
||||
timeZone: 'Asia/Shanghai',
|
||||
year: 'numeric',
|
||||
month: '2-digit',
|
||||
day: '2-digit',
|
||||
}).format(new Date());
|
||||
}
|
||||
|
||||
function parseTargetDate(prompt) {
|
||||
const text = String(prompt || '');
|
||||
if (!text) return null;
|
||||
|
||||
const fullDate = text.match(/(20\d{2})[-/.年](\d{1,2})[-/.月](\d{1,2})日?/);
|
||||
if (fullDate) {
|
||||
return formatDate(fullDate[1], fullDate[2], fullDate[3]);
|
||||
}
|
||||
|
||||
if (/(今天|今日|当日)/.test(text)) {
|
||||
return getTodayInShanghai();
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function normalizeDate(input) {
|
||||
if (!input) return '';
|
||||
const text = String(input).trim();
|
||||
if (!text) return '';
|
||||
|
||||
let m = text.match(/(20\d{2})[-/.年](\d{1,2})[-/.月](\d{1,2})日?/);
|
||||
if (m) return formatDate(m[1], m[2], m[3]);
|
||||
|
||||
m = text.match(/(\d{1,2})[-/.月](\d{1,2})日?/);
|
||||
if (m) {
|
||||
const currentYear = Number(getTodayInShanghai().slice(0, 4));
|
||||
return formatDate(currentYear, m[1], m[2]);
|
||||
}
|
||||
|
||||
return '';
|
||||
}
|
||||
|
||||
function extractDateFromText(text) {
|
||||
if (!text) return '';
|
||||
const m = String(text).match(/(20\d{2}[-/.年]\d{1,2}[-/.月]\d{1,2}日?)|(\d{1,2}[-/.月]\d{1,2}日?)/);
|
||||
return m ? normalizeDate(m[0]) : '';
|
||||
}
|
||||
|
||||
function extractAmountFromText(text) {
|
||||
if (!text) return null;
|
||||
const m = String(text).match(/([0-9][0-9,.\s]*(?:亿元|万元|万|元))/);
|
||||
if (!m) return null;
|
||||
return m[1].replace(/\s+/g, '').trim();
|
||||
}
|
||||
|
||||
function cleanText(text) {
|
||||
return String(text || '').replace(/\s+/g, ' ').trim();
|
||||
}
|
||||
|
||||
function toFiniteNumber(value, fallback) {
|
||||
const n = Number(value);
|
||||
return Number.isFinite(n) ? n : fallback;
|
||||
}
|
||||
|
||||
function parsePayloadFromText(rawText) {
|
||||
if (!rawText) return null;
|
||||
const text = String(rawText);
|
||||
|
||||
const markerIndex = text.lastIndexOf(PAYLOAD_MARKER);
|
||||
if (markerIndex >= 0) {
|
||||
const tail = text.slice(markerIndex + PAYLOAD_MARKER.length);
|
||||
const firstLine = tail.split(/\r?\n/).find(line => line.trim());
|
||||
if (firstLine) {
|
||||
try {
|
||||
return JSON.parse(firstLine.trim());
|
||||
} catch {
|
||||
// Continue fallback parsing.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(text.trim());
|
||||
} catch {
|
||||
// Continue fallback parsing.
|
||||
}
|
||||
|
||||
const lines = text.split(/\r?\n/).map(line => line.trim()).filter(Boolean).reverse();
|
||||
for (const line of lines) {
|
||||
try {
|
||||
return JSON.parse(line);
|
||||
} catch {
|
||||
// Try next line.
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function parseBrowserExecutePayload(executeResult) {
|
||||
const sources = [executeResult?.result, executeResult?.stdout]
|
||||
.filter(value => typeof value === 'string' && value.trim().length > 0);
|
||||
|
||||
for (const source of sources) {
|
||||
const payload = parsePayloadFromText(source);
|
||||
if (payload && typeof payload === 'object') return payload;
|
||||
}
|
||||
|
||||
return { items: [] };
|
||||
}
|
||||
|
||||
function splitKeywords(input) {
|
||||
return String(input || '')
|
||||
.split(/[、/，,|\s]+/)
|
||||
.map(item => item.trim())
|
||||
.filter(item => item.length >= 2);
|
||||
}
|
||||
|
||||
function filterByTypeIfPossible(items, type) {
|
||||
const keywords = splitKeywords(type);
|
||||
if (keywords.length === 0) return items;
|
||||
|
||||
const filtered = items.filter(item => {
|
||||
const haystack = `${item.title} ${item.context || ''}`;
|
||||
return keywords.some(keyword => haystack.includes(keyword));
|
||||
});
|
||||
|
||||
return filtered.length > 0 ? filtered : items;
|
||||
}
|
||||
|
||||
function normalizeItems(rawItems, targetDate, scraperType) {
|
||||
const dedup = new Map();
|
||||
|
||||
for (const raw of rawItems) {
|
||||
const title = cleanText(raw?.title);
|
||||
const url = cleanText(raw?.url);
|
||||
if (!title || !url) continue;
|
||||
|
||||
const context = cleanText(raw?.context);
|
||||
const date = normalizeDate(raw?.date) || extractDateFromText(context);
|
||||
const amount = cleanText(raw?.amount) || extractAmountFromText(context) || null;
|
||||
const key = `${title}@@${url}`;
|
||||
|
||||
if (!dedup.has(key)) {
|
||||
dedup.set(key, { title, amount, date, url, context });
|
||||
}
|
||||
}
|
||||
|
||||
let items = Array.from(dedup.values());
|
||||
items = filterByTypeIfPossible(items, scraperType);
|
||||
|
||||
if (targetDate) {
|
||||
items = items.filter(item => item.date === targetDate);
|
||||
}
|
||||
|
||||
return items
|
||||
.map(({ title, amount, date, url }) => ({ title, amount, date, url }))
|
||||
.slice(0, 100);
|
||||
}
|
||||
|
||||
function buildBrowserScript(url) {
|
||||
return `
|
||||
const targetUrl = ${JSON.stringify(url)};
|
||||
await page.goto(targetUrl, { waitUntil: 'domcontentloaded', timeout: 60000 });
|
||||
await page.waitForTimeout(1500);
|
||||
|
||||
const payload = await page.evaluate(() => {
|
||||
const normalize = (value) => String(value || '').replace(/\\s+/g, ' ').trim();
|
||||
const blockedTitles = new Set(['首页', '尾页', '上一页', '下一页', '更多', '详情', '查看', '返回', '跳转']);
|
||||
|
||||
const links = Array.from(document.querySelectorAll('a[href]'));
|
||||
const rows = [];
|
||||
const seen = new Set();
|
||||
|
||||
for (const a of links) {
|
||||
const href = a.getAttribute('href') || '';
|
||||
if (!href || href.startsWith('javascript:') || href.startsWith('#')) continue;
|
||||
|
||||
const title = normalize(a.textContent);
|
||||
if (!title || title.length < 6 || title.length > 180) continue;
|
||||
if (blockedTitles.has(title)) continue;
|
||||
|
||||
let absoluteUrl = '';
|
||||
try {
|
||||
absoluteUrl = new URL(href, location.href).href;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
|
||||
const container = a.closest('tr,li,article,section,div,p,dd,dt') || a.parentElement;
|
||||
const context = normalize(container ? container.textContent : title);
|
||||
|
||||
const dateMatch = context.match(/(20\\d{2}[-/.年]\\d{1,2}[-/.月]\\d{1,2}日?)|(\\d{1,2}[-/.月]\\d{1,2}日?)/);
|
||||
const amountMatch = context.match(/([0-9][0-9,.\\s]*(?:亿元|万元|万|元))/);
|
||||
|
||||
const key = (title + '@@' + absoluteUrl).toLowerCase();
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
|
||||
rows.push({
|
||||
title,
|
||||
url: absoluteUrl,
|
||||
date: dateMatch ? dateMatch[0] : '',
|
||||
amount: amountMatch ? amountMatch[0].replace(/\\s+/g, '') : null,
|
||||
context,
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
pageUrl: location.href,
|
||||
items: rows.slice(0, 300),
|
||||
};
|
||||
});
|
||||
|
||||
console.log('${PAYLOAD_MARKER}' + JSON.stringify(payload));
|
||||
JSON.stringify(payload);
|
||||
`;
|
||||
}
|
||||
|
||||
export async function runScraperWithBrowser(firecrawl, scraper, options = {}) {
|
||||
const prefix = options.logPrefix || '[Browser]';
|
||||
if (!scraper?.url) throw new Error('抓取 URL 不能为空');
|
||||
|
||||
const prompt = scraper.prompt || DEFAULT_SCRAPER_PROMPT;
|
||||
const targetDate = parseTargetDate(prompt);
|
||||
|
||||
const ttl = toFiniteNumber(scraper.browserTtl, 180);
|
||||
const activityTtl = toFiniteNumber(scraper.browserActivityTtl, 90);
|
||||
|
||||
const session = await firecrawl.browser({ ttl, activityTtl });
|
||||
if (!session?.success || !session.id) {
|
||||
throw new Error(session?.error || '创建 Browser 会话失败');
|
||||
}
|
||||
|
||||
let executeResult;
|
||||
try {
|
||||
executeResult = await firecrawl.browserExecute(session.id, {
|
||||
code: buildBrowserScript(scraper.url),
|
||||
language: 'node',
|
||||
});
|
||||
} finally {
|
||||
try {
|
||||
await firecrawl.deleteBrowser(session.id);
|
||||
} catch (closeError) {
|
||||
console.warn(`${prefix} 会话关闭失败: ${closeError.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (!executeResult?.success) {
|
||||
throw new Error(executeResult?.error || executeResult?.stderr || 'Browser 执行失败');
|
||||
}
|
||||
|
||||
const payload = parseBrowserExecutePayload(executeResult);
|
||||
const rawItems = Array.isArray(payload.items) ? payload.items : [];
|
||||
const items = normalizeItems(rawItems, targetDate, scraper.type);
|
||||
|
||||
console.log(`${prefix} URL=${scraper.url} raw=${rawItems.length} normalized=${items.length}${targetDate ? ` targetDate=${targetDate}` : ''}`);
|
||||
|
||||
return {
|
||||
items,
|
||||
targetDate,
|
||||
pageUrl: payload.pageUrl || scraper.url,
|
||||
};
|
||||
}
|
||||
731
src/resultStore.js
Normal file
731
src/resultStore.js
Normal file
@@ -0,0 +1,731 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'fs';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { dirname, join } from 'path';
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = dirname(__filename);
|
||||
|
||||
const DB_PATH =
|
||||
process.env.APP_DB_PATH ||
|
||||
process.env.RESULTS_DB_PATH ||
|
||||
join(__dirname, '..', 'data', 'results.sqlite');
|
||||
const CONFIG_PATH = join(__dirname, '..', 'config.json');
|
||||
const MAX_RESULT_RECORDS = 500;
|
||||
const DEFAULT_TASK_MODE = 'qwen3.5-plus';
|
||||
const TASK_COLUMNS = ['id', 'city', 'plate_name', 'prompt', 'enabled', 'mode', 'created_at', 'updated_at'];
|
||||
|
||||
let db = null;
|
||||
let initialized = false;
|
||||
|
||||
function clone(value) {
|
||||
return JSON.parse(JSON.stringify(value));
|
||||
}
|
||||
|
||||
function generateResultId() {
|
||||
return `result-${Date.now()}-${Math.random().toString(36).slice(2, 7)}`;
|
||||
}
|
||||
|
||||
function generateTaskId() {
|
||||
return `task-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`;
|
||||
}
|
||||
|
||||
function getDefaultJsonConfig() {
|
||||
return {
|
||||
agent: {
|
||||
baseUrl: '',
|
||||
pollInterval: 3000,
|
||||
timeout: 300000,
|
||||
},
|
||||
scheduler: {
|
||||
enabled: false,
|
||||
cronTime: '0 9 * * *',
|
||||
threshold: 100000,
|
||||
description: '',
|
||||
timeRange: 'thisMonth',
|
||||
},
|
||||
email: {
|
||||
smtpHost: '',
|
||||
smtpPort: 587,
|
||||
smtpUser: '',
|
||||
smtpPass: '',
|
||||
recipients: '',
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeJsonConfig(input = {}) {
|
||||
const defaults = getDefaultJsonConfig();
|
||||
const agent = input.agent || {};
|
||||
const scheduler = input.scheduler || {};
|
||||
const email = input.email || {};
|
||||
|
||||
return {
|
||||
agent: {
|
||||
baseUrl: typeof agent.baseUrl === 'string' ? agent.baseUrl : defaults.agent.baseUrl,
|
||||
pollInterval: Number.isFinite(agent.pollInterval) ? agent.pollInterval : defaults.agent.pollInterval,
|
||||
timeout: Number.isFinite(agent.timeout) ? agent.timeout : defaults.agent.timeout,
|
||||
},
|
||||
scheduler: {
|
||||
enabled: scheduler.enabled === true,
|
||||
cronTime: typeof scheduler.cronTime === 'string' && scheduler.cronTime.trim()
|
||||
? scheduler.cronTime
|
||||
: defaults.scheduler.cronTime,
|
||||
threshold: Number.isFinite(scheduler.threshold) ? scheduler.threshold : defaults.scheduler.threshold,
|
||||
description: typeof scheduler.description === 'string' ? scheduler.description : defaults.scheduler.description,
|
||||
timeRange: typeof scheduler.timeRange === 'string' && scheduler.timeRange.trim()
|
||||
? scheduler.timeRange
|
||||
: defaults.scheduler.timeRange,
|
||||
},
|
||||
email: {
|
||||
smtpHost: typeof email.smtpHost === 'string' ? email.smtpHost : defaults.email.smtpHost,
|
||||
smtpPort: Number.isFinite(email.smtpPort) ? email.smtpPort : defaults.email.smtpPort,
|
||||
smtpUser: typeof email.smtpUser === 'string' ? email.smtpUser : defaults.email.smtpUser,
|
||||
smtpPass: typeof email.smtpPass === 'string' ? email.smtpPass : defaults.email.smtpPass,
|
||||
recipients: typeof email.recipients === 'string' ? email.recipients : defaults.email.recipients,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeTaskMode(value) {
|
||||
if (typeof value === 'string' && value.trim()) return value.trim();
|
||||
return DEFAULT_TASK_MODE;
|
||||
}
|
||||
|
||||
function buildTaskRecord(task = {}) {
|
||||
return {
|
||||
id: task.id || generateTaskId(),
|
||||
city: task.city || '',
|
||||
plateName: task.plateName || '',
|
||||
prompt: task.prompt || '',
|
||||
enabled: task.enabled !== false,
|
||||
mode: normalizeTaskMode(task.mode),
|
||||
};
|
||||
}
|
||||
|
||||
function buildResultRecord(result = {}) {
|
||||
return {
|
||||
id: result.id || generateResultId(),
|
||||
taskId: result.taskId || null,
|
||||
city: result.city || null,
|
||||
scrapedAt: result.scrapedAt || new Date().toISOString(),
|
||||
error: result.error || null,
|
||||
data: result.data ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
function parseTaskRow(row) {
|
||||
return {
|
||||
id: row.id,
|
||||
city: row.city,
|
||||
plateName: row.plate_name,
|
||||
prompt: row.prompt,
|
||||
enabled: row.enabled === 1,
|
||||
mode: normalizeTaskMode(row.mode),
|
||||
};
|
||||
}
|
||||
|
||||
function parseResultRow(row) {
|
||||
return {
|
||||
id: row.id,
|
||||
taskId: row.task_id,
|
||||
city: row.city,
|
||||
scrapedAt: row.scraped_at,
|
||||
error: row.error,
|
||||
data: row.data_json ? JSON.parse(row.data_json) : null,
|
||||
};
|
||||
}
|
||||
|
||||
function getDb() {
|
||||
if (db) return db;
|
||||
|
||||
mkdirSync(dirname(DB_PATH), { recursive: true });
|
||||
db = new Database(DB_PATH);
|
||||
db.pragma('journal_mode = WAL');
|
||||
return db;
|
||||
}
|
||||
|
||||
function ensureSchema() {
|
||||
getDb().exec(`
|
||||
CREATE TABLE IF NOT EXISTS results (
|
||||
id TEXT PRIMARY KEY,
|
||||
task_id TEXT,
|
||||
city TEXT,
|
||||
scraped_at TEXT NOT NULL,
|
||||
error TEXT,
|
||||
data_json TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_results_scraped_at ON results (scraped_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_results_city ON results (city);
|
||||
CREATE INDEX IF NOT EXISTS idx_results_task_id ON results (task_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
city TEXT,
|
||||
plate_name TEXT,
|
||||
prompt TEXT,
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
mode TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
`);
|
||||
}
|
||||
|
||||
function ensureTasksTableShape() {
|
||||
const columns = getDb().prepare(`PRAGMA table_info(tasks)`).all();
|
||||
const columnNames = columns.map((column) => column.name);
|
||||
const hasLegacyBrowserColumn = columnNames.includes('use_browser');
|
||||
const matchesExpectedShape =
|
||||
columnNames.length === TASK_COLUMNS.length &&
|
||||
TASK_COLUMNS.every((column, index) => columnNames[index] === column);
|
||||
|
||||
if (!hasLegacyBrowserColumn && matchesExpectedShape) return;
|
||||
|
||||
getDb().exec(`
|
||||
BEGIN;
|
||||
|
||||
ALTER TABLE tasks RENAME TO tasks_legacy;
|
||||
|
||||
CREATE TABLE tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
city TEXT,
|
||||
plate_name TEXT,
|
||||
prompt TEXT,
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
mode TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
INSERT INTO tasks (id, city, plate_name, prompt, enabled, mode, created_at, updated_at)
|
||||
SELECT
|
||||
id,
|
||||
city,
|
||||
plate_name,
|
||||
prompt,
|
||||
COALESCE(enabled, 1),
|
||||
COALESCE(mode, '${DEFAULT_TASK_MODE}'),
|
||||
COALESCE(created_at, datetime('now')),
|
||||
COALESCE(updated_at, datetime('now'))
|
||||
FROM tasks_legacy;
|
||||
|
||||
DROP TABLE tasks_legacy;
|
||||
|
||||
COMMIT;
|
||||
`);
|
||||
}
|
||||
|
||||
function trimResults(limit = MAX_RESULT_RECORDS) {
|
||||
getDb().prepare(`
|
||||
DELETE FROM results
|
||||
WHERE id NOT IN (
|
||||
SELECT id
|
||||
FROM results
|
||||
ORDER BY scraped_at DESC, rowid DESC
|
||||
LIMIT ?
|
||||
)
|
||||
`).run(limit);
|
||||
}
|
||||
|
||||
function readJsonIfExists(filePath) {
|
||||
if (!existsSync(filePath)) return null;
|
||||
return JSON.parse(readFileSync(filePath, 'utf-8'));
|
||||
}
|
||||
|
||||
function stripTasksFromConfig(config) {
|
||||
if (!config || typeof config !== 'object') return getDefaultJsonConfig();
|
||||
const { agent, scheduler, email } = config;
|
||||
return normalizeJsonConfig({ agent, scheduler, email });
|
||||
}
|
||||
|
||||
function ensureJsonConfigExists() {
|
||||
if (existsSync(CONFIG_PATH)) return;
|
||||
writeFileSync(CONFIG_PATH, JSON.stringify(getDefaultJsonConfig(), null, 2), 'utf-8');
|
||||
}
|
||||
|
||||
function queryBaseRows({ city, taskId }) {
|
||||
const clauses = [];
|
||||
const params = [];
|
||||
|
||||
if (city) {
|
||||
clauses.push('city = ?');
|
||||
params.push(city);
|
||||
}
|
||||
|
||||
if (taskId) {
|
||||
clauses.push('task_id = ?');
|
||||
params.push(taskId);
|
||||
}
|
||||
|
||||
const whereSql = clauses.length > 0 ? `WHERE ${clauses.join(' AND ')}` : '';
|
||||
const sql = `
|
||||
SELECT id, task_id, city, scraped_at, error, data_json
|
||||
FROM results
|
||||
${whereSql}
|
||||
ORDER BY scraped_at DESC, rowid DESC
|
||||
`;
|
||||
|
||||
return getDb().prepare(sql).all(...params).map(parseResultRow);
|
||||
}
|
||||
|
||||
function matchSection(record, section) {
|
||||
if (!section) return true;
|
||||
if (record.section === section || record.subsection === section) return true;
|
||||
|
||||
const items = record.data?.results || [];
|
||||
return items.some((item) => item.section === section || item.subsection === section);
|
||||
}
|
||||
|
||||
function matchType(record, type) {
|
||||
if (!type) return true;
|
||||
if (record.type === type) return true;
|
||||
|
||||
const items = record.data?.results || [];
|
||||
return items.some((item) => item.type === type);
|
||||
}
|
||||
|
||||
function normalizeProjectName(item) {
|
||||
if (!item || typeof item !== 'object') return '';
|
||||
|
||||
const candidates = [
|
||||
item.project_name,
|
||||
item.projectName,
|
||||
item.title,
|
||||
item.name,
|
||||
item.bidName,
|
||||
];
|
||||
|
||||
for (const candidate of candidates) {
|
||||
if (typeof candidate === 'string' && candidate.trim()) {
|
||||
return candidate.replace(/\s+/g, ' ').trim();
|
||||
}
|
||||
}
|
||||
|
||||
return '';
|
||||
}
|
||||
|
||||
function dedupeRowsByProjectName(rows) {
|
||||
const seenProjectNames = new Set();
|
||||
const dedupedRows = [];
|
||||
|
||||
for (const row of rows) {
|
||||
const items = Array.isArray(row.data?.results) ? row.data.results : [];
|
||||
if (items.length === 0) continue;
|
||||
|
||||
const uniqueItems = [];
|
||||
|
||||
for (const item of items) {
|
||||
const projectName = normalizeProjectName(item);
|
||||
|
||||
if (!projectName) {
|
||||
uniqueItems.push(item);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (seenProjectNames.has(projectName)) continue;
|
||||
|
||||
seenProjectNames.add(projectName);
|
||||
uniqueItems.push(item);
|
||||
}
|
||||
|
||||
if (uniqueItems.length === 0) continue;
|
||||
|
||||
dedupedRows.push({
|
||||
...row,
|
||||
data: {
|
||||
...(row.data || {}),
|
||||
results: uniqueItems,
|
||||
total: uniqueItems.length,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
return dedupedRows;
|
||||
}
|
||||
|
||||
function pickProjectLink(item) {
|
||||
const candidates = [item?.detail_link, item?.target_link, item?.url, item?.href];
|
||||
for (const candidate of candidates) {
|
||||
if (typeof candidate === 'string' && candidate.trim()) return candidate.trim();
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function parseProjectSection(item) {
|
||||
if (typeof item?.section === 'string' && item.section.trim()) return item.section.trim();
|
||||
if (typeof item?.type === 'string' && item.type.trim()) {
|
||||
return item.type.split(/[-/]/)[0].trim();
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function parseProjectAmount(item) {
|
||||
if (typeof item?.amount_yuan === 'number' && Number.isFinite(item.amount_yuan)) {
|
||||
return item.amount_yuan;
|
||||
}
|
||||
|
||||
if (typeof item?.amount === 'number' && Number.isFinite(item.amount)) {
|
||||
return item.amount;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function projectSortValue(project) {
|
||||
const dateValue = Date.parse(project.date || '');
|
||||
const scrapedAtValue = Date.parse(project.scrapedAt || '');
|
||||
|
||||
return {
|
||||
dateValue: Number.isFinite(dateValue) ? dateValue : 0,
|
||||
scrapedAtValue: Number.isFinite(scrapedAtValue) ? scrapedAtValue : 0,
|
||||
};
|
||||
}
|
||||
|
||||
function compareProjectsDesc(a, b) {
|
||||
const aValue = projectSortValue(a);
|
||||
const bValue = projectSortValue(b);
|
||||
|
||||
if (bValue.dateValue !== aValue.dateValue) return bValue.dateValue - aValue.dateValue;
|
||||
if (bValue.scrapedAtValue !== aValue.scrapedAtValue) return bValue.scrapedAtValue - aValue.scrapedAtValue;
|
||||
return (a.projectName || '').localeCompare(b.projectName || '', 'zh-CN');
|
||||
}
|
||||
|
||||
function normalizeSearchText(value) {
|
||||
if (typeof value !== 'string') return '';
|
||||
return value.trim().toLowerCase();
|
||||
}
|
||||
|
||||
function includesSearchText(source, keyword) {
|
||||
if (!keyword) return true;
|
||||
return normalizeSearchText(source).includes(keyword);
|
||||
}
|
||||
|
||||
function parseNumberFilter(value) {
|
||||
if (value === null || value === undefined || value === '') return null;
|
||||
const parsed = Number(value);
|
||||
return Number.isFinite(parsed) ? parsed : null;
|
||||
}
|
||||
|
||||
function parseDateFilter(value) {
|
||||
if (typeof value !== 'string' || !value.trim()) return null;
|
||||
const timestamp = Date.parse(value);
|
||||
return Number.isFinite(timestamp) ? timestamp : null;
|
||||
}
|
||||
|
||||
function matchProjectFilters(project, filters = {}) {
|
||||
const cityKeyword = normalizeSearchText(filters.city);
|
||||
const sectionKeyword = normalizeSearchText(filters.section);
|
||||
const projectNameKeyword = normalizeSearchText(filters.projectName);
|
||||
const minAmount = parseNumberFilter(filters.minAmount);
|
||||
const maxAmount = parseNumberFilter(filters.maxAmount);
|
||||
const startDate = parseDateFilter(filters.startDate);
|
||||
const endDate = parseDateFilter(filters.endDate);
|
||||
const projectDate = parseDateFilter(project.date);
|
||||
|
||||
if (!includesSearchText(project.city, cityKeyword)) return false;
|
||||
if (!includesSearchText(project.section || project.type, sectionKeyword)) return false;
|
||||
if (!includesSearchText(project.projectName, projectNameKeyword)) return false;
|
||||
if (minAmount !== null && (project.amountYuan === null || project.amountYuan < minAmount)) return false;
|
||||
if (maxAmount !== null && (project.amountYuan === null || project.amountYuan > maxAmount)) return false;
|
||||
if (startDate !== null && (projectDate === null || projectDate < startDate)) return false;
|
||||
if (endDate !== null && (projectDate === null || projectDate > endDate)) return false;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
function buildProjectList(rows, {
|
||||
dedupeByName = false,
|
||||
city,
|
||||
section,
|
||||
projectNameKeyword,
|
||||
minAmount,
|
||||
maxAmount,
|
||||
startDate,
|
||||
endDate,
|
||||
} = {}) {
|
||||
const seenProjectNames = new Set();
|
||||
const projects = [];
|
||||
|
||||
for (const row of rows) {
|
||||
const items = Array.isArray(row.data?.results) ? row.data.results : [];
|
||||
|
||||
for (let index = 0; index < items.length; index += 1) {
|
||||
const item = items[index];
|
||||
const projectName = normalizeProjectName(item);
|
||||
if (!projectName) continue;
|
||||
|
||||
const projectSection = parseProjectSection(item);
|
||||
if (city && row.city !== city) continue;
|
||||
if (section && projectSection !== section) continue;
|
||||
|
||||
if (dedupeByName) {
|
||||
if (seenProjectNames.has(projectName)) continue;
|
||||
seenProjectNames.add(projectName);
|
||||
}
|
||||
|
||||
projects.push({
|
||||
id: `${row.id}:${index}`,
|
||||
resultId: row.id,
|
||||
taskId: row.taskId,
|
||||
city: row.city || '',
|
||||
section: projectSection,
|
||||
type: typeof item?.type === 'string' ? item.type : '',
|
||||
projectName,
|
||||
amountYuan: parseProjectAmount(item),
|
||||
date: typeof item?.date === 'string' ? item.date : '',
|
||||
detailLink: pickProjectLink(item),
|
||||
scrapedAt: row.scrapedAt,
|
||||
raw: item,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return projects
|
||||
.filter((project) => matchProjectFilters(project, {
|
||||
city,
|
||||
section,
|
||||
projectName: projectNameKeyword,
|
||||
minAmount,
|
||||
maxAmount,
|
||||
startDate,
|
||||
endDate,
|
||||
}))
|
||||
.sort(compareProjectsDesc);
|
||||
}
|
||||
|
||||
export function initResultsStore() {
|
||||
if (initialized) return;
|
||||
ensureSchema();
|
||||
ensureTasksTableShape();
|
||||
ensureJsonConfigExists();
|
||||
initialized = true;
|
||||
}
|
||||
|
||||
export function loadConfig() {
|
||||
initResultsStore();
|
||||
const jsonConfig = normalizeJsonConfig(readJsonIfExists(CONFIG_PATH) || getDefaultJsonConfig());
|
||||
return {
|
||||
...clone(jsonConfig),
|
||||
tasks: listTasks(),
|
||||
};
|
||||
}
|
||||
|
||||
export function saveConfig(config) {
|
||||
initResultsStore();
|
||||
const jsonConfig = stripTasksFromConfig(config);
|
||||
writeFileSync(CONFIG_PATH, JSON.stringify(jsonConfig, null, 2), 'utf-8');
|
||||
return {
|
||||
...clone(jsonConfig),
|
||||
tasks: listTasks(),
|
||||
};
|
||||
}
|
||||
|
||||
export function listTasks() {
|
||||
initResultsStore();
|
||||
return getDb()
|
||||
.prepare(`
|
||||
SELECT id, city, plate_name, prompt, enabled, mode
|
||||
FROM tasks
|
||||
ORDER BY rowid DESC
|
||||
`)
|
||||
.all()
|
||||
.map(parseTaskRow);
|
||||
}
|
||||
|
||||
export function getTaskById(id) {
|
||||
initResultsStore();
|
||||
const row = getDb()
|
||||
.prepare(`
|
||||
SELECT id, city, plate_name, prompt, enabled, mode
|
||||
FROM tasks
|
||||
WHERE id = ?
|
||||
`)
|
||||
.get(id);
|
||||
|
||||
return row ? parseTaskRow(row) : null;
|
||||
}
|
||||
|
||||
export function createTask(task) {
|
||||
initResultsStore();
|
||||
const record = buildTaskRecord(task);
|
||||
getDb().prepare(`
|
||||
INSERT INTO tasks (id, city, plate_name, prompt, enabled, mode, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, datetime('now'))
|
||||
`).run(
|
||||
record.id,
|
||||
record.city,
|
||||
record.plateName,
|
||||
record.prompt,
|
||||
record.enabled ? 1 : 0,
|
||||
record.mode,
|
||||
);
|
||||
return record;
|
||||
}
|
||||
|
||||
export function updateTask(id, patch) {
|
||||
initResultsStore();
|
||||
const current = getTaskById(id);
|
||||
if (!current) return null;
|
||||
|
||||
const next = buildTaskRecord({ ...current, ...patch, id });
|
||||
getDb().prepare(`
|
||||
UPDATE tasks
|
||||
SET city = ?,
|
||||
plate_name = ?,
|
||||
prompt = ?,
|
||||
enabled = ?,
|
||||
mode = ?,
|
||||
updated_at = datetime('now')
|
||||
WHERE id = ?
|
||||
`).run(
|
||||
next.city,
|
||||
next.plateName,
|
||||
next.prompt,
|
||||
next.enabled ? 1 : 0,
|
||||
next.mode,
|
||||
id,
|
||||
);
|
||||
|
||||
return next;
|
||||
}
|
||||
|
||||
export function deleteTaskById(id) {
|
||||
initResultsStore();
|
||||
const result = getDb().prepare('DELETE FROM tasks WHERE id = ?').run(id);
|
||||
return result.changes > 0;
|
||||
}
|
||||
|
||||
export function appendResult(result) {
|
||||
initResultsStore();
|
||||
const record = buildResultRecord(result);
|
||||
|
||||
getDb().prepare(`
|
||||
INSERT INTO results (id, task_id, city, scraped_at, error, data_json)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
`).run(
|
||||
record.id,
|
||||
record.taskId,
|
||||
record.city,
|
||||
record.scrapedAt,
|
||||
record.error,
|
||||
record.data === null ? null : JSON.stringify(record.data),
|
||||
);
|
||||
|
||||
trimResults();
|
||||
return record;
|
||||
}
|
||||
|
||||
export function listResults({ city, section, type, taskId, page = 1, pageSize = 20, projectMode = false } = {}) {
|
||||
initResultsStore();
|
||||
|
||||
let results = queryBaseRows({ city, taskId });
|
||||
if (section) results = results.filter((record) => matchSection(record, section));
|
||||
if (type) results = results.filter((record) => matchType(record, type));
|
||||
if (projectMode) results = dedupeRowsByProjectName(results);
|
||||
|
||||
const normalizedPage = Math.max(1, parseInt(page, 10) || 1);
|
||||
const normalizedPageSize = Math.max(1, parseInt(pageSize, 10) || 20);
|
||||
const start = (normalizedPage - 1) * normalizedPageSize;
|
||||
|
||||
return {
|
||||
total: results.length,
|
||||
page: normalizedPage,
|
||||
pageSize: normalizedPageSize,
|
||||
data: results.slice(start, start + normalizedPageSize),
|
||||
};
|
||||
}
|
||||
|
||||
export function deleteResultById(id) {
|
||||
initResultsStore();
|
||||
const result = getDb().prepare('DELETE FROM results WHERE id = ?').run(id);
|
||||
return result.changes > 0;
|
||||
}
|
||||
|
||||
export function clearResults() {
|
||||
initResultsStore();
|
||||
getDb().prepare('DELETE FROM results').run();
|
||||
}
|
||||
|
||||
export function getResultFilters({ projectMode = false } = {}) {
|
||||
initResultsStore();
|
||||
|
||||
const rows = projectMode ? dedupeRowsByProjectName(queryBaseRows({})) : queryBaseRows({});
|
||||
const cities = [...new Set(rows.map((row) => row.city).filter(Boolean))];
|
||||
const sections = new Set();
|
||||
const types = new Set();
|
||||
|
||||
for (const row of rows) {
|
||||
if (row.section) sections.add(row.section);
|
||||
if (row.subsection) sections.add(row.subsection);
|
||||
if (row.type) types.add(row.type);
|
||||
|
||||
for (const item of row.data?.results || []) {
|
||||
if (item.section) sections.add(item.section);
|
||||
if (item.subsection) sections.add(item.subsection);
|
||||
if (item.type) types.add(item.type);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
cities,
|
||||
sections: [...sections],
|
||||
types: [...types],
|
||||
};
|
||||
}
|
||||
|
||||
export function listProjects({
|
||||
city,
|
||||
section,
|
||||
projectName,
|
||||
minAmount,
|
||||
maxAmount,
|
||||
startDate,
|
||||
endDate,
|
||||
page = 1,
|
||||
pageSize = 20,
|
||||
dedupeByName = true,
|
||||
} = {}) {
|
||||
initResultsStore();
|
||||
|
||||
const rows = queryBaseRows({});
|
||||
const projects = buildProjectList(rows, {
|
||||
dedupeByName,
|
||||
city,
|
||||
section,
|
||||
projectNameKeyword: projectName,
|
||||
minAmount,
|
||||
maxAmount,
|
||||
startDate,
|
||||
endDate,
|
||||
});
|
||||
const normalizedPage = Math.max(1, parseInt(page, 10) || 1);
|
||||
const normalizedPageSize = Math.max(1, parseInt(pageSize, 10) || 20);
|
||||
const start = (normalizedPage - 1) * normalizedPageSize;
|
||||
|
||||
return {
|
||||
total: projects.length,
|
||||
page: normalizedPage,
|
||||
pageSize: normalizedPageSize,
|
||||
data: projects.slice(start, start + normalizedPageSize),
|
||||
};
|
||||
}
|
||||
|
||||
export function getProjectFilters({ dedupeByName = true } = {}) {
|
||||
initResultsStore();
|
||||
|
||||
const projects = buildProjectList(queryBaseRows({}), { dedupeByName });
|
||||
const cities = [...new Set(projects.map((project) => project.city).filter(Boolean))];
|
||||
const sections = [...new Set(projects.map((project) => project.section).filter(Boolean))];
|
||||
|
||||
return {
|
||||
cities,
|
||||
sections,
|
||||
};
|
||||
}
|
||||
|
||||
export function getResultsDbPath() {
|
||||
return DB_PATH;
|
||||
}
|
||||
429
src/scheduler.js
429
src/scheduler.js
@@ -1,396 +1,177 @@
|
||||
import 'dotenv/config';
|
||||
import cron from 'node-cron';
|
||||
import { readFileSync } from 'fs';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { dirname, join } from 'path';
|
||||
import axios from 'axios';
|
||||
import * as cheerio from 'cheerio';
|
||||
import iconv from 'iconv-lite';
|
||||
import { sendReportEmail } from './emailService.js';
|
||||
import { sendScraperResultsEmail } from './emailService.js';
|
||||
import { runAgentTask } from './agentService.js';
|
||||
import { initResultsStore, loadConfig, appendResult } from './resultStore.js';
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = dirname(__filename);
|
||||
const DEFAULT_TASK_MODE = 'qwen3.5-plus';
|
||||
|
||||
// 加载配置文件
|
||||
function loadConfig() {
|
||||
try {
|
||||
const configPath = join(__dirname, '..', 'config.json');
|
||||
const configContent = readFileSync(configPath, 'utf-8');
|
||||
return JSON.parse(configContent);
|
||||
} catch (error) {
|
||||
console.error('加载配置文件失败:', error.message);
|
||||
console.error('请确保 config.json 文件存在并配置正确');
|
||||
return null;
|
||||
}
|
||||
let currentScheduledTask = null;
|
||||
|
||||
function normalizeTaskMode(value) {
|
||||
if (typeof value === 'string' && value.trim()) return value.trim();
|
||||
return DEFAULT_TASK_MODE;
|
||||
}
|
||||
|
||||
// 根据时间范围类型获取开始和结束日期
|
||||
function getDateRangeByType(timeRange) {
|
||||
const now = new Date();
|
||||
const year = now.getFullYear();
|
||||
const month = String(now.getMonth() + 1).padStart(2, '0');
|
||||
const day = String(now.getDate()).padStart(2, '0');
|
||||
async function runTask(task, agentCfg) {
|
||||
const mode = normalizeTaskMode(task.mode);
|
||||
|
||||
let startDate, endDate;
|
||||
endDate = `${year}-${month}-${day}`; // 结束日期都是今天
|
||||
console.log(`[Scheduler][Agent] ${task.city}: start`);
|
||||
console.log(`[Scheduler][Agent] ${task.city}: mode=${mode}`);
|
||||
|
||||
switch (timeRange) {
|
||||
case 'today':
|
||||
// 今日
|
||||
startDate = `${year}-${month}-${day}`;
|
||||
break;
|
||||
|
||||
case 'thisWeek': {
|
||||
// 本周 (从周一开始)
|
||||
const dayOfWeek = now.getDay(); // 0是周日,1是周一
|
||||
const diff = dayOfWeek === 0 ? 6 : dayOfWeek - 1; // 计算到周一的天数差
|
||||
const monday = new Date(now);
|
||||
monday.setDate(now.getDate() - diff);
|
||||
const weekYear = monday.getFullYear();
|
||||
const weekMonth = String(monday.getMonth() + 1).padStart(2, '0');
|
||||
const weekDay = String(monday.getDate()).padStart(2, '0');
|
||||
startDate = `${weekYear}-${weekMonth}-${weekDay}`;
|
||||
break;
|
||||
}
|
||||
|
||||
case 'thisMonth':
|
||||
default:
|
||||
// 本月
|
||||
startDate = `${year}-${month}-01`;
|
||||
break;
|
||||
}
|
||||
|
||||
return { startDate, endDate };
|
||||
}
|
||||
|
||||
// 南京市公共资源交易平台 - 房建市政招标公告
|
||||
const BASE_URL = 'https://njggzy.nanjing.gov.cn/njweb/fjsz/068001/068001002/';
|
||||
|
||||
const http = axios.create({
|
||||
responseType: 'arraybuffer',
|
||||
timeout: 15000,
|
||||
headers: {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
|
||||
},
|
||||
});
|
||||
|
||||
function pickEncoding(contentType = '') {
|
||||
const match = /charset=([^;]+)/i.exec(contentType);
|
||||
if (!match) return 'utf-8';
|
||||
const charset = match[1].trim().toLowerCase();
|
||||
if (charset.includes('gb')) return 'gbk';
|
||||
return charset;
|
||||
}
|
||||
|
||||
async function fetchHtml(url) {
|
||||
const res = await http.get(url);
|
||||
const encoding = pickEncoding(res.headers['content-type']);
|
||||
const html = iconv.decode(res.data, encoding || 'utf-8');
|
||||
return html;
|
||||
}
|
||||
|
||||
function getPageUrl(pageIndex) {
|
||||
if (pageIndex === 1) {
|
||||
return `${BASE_URL}moreinfo.html`;
|
||||
}
|
||||
return `${BASE_URL}${pageIndex}.html`;
|
||||
}
|
||||
|
||||
// 解析列表页HTML，提取招标信息
|
||||
function parseList(html) {
|
||||
const $ = cheerio.load(html);
|
||||
const items = [];
|
||||
|
||||
$('li.ewb-info-item2').each((_, row) => {
|
||||
const $row = $(row);
|
||||
const cells = $row.find('div.ewb-info-num2');
|
||||
|
||||
if (cells.length >= 5) {
|
||||
const bidNo = $(cells[0]).find('p').attr('title') || $(cells[0]).find('p').text().trim();
|
||||
const projectName = $(cells[1]).find('p').attr('title') || $(cells[1]).find('p').text().trim();
|
||||
const bidName = $(cells[2]).find('p').attr('title') || $(cells[2]).find('p').text().trim();
|
||||
const estimatedPrice = $(cells[3]).find('p').text().trim();
|
||||
const publishDate = $(cells[4]).find('p').text().trim();
|
||||
|
||||
const onclick = $row.attr('onclick') || '';
|
||||
const hrefMatch = onclick.match(/window\.open\(['"]([^'"]+)['"]\)/);
|
||||
let href = '';
|
||||
if (hrefMatch) {
|
||||
href = hrefMatch[1];
|
||||
if (href.startsWith('/')) {
|
||||
href = `https://njggzy.nanjing.gov.cn${href}`;
|
||||
}
|
||||
}
|
||||
|
||||
if (!/^\d{4}-\d{2}-\d{2}$/.test(publishDate)) return;
|
||||
|
||||
const price = parseFloat(estimatedPrice);
|
||||
if (isNaN(price)) return;
|
||||
|
||||
items.push({
|
||||
bidNo,
|
||||
title: projectName,
|
||||
bidName,
|
||||
budget: {
|
||||
amount: price,
|
||||
unit: '万元'
|
||||
},
|
||||
date: publishDate,
|
||||
href
|
||||
});
|
||||
}
|
||||
const { results } = await runAgentTask(task.prompt, {
|
||||
baseUrl: agentCfg.baseUrl,
|
||||
mode,
|
||||
pollInterval: agentCfg.pollInterval,
|
||||
timeout: agentCfg.timeout,
|
||||
logPrefix: `[Scheduler][Agent][${task.city}]`,
|
||||
});
|
||||
|
||||
return items;
|
||||
console.log(`[Scheduler][Agent] ${task.city}: ${results.length} results`);
|
||||
|
||||
return appendResult({
|
||||
taskId: task.id,
|
||||
city: task.city,
|
||||
scrapedAt: new Date().toISOString(),
|
||||
data: { results, total: results.length },
|
||||
});
|
||||
}
|
||||
|
||||
function isDateInRange(dateStr, startDate, endDate) {
|
||||
if (!dateStr) return false;
|
||||
const date = new Date(dateStr);
|
||||
if (isNaN(date.getTime())) return false;
|
||||
|
||||
if (startDate && date < new Date(startDate)) return false;
|
||||
if (endDate && date > new Date(endDate)) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
async function fetchListByDateRange(startDate, endDate, maxPages = 50) {
|
||||
const allItems = [];
|
||||
let shouldContinue = true;
|
||||
let pageIndex = 1;
|
||||
|
||||
console.log(`开始按时间范围采集: ${startDate || '不限'} 至 ${endDate || '不限'}`);
|
||||
|
||||
while (shouldContinue && pageIndex <= maxPages) {
|
||||
const pageUrl = getPageUrl(pageIndex);
|
||||
console.log(`正在采集第 ${pageIndex} 页: ${pageUrl}`);
|
||||
|
||||
try {
|
||||
const html = await fetchHtml(pageUrl);
|
||||
const items = parseList(html);
|
||||
|
||||
if (items.length === 0) {
|
||||
console.log(`第 ${pageIndex} 页没有数据，停止采集`);
|
||||
break;
|
||||
}
|
||||
|
||||
let hasItemsInRange = false;
|
||||
let allItemsBeforeRange = true;
|
||||
|
||||
for (const item of items) {
|
||||
if (isDateInRange(item.date, startDate, endDate)) {
|
||||
allItems.push(item);
|
||||
hasItemsInRange = true;
|
||||
allItemsBeforeRange = false;
|
||||
} else if (startDate && new Date(item.date) < new Date(startDate)) {
|
||||
allItemsBeforeRange = allItemsBeforeRange && true;
|
||||
} else {
|
||||
allItemsBeforeRange = false;
|
||||
}
|
||||
}
|
||||
|
||||
if (allItemsBeforeRange && startDate) {
|
||||
console.log(`第 ${pageIndex} 页所有项目都早于起始日期，停止采集`);
|
||||
shouldContinue = false;
|
||||
}
|
||||
|
||||
console.log(`第 ${pageIndex} 页找到 ${items.length} 条，符合条件 ${hasItemsInRange ? '有' : '无'}`);
|
||||
|
||||
pageIndex++;
|
||||
|
||||
if (shouldContinue && pageIndex <= maxPages) {
|
||||
await new Promise(resolve => setTimeout(resolve, 500));
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`采集第 ${pageIndex} 页失败: ${err.message}`);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`总共采集了 ${pageIndex - 1} 页，找到 ${allItems.length} 条符合条件的公告`);
|
||||
return allItems;
|
||||
}
|
||||
|
||||
// 定时任务执行函数
|
||||
async function executeScheduledTask(config) {
|
||||
try {
|
||||
console.log('========================================');
|
||||
console.log('定时任务开始执行');
|
||||
console.log('执行时间:', new Date().toLocaleString('zh-CN'));
|
||||
console.log('Scheduler started');
|
||||
console.log('Time:', new Date().toLocaleString('zh-CN'));
|
||||
console.log('========================================');
|
||||
|
||||
const timeRange = config.scheduler.timeRange || 'thisMonth';
|
||||
const { startDate, endDate } = getDateRangeByType(timeRange);
|
||||
const threshold = config.scheduler.threshold || 10000; // 默认1亿(10000万元)
|
||||
const tasks = (config.tasks || []).filter((task) => task.enabled);
|
||||
const agentCfg = config.agent || {};
|
||||
|
||||
const timeRangeNames = {
|
||||
'today': '今日',
|
||||
'thisWeek': '本周',
|
||||
'thisMonth': '本月'
|
||||
};
|
||||
console.log(`采集时间段: ${timeRangeNames[timeRange] || '本月'}`);
|
||||
console.log(`采集时间范围: ${startDate} 至 ${endDate}`);
|
||||
console.log(`金额阈值: ${threshold}万元 (${(threshold / 10000).toFixed(2)}亿元)`);
|
||||
|
||||
// 采集列表（直接包含合同估算价）
|
||||
const items = await fetchListByDateRange(startDate, endDate, 50);
|
||||
|
||||
if (items.length === 0) {
|
||||
console.log('暂无公告数据');
|
||||
if (tasks.length === 0) {
|
||||
console.log('No enabled tasks, skip');
|
||||
return;
|
||||
}
|
||||
|
||||
// 筛选大于阈值的项目
|
||||
const filtered = items.filter((item) => {
|
||||
return item.budget && item.budget.amount > threshold;
|
||||
});
|
||||
console.log(`Enabled tasks: ${tasks.length}`);
|
||||
|
||||
console.log('========================================');
|
||||
console.log(`筛选结果: 找到 ${filtered.length} 个大于 ${threshold}万元 的项目`);
|
||||
const results = [];
|
||||
for (const task of tasks) {
|
||||
try {
|
||||
console.log(`\n---------- Task: ${task.city} ----------`);
|
||||
const record = await runTask(task, agentCfg);
|
||||
results.push(record);
|
||||
console.log('Task completed');
|
||||
} catch (error) {
|
||||
console.error(`Task failed: ${error.message}`);
|
||||
const errorRecord = appendResult({
|
||||
taskId: task.id,
|
||||
city: task.city,
|
||||
scrapedAt: new Date().toISOString(),
|
||||
error: error.message,
|
||||
data: null,
|
||||
});
|
||||
results.push(errorRecord);
|
||||
}
|
||||
}
|
||||
|
||||
if (filtered.length === 0) {
|
||||
console.log('暂无符合条件的大额项目');
|
||||
const successCount = results.filter((item) => !item.error).length;
|
||||
const failCount = results.filter((item) => item.error).length;
|
||||
console.log('\n========== Scheduler finished ==========');
|
||||
console.log(`Success: ${successCount}, Failed: ${failCount}`);
|
||||
|
||||
if (successCount === 0) {
|
||||
console.log('No successful results, skip email');
|
||||
return;
|
||||
}
|
||||
|
||||
// 计算总金额
|
||||
const total = filtered.reduce(
|
||||
(sum, item) => sum + (item.budget?.amount || 0),
|
||||
0
|
||||
);
|
||||
if (config.email?.smtpHost && config.email?.smtpUser) {
|
||||
console.log('\nSending email...');
|
||||
try {
|
||||
const emailResult = await sendScraperResultsEmail(config.email, results);
|
||||
console.log('Email sent:', emailResult.messageId);
|
||||
} catch (error) {
|
||||
console.error('Email failed:', error.message);
|
||||
}
|
||||
} else {
|
||||
console.log('Email config incomplete, skip email');
|
||||
}
|
||||
|
||||
// 生成报告
|
||||
const report = {
|
||||
summary: {
|
||||
total_count: items.length,
|
||||
filtered_count: filtered.length,
|
||||
threshold: `${threshold}万元`,
|
||||
total_amount: `${total.toFixed(2)}万元`,
|
||||
generated_at: new Date().toISOString(),
|
||||
date_range: { startDate, endDate },
|
||||
},
|
||||
projects: filtered.map((item) => ({
|
||||
bidNo: item.bidNo,
|
||||
title: item.title,
|
||||
bidName: item.bidName,
|
||||
date: item.date,
|
||||
budget: item.budget,
|
||||
url: item.href,
|
||||
})),
|
||||
};
|
||||
|
||||
// 发送邮件
|
||||
console.log('========================================');
|
||||
console.log('正在发送邮件报告...');
|
||||
const emailConfig = config.email;
|
||||
|
||||
const result = await sendReportEmail(emailConfig, report);
|
||||
|
||||
console.log('邮件发送成功!');
|
||||
console.log('收件人:', emailConfig.recipients);
|
||||
console.log('MessageId:', result.messageId);
|
||||
console.log('========================================');
|
||||
console.log('定时任务执行完成');
|
||||
console.log('========================================');
|
||||
|
||||
} catch (error) {
|
||||
console.error('========================================');
|
||||
console.error('定时任务执行失败:', error.message);
|
||||
console.error('Scheduler failed:', error.message);
|
||||
console.error(error.stack);
|
||||
console.error('========================================');
|
||||
}
|
||||
}
|
||||
|
||||
// 存储当前的定时任务
|
||||
let currentScheduledTask = null;
|
||||
|
||||
// 初始化定时任务
|
||||
export function initScheduler() {
|
||||
initResultsStore();
|
||||
|
||||
const config = loadConfig();
|
||||
|
||||
if (!config) {
|
||||
console.error('无法启动定时任务: 配置文件加载失败');
|
||||
return;
|
||||
}
|
||||
|
||||
if (!config.scheduler || !config.scheduler.enabled) {
|
||||
console.log('定时任务已禁用');
|
||||
return;
|
||||
}
|
||||
|
||||
if (!config.email || !config.email.smtpHost || !config.email.smtpUser) {
|
||||
console.error('无法启动定时任务: 邮件配置不完整');
|
||||
console.error('请在 config.json 中配置邮件信息');
|
||||
if (!config.scheduler?.enabled) {
|
||||
console.log('Scheduler disabled');
|
||||
return;
|
||||
}
|
||||
|
||||
const cronTime = config.scheduler.cronTime || '0 9 * * *';
|
||||
const enabledCount = (config.tasks || []).filter((task) => task.enabled).length;
|
||||
|
||||
console.log('========================================');
|
||||
console.log('定时任务已启动');
|
||||
console.log('执行计划:', cronTime);
|
||||
console.log('金额阈值:', config.scheduler.threshold, '万元');
|
||||
console.log('收件人:', config.email.recipients);
|
||||
console.log('Scheduler enabled:', cronTime);
|
||||
console.log(`Enabled tasks: ${enabledCount}`);
|
||||
if (config.email?.recipients) {
|
||||
console.log('Recipients:', config.email.recipients);
|
||||
}
|
||||
console.log('========================================');
|
||||
|
||||
// 如果已有任务在运行,先停止
|
||||
if (currentScheduledTask) {
|
||||
currentScheduledTask.stop();
|
||||
console.log('已停止旧的定时任务');
|
||||
}
|
||||
|
||||
// 创建定时任务
|
||||
currentScheduledTask = cron.schedule(cronTime, () => {
|
||||
executeScheduledTask(config);
|
||||
}, {
|
||||
timezone: 'Asia/Shanghai'
|
||||
});
|
||||
currentScheduledTask = cron.schedule(
|
||||
cronTime,
|
||||
() => {
|
||||
executeScheduledTask(loadConfig());
|
||||
},
|
||||
{ timezone: 'Asia/Shanghai' },
|
||||
);
|
||||
}
|
||||
|
||||
// 重新加载配置并重启定时任务
|
||||
export function reloadScheduler() {
|
||||
console.log('重新加载定时任务配置...');
|
||||
|
||||
// 停止当前任务
|
||||
console.log('Reloading scheduler...');
|
||||
if (currentScheduledTask) {
|
||||
currentScheduledTask.stop();
|
||||
currentScheduledTask = null;
|
||||
console.log('已停止当前定时任务');
|
||||
}
|
||||
|
||||
// 重新初始化
|
||||
initScheduler();
|
||||
}
|
||||
|
||||
// 停止定时任务
|
||||
export function stopScheduler() {
|
||||
if (currentScheduledTask) {
|
||||
currentScheduledTask.stop();
|
||||
currentScheduledTask = null;
|
||||
console.log('定时任务已停止');
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
if (!currentScheduledTask) return false;
|
||||
|
||||
currentScheduledTask.stop();
|
||||
currentScheduledTask = null;
|
||||
console.log('Scheduler stopped');
|
||||
return true;
|
||||
}
|
||||
|
||||
// 获取定时任务状态
|
||||
export function getSchedulerStatus() {
|
||||
const config = loadConfig();
|
||||
const enabledTasks = (config.tasks || []).filter((task) => task.enabled).length;
|
||||
|
||||
return {
|
||||
isRunning: currentScheduledTask !== null,
|
||||
config: config ? {
|
||||
enabledTasks,
|
||||
config: {
|
||||
enabled: config.scheduler?.enabled || false,
|
||||
cronTime: config.scheduler?.cronTime || '0 9 * * *',
|
||||
threshold: config.scheduler?.threshold || 10000,
|
||||
timeRange: config.scheduler?.timeRange || 'thisMonth',
|
||||
} : null,
|
||||
description: config.scheduler?.description || '',
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// 手动执行任务(用于测试)
|
||||
export async function runTaskNow() {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
throw new Error('配置文件加载失败');
|
||||
}
|
||||
await executeScheduledTask(config);
|
||||
initResultsStore();
|
||||
await executeScheduledTask(loadConfig());
|
||||
}
|
||||
|
||||
785
src/server.js
785
src/server.js
@@ -1,474 +1,475 @@
|
||||
import 'dotenv/config';
|
||||
import express from 'express';
|
||||
import cors from 'cors';
|
||||
import axios from 'axios';
|
||||
import * as cheerio from 'cheerio';
|
||||
import iconv from 'iconv-lite';
|
||||
import { sendReportEmail } from './emailService.js';
|
||||
import { existsSync } from 'node:fs';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import {
|
||||
initResultsStore,
|
||||
loadConfig,
|
||||
saveConfig,
|
||||
listTasks,
|
||||
getTaskById,
|
||||
createTask,
|
||||
updateTask,
|
||||
deleteTaskById,
|
||||
listResults,
|
||||
listProjects,
|
||||
deleteResultById,
|
||||
clearResults,
|
||||
getResultFilters,
|
||||
getProjectFilters,
|
||||
appendResult,
|
||||
} from './resultStore.js';
|
||||
import { initScheduler, runTaskNow, reloadScheduler, getSchedulerStatus } from './scheduler.js';
|
||||
import { runAgentTask } from './agentService.js';
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = dirname(__filename);
|
||||
const DIST_DIR = join(__dirname, '..', 'dist');
|
||||
const DIST_INDEX = join(DIST_DIR, 'index.html');
|
||||
const HAS_DIST = existsSync(DIST_INDEX);
|
||||
|
||||
const app = express();
|
||||
const PORT = process.env.PORT || 5000;
|
||||
const DEFAULT_TASK_MODE = 'qwen3.5-plus';
|
||||
const MASKED_PASSWORD = '***已配置***';
|
||||
|
||||
app.use(cors());
|
||||
app.use(express.json());
|
||||
app.use(express.static('public'));
|
||||
app.use(express.json({ limit: '50mb' }));
|
||||
app.use(express.urlencoded({ limit: '50mb', extended: true }));
|
||||
|
||||
// 南京市公共资源交易平台 - 房建市政招标公告
|
||||
const BASE_URL = 'https://njggzy.nanjing.gov.cn/njweb/fjsz/068001/068001002/';
|
||||
|
||||
// 获取分页URL
|
||||
function getPageUrl(pageIndex) {
|
||||
if (pageIndex === 1) {
|
||||
return `${BASE_URL}moreinfo.html`;
|
||||
}
|
||||
return `${BASE_URL}${pageIndex}.html`;
|
||||
if (HAS_DIST) {
|
||||
app.use(express.static(DIST_DIR));
|
||||
}
|
||||
|
||||
// 检查日期是否在范围内
|
||||
function isDateInRange(dateStr, startDate, endDate) {
|
||||
if (!dateStr) return false;
|
||||
const date = new Date(dateStr);
|
||||
if (isNaN(date.getTime())) return false;
|
||||
|
||||
if (startDate && date < new Date(startDate)) return false;
|
||||
if (endDate && date > new Date(endDate)) return false;
|
||||
return true;
|
||||
function normalizeTaskMode(value) {
|
||||
if (typeof value === 'string' && value.trim()) return value.trim();
|
||||
return DEFAULT_TASK_MODE;
|
||||
}
|
||||
|
||||
// 按时间范围采集多页列表
|
||||
async function fetchListByDateRange(startDate, endDate, maxPages = 50) {
|
||||
const allItems = [];
|
||||
let shouldContinue = true;
|
||||
let pageIndex = 1;
|
||||
function buildTaskPayload(body = {}, { partial = false } = {}) {
|
||||
const payload = {};
|
||||
|
||||
console.log(`开始按时间范围采集: ${startDate || '不限'} 至 ${endDate || '不限'}`);
|
||||
|
||||
while (shouldContinue && pageIndex <= maxPages) {
|
||||
const pageUrl = getPageUrl(pageIndex);
|
||||
console.log(`正在采集第 ${pageIndex} 页: ${pageUrl}`);
|
||||
|
||||
try {
|
||||
const html = await fetchHtml(pageUrl);
|
||||
const items = parseList(html);
|
||||
|
||||
if (items.length === 0) {
|
||||
console.log(`第 ${pageIndex} 页没有数据，停止采集`);
|
||||
break;
|
||||
}
|
||||
|
||||
let hasItemsInRange = false;
|
||||
let allItemsBeforeRange = true;
|
||||
|
||||
for (const item of items) {
|
||||
if (isDateInRange(item.date, startDate, endDate)) {
|
||||
allItems.push(item);
|
||||
hasItemsInRange = true;
|
||||
allItemsBeforeRange = false;
|
||||
} else if (startDate && new Date(item.date) < new Date(startDate)) {
|
||||
allItemsBeforeRange = allItemsBeforeRange && true;
|
||||
} else {
|
||||
allItemsBeforeRange = false;
|
||||
}
|
||||
}
|
||||
|
||||
if (allItemsBeforeRange && startDate) {
|
||||
console.log(`第 ${pageIndex} 页所有项目都早于起始日期，停止采集`);
|
||||
shouldContinue = false;
|
||||
}
|
||||
|
||||
console.log(`第 ${pageIndex} 页找到 ${items.length} 条，符合条件 ${hasItemsInRange ? '有' : '无'}`);
|
||||
|
||||
pageIndex++;
|
||||
|
||||
if (shouldContinue && pageIndex <= maxPages) {
|
||||
await new Promise(resolve => setTimeout(resolve, 500));
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`采集第 ${pageIndex} 页失败: ${err.message}`);
|
||||
break;
|
||||
}
|
||||
if (!partial || Object.prototype.hasOwnProperty.call(body, 'city')) {
|
||||
payload.city = body.city || '';
|
||||
}
|
||||
|
||||
console.log(`总共采集了 ${pageIndex - 1} 页，找到 ${allItems.length} 条符合条件的公告`);
|
||||
return allItems;
|
||||
if (!partial || Object.prototype.hasOwnProperty.call(body, 'plateName')) {
|
||||
payload.plateName = body.plateName || '';
|
||||
}
|
||||
|
||||
if (!partial || Object.prototype.hasOwnProperty.call(body, 'prompt')) {
|
||||
payload.prompt = body.prompt || '';
|
||||
}
|
||||
|
||||
if (!partial || Object.prototype.hasOwnProperty.call(body, 'enabled')) {
|
||||
payload.enabled = body.enabled !== false;
|
||||
}
|
||||
|
||||
if (!partial || Object.prototype.hasOwnProperty.call(body, 'mode')) {
|
||||
payload.mode = normalizeTaskMode(body.mode);
|
||||
}
|
||||
|
||||
return payload;
|
||||
}
|
||||
|
||||
const http = axios.create({
|
||||
responseType: 'arraybuffer',
|
||||
timeout: 15000,
|
||||
headers: {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
|
||||
},
|
||||
});
|
||||
function maskConfigSecrets(config) {
|
||||
const next = { ...config };
|
||||
|
||||
function pickEncoding(contentType = '') {
|
||||
const match = /charset=([^;]+)/i.exec(contentType);
|
||||
if (!match) return 'utf-8';
|
||||
const charset = match[1].trim().toLowerCase();
|
||||
if (charset.includes('gb')) return 'gbk';
|
||||
return charset;
|
||||
if (config.email) {
|
||||
next.email = {
|
||||
...config.email,
|
||||
smtpPass: config.email.smtpPass ? MASKED_PASSWORD : '',
|
||||
};
|
||||
}
|
||||
|
||||
return next;
|
||||
}
|
||||
|
||||
async function fetchHtml(url) {
|
||||
const res = await http.get(url);
|
||||
const encoding = pickEncoding(res.headers['content-type']);
|
||||
const html = iconv.decode(res.data, encoding || 'utf-8');
|
||||
return html;
|
||||
function mergeConfigWithExistingSecrets(incoming = {}) {
|
||||
const current = loadConfig();
|
||||
const next = {
|
||||
...current,
|
||||
...incoming,
|
||||
agent: { ...(current.agent || {}), ...(incoming.agent || {}) },
|
||||
scheduler: { ...(current.scheduler || {}), ...(incoming.scheduler || {}) },
|
||||
email: { ...(current.email || {}), ...(incoming.email || {}) },
|
||||
};
|
||||
|
||||
if (next.email?.smtpPass === MASKED_PASSWORD) {
|
||||
next.email.smtpPass = current.email?.smtpPass || '';
|
||||
}
|
||||
|
||||
return next;
|
||||
}
|
||||
|
||||
// 解析列表页HTML，提取招标信息
|
||||
function parseList(html) {
|
||||
const $ = cheerio.load(html);
|
||||
const items = [];
|
||||
let isRunning = false;
|
||||
let runningStatus = null;
|
||||
|
||||
// 解析南京公共资源交易平台的列表结构
|
||||
// <li class="ewb-info-item2 clearfix" onclick="window.open('详情URL');">
|
||||
$('li.ewb-info-item2').each((_, row) => {
|
||||
const $row = $(row);
|
||||
const cells = $row.find('div.ewb-info-num2');
|
||||
async function runTask(task) {
|
||||
const config = loadConfig();
|
||||
const agentCfg = config.agent || {};
|
||||
const mode = normalizeTaskMode(task.mode);
|
||||
|
||||
if (cells.length >= 5) {
|
||||
// 获取各字段
|
||||
const bidNo = $(cells[0]).find('p').attr('title') || $(cells[0]).find('p').text().trim();
|
||||
const projectName = $(cells[1]).find('p').attr('title') || $(cells[1]).find('p').text().trim();
|
||||
const bidName = $(cells[2]).find('p').attr('title') || $(cells[2]).find('p').text().trim();
|
||||
const estimatedPrice = $(cells[3]).find('p').text().trim();
|
||||
const publishDate = $(cells[4]).find('p').text().trim();
|
||||
console.log(`[Agent] ${task.city}: start`);
|
||||
console.log(`[Agent] ${task.city}: mode=${mode}`);
|
||||
|
||||
// 从onclick提取详情链接
|
||||
const onclick = $row.attr('onclick') || '';
|
||||
const hrefMatch = onclick.match(/window\.open\(['"]([^'"]+)['"]\)/);
|
||||
let href = '';
|
||||
if (hrefMatch) {
|
||||
href = hrefMatch[1];
|
||||
// 转换为绝对URL
|
||||
if (href.startsWith('/')) {
|
||||
href = `https://njggzy.nanjing.gov.cn${href}`;
|
||||
}
|
||||
}
|
||||
|
||||
// 验证日期格式 (YYYY-MM-DD)
|
||||
if (!/^\d{4}-\d{2}-\d{2}$/.test(publishDate)) return;
|
||||
|
||||
// 解析合同估算价
|
||||
const price = parseFloat(estimatedPrice);
|
||||
if (isNaN(price)) return;
|
||||
|
||||
items.push({
|
||||
bidNo, // 标段编号
|
||||
title: projectName, // 项目名称
|
||||
bidName, // 标段名称
|
||||
budget: {
|
||||
amount: price,
|
||||
unit: '万元'
|
||||
},
|
||||
date: publishDate,
|
||||
href
|
||||
});
|
||||
}
|
||||
const { results } = await runAgentTask(task.prompt, {
|
||||
baseUrl: agentCfg.baseUrl,
|
||||
mode,
|
||||
pollInterval: agentCfg.pollInterval,
|
||||
timeout: agentCfg.timeout,
|
||||
logPrefix: `[Agent][${task.city}]`,
|
||||
});
|
||||
|
||||
return items;
|
||||
return appendResult({
|
||||
taskId: task.id,
|
||||
city: task.city,
|
||||
scrapedAt: new Date().toISOString(),
|
||||
data: { results, total: results.length },
|
||||
});
|
||||
}
|
||||
|
||||
// API 路由
|
||||
function runTaskInBackground(task) {
|
||||
runningStatus = {
|
||||
taskId: task.id,
|
||||
city: task.city,
|
||||
startTime: Date.now(),
|
||||
current: 0,
|
||||
total: 1,
|
||||
finished: false,
|
||||
error: null,
|
||||
};
|
||||
|
||||
// 获取列表
|
||||
app.get('/api/list', async (req, res) => {
|
||||
try {
|
||||
const page = parseInt(req.query.page) || 1;
|
||||
const pageUrl = getPageUrl(page);
|
||||
runTask(task)
|
||||
.then((record) => {
|
||||
runningStatus = { ...runningStatus, finished: true, result: record, current: 1 };
|
||||
})
|
||||
.catch((error) => {
|
||||
console.error('Task failed:', error.message);
|
||||
runningStatus = { ...runningStatus, finished: true, error: error.message, current: 1 };
|
||||
})
|
||||
.finally(() => {
|
||||
isRunning = false;
|
||||
});
|
||||
}
|
||||
|
||||
const html = await fetchHtml(pageUrl);
|
||||
const items = parseList(html);
|
||||
res.json({ success: true, data: items, page });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
function runTasksInBackground(tasks) {
|
||||
runningStatus = {
|
||||
taskId: null,
|
||||
city: null,
|
||||
startTime: Date.now(),
|
||||
current: 0,
|
||||
total: tasks.length,
|
||||
finished: false,
|
||||
error: null,
|
||||
results: [],
|
||||
};
|
||||
|
||||
// 按时间范围获取列表
|
||||
app.post('/api/list-daterange', async (req, res) => {
|
||||
try {
|
||||
const { startDate, endDate, maxPages = 50 } = req.body;
|
||||
const items = await fetchListByDateRange(startDate, endDate, maxPages);
|
||||
res.json({ success: true, data: items });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// 生成报告
|
||||
app.post('/api/report', async (req, res) => {
|
||||
try {
|
||||
const { limit = 50, threshold = 50 } = req.body;
|
||||
|
||||
// 采集列表
|
||||
const items = [];
|
||||
let pageIndex = 1;
|
||||
const maxPagesToFetch = Math.ceil(limit / 10) + 1;
|
||||
|
||||
while (items.length < limit && pageIndex <= maxPagesToFetch) {
|
||||
const pageUrl = getPageUrl(pageIndex);
|
||||
console.log(`正在采集第 ${pageIndex} 页: ${pageUrl}`);
|
||||
(async () => {
|
||||
for (const task of tasks) {
|
||||
runningStatus = { ...runningStatus, taskId: task.id, city: task.city };
|
||||
|
||||
try {
|
||||
const html = await fetchHtml(pageUrl);
|
||||
const pageItems = parseList(html);
|
||||
|
||||
if (pageItems.length === 0) {
|
||||
console.log(`第 ${pageIndex} 页没有数据，停止采集`);
|
||||
break;
|
||||
}
|
||||
|
||||
items.push(...pageItems);
|
||||
pageIndex++;
|
||||
|
||||
if (items.length < limit && pageIndex <= maxPagesToFetch) {
|
||||
await new Promise(resolve => setTimeout(resolve, 500));
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`采集第 ${pageIndex} 页失败: ${err.message}`);
|
||||
break;
|
||||
const record = await runTask(task);
|
||||
runningStatus.results.push(record);
|
||||
} catch (error) {
|
||||
const errorRecord = appendResult({
|
||||
taskId: task.id,
|
||||
city: task.city,
|
||||
scrapedAt: new Date().toISOString(),
|
||||
error: error.message,
|
||||
data: null,
|
||||
});
|
||||
runningStatus.results.push(errorRecord);
|
||||
}
|
||||
|
||||
runningStatus.current += 1;
|
||||
}
|
||||
|
||||
const results = items.slice(0, limit);
|
||||
|
||||
// 按阈值筛选
|
||||
const filtered = results.filter((item) => {
|
||||
return item.budget && item.budget.amount > threshold;
|
||||
runningStatus.finished = true;
|
||||
})()
|
||||
.catch((error) => {
|
||||
runningStatus = { ...runningStatus, finished: true, error: error.message };
|
||||
})
|
||||
.finally(() => {
|
||||
isRunning = false;
|
||||
});
|
||||
}
|
||||
|
||||
const total = filtered.reduce(
|
||||
(sum, item) => sum + (item.budget?.amount || 0),
|
||||
0
|
||||
);
|
||||
|
||||
const report = {
|
||||
summary: {
|
||||
total_count: results.length,
|
||||
filtered_count: filtered.length,
|
||||
threshold: `${threshold}万元`,
|
||||
total_amount: `${total.toFixed(2)}万元`,
|
||||
generated_at: new Date().toISOString(),
|
||||
},
|
||||
projects: filtered.map((item) => ({
|
||||
bidNo: item.bidNo,
|
||||
title: item.title,
|
||||
bidName: item.bidName,
|
||||
date: item.date,
|
||||
budget: item.budget,
|
||||
url: item.href,
|
||||
})),
|
||||
};
|
||||
|
||||
res.json({ success: true, data: report });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// 按时间范围生成报告
|
||||
app.post('/api/report-daterange', async (req, res) => {
|
||||
app.get('/api/results', (req, res) => {
|
||||
try {
|
||||
const { startDate, endDate, threshold = 50, maxPages = 50 } = req.body;
|
||||
|
||||
// 按时间范围采集列表
|
||||
const items = await fetchListByDateRange(startDate, endDate, maxPages);
|
||||
|
||||
if (items.length === 0) {
|
||||
return res.json({
|
||||
success: true,
|
||||
data: {
|
||||
summary: {
|
||||
total_count: 0,
|
||||
filtered_count: 0,
|
||||
threshold: `${threshold}万元`,
|
||||
total_amount: '0.00万元',
|
||||
generated_at: new Date().toISOString(),
|
||||
date_range: { startDate, endDate },
|
||||
},
|
||||
projects: [],
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
// 按阈值筛选
|
||||
const filtered = items.filter((item) => {
|
||||
return item.budget && item.budget.amount > threshold;
|
||||
});
|
||||
|
||||
const total = filtered.reduce(
|
||||
(sum, item) => sum + (item.budget?.amount || 0),
|
||||
0
|
||||
);
|
||||
|
||||
const report = {
|
||||
summary: {
|
||||
total_count: items.length,
|
||||
filtered_count: filtered.length,
|
||||
threshold: `${threshold}万元`,
|
||||
total_amount: `${total.toFixed(2)}万元`,
|
||||
generated_at: new Date().toISOString(),
|
||||
date_range: { startDate, endDate },
|
||||
},
|
||||
projects: filtered.map((item) => ({
|
||||
bidNo: item.bidNo,
|
||||
title: item.title,
|
||||
bidName: item.bidName,
|
||||
date: item.date,
|
||||
budget: item.budget,
|
||||
url: item.href,
|
||||
})),
|
||||
};
|
||||
|
||||
res.json({ success: true, data: report });
|
||||
const { city, section, type, page = 1, pageSize = 20, taskId, view } = req.query;
|
||||
const projectMode = view === 'projects';
|
||||
const result = listResults({ city, section, type, page, pageSize, taskId, projectMode });
|
||||
res.json({ success: true, ...result });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.delete('/api/results/:id', (req, res) => {
|
||||
try {
|
||||
const deleted = deleteResultById(req.params.id);
|
||||
if (!deleted) {
|
||||
return res.status(404).json({ success: false, error: '未找到该结果' });
|
||||
}
|
||||
|
||||
res.json({ success: true });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.delete('/api/results', (_req, res) => {
|
||||
try {
|
||||
clearResults();
|
||||
res.json({ success: true, message: '已清空所有结果' });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.get('/api/results/filters', (req, res) => {
|
||||
try {
|
||||
const projectMode = req.query.view === 'projects';
|
||||
res.json({ success: true, data: getResultFilters({ projectMode }) });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.get('/api/projects', (req, res) => {
|
||||
try {
|
||||
const {
|
||||
city,
|
||||
section,
|
||||
projectName,
|
||||
minAmount,
|
||||
maxAmount,
|
||||
startDate,
|
||||
endDate,
|
||||
page = 1,
|
||||
pageSize = 20,
|
||||
} = req.query;
|
||||
|
||||
const result = listProjects({
|
||||
city,
|
||||
section,
|
||||
projectName,
|
||||
minAmount,
|
||||
maxAmount,
|
||||
startDate,
|
||||
endDate,
|
||||
page,
|
||||
pageSize,
|
||||
dedupeByName: true,
|
||||
});
|
||||
|
||||
res.json({ success: true, ...result });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.get('/api/projects/filters', (_req, res) => {
|
||||
try {
|
||||
res.json({ success: true, data: getProjectFilters({ dedupeByName: true }) });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.get('/api/tasks', (_req, res) => {
|
||||
try {
|
||||
res.json({ success: true, data: listTasks() });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.post('/api/tasks', (req, res) => {
|
||||
try {
|
||||
const task = createTask(buildTaskPayload(req.body));
|
||||
res.json({ success: true, data: task });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.put('/api/tasks/:id', (req, res) => {
|
||||
try {
|
||||
const task = updateTask(req.params.id, buildTaskPayload(req.body, { partial: true }));
|
||||
if (!task) {
|
||||
return res.status(404).json({ success: false, error: '未找到该任务' });
|
||||
}
|
||||
|
||||
res.json({ success: true, data: task });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.delete('/api/tasks/:id', (req, res) => {
|
||||
try {
|
||||
const deleted = deleteTaskById(req.params.id);
|
||||
if (!deleted) {
|
||||
return res.status(404).json({ success: false, error: '未找到该任务' });
|
||||
}
|
||||
|
||||
res.json({ success: true });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.get('/api/tasks/status', (_req, res) => {
|
||||
if (!runningStatus) {
|
||||
return res.json({ success: true, data: { isRunning: false } });
|
||||
}
|
||||
|
||||
const elapsed = Math.round((Date.now() - runningStatus.startTime) / 1000);
|
||||
res.json({
|
||||
success: true,
|
||||
data: {
|
||||
isRunning,
|
||||
elapsed,
|
||||
city: runningStatus.city,
|
||||
current: runningStatus.current,
|
||||
total: runningStatus.total,
|
||||
finished: runningStatus.finished,
|
||||
error: runningStatus.error,
|
||||
results: runningStatus.finished
|
||||
? (runningStatus.results || (runningStatus.result ? [runningStatus.result] : []))
|
||||
: undefined,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
app.post('/api/tasks/:id/run', (req, res) => {
|
||||
if (isRunning) {
|
||||
return res.status(409).json({ success: false, error: '当前已有任务在运行，请稍后再试' });
|
||||
}
|
||||
|
||||
try {
|
||||
const task = getTaskById(req.params.id);
|
||||
if (!task) {
|
||||
return res.status(404).json({ success: false, error: '未找到该任务' });
|
||||
}
|
||||
|
||||
isRunning = true;
|
||||
runTaskInBackground(task);
|
||||
res.json({ success: true, message: `任务“${task.city}”已开始执行` });
|
||||
} catch (error) {
|
||||
isRunning = false;
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.post('/api/tasks/run', (req, res) => {
|
||||
if (isRunning) {
|
||||
return res.status(409).json({ success: false, error: '当前已有任务在运行，请稍后再试' });
|
||||
}
|
||||
|
||||
try {
|
||||
let tasks = listTasks();
|
||||
|
||||
if (Array.isArray(req.body?.ids) && req.body.ids.length > 0) {
|
||||
const idSet = new Set(req.body.ids);
|
||||
tasks = tasks.filter((task) => idSet.has(task.id));
|
||||
} else {
|
||||
tasks = tasks.filter((task) => task.enabled);
|
||||
}
|
||||
|
||||
if (tasks.length === 0) {
|
||||
return res.json({ success: true, data: [], message: '没有可运行的任务' });
|
||||
}
|
||||
|
||||
isRunning = true;
|
||||
runTasksInBackground(tasks);
|
||||
res.json({ success: true, message: `${tasks.length} 个任务已开始执行` });
|
||||
} catch (error) {
|
||||
isRunning = false;
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.get('/api/config', (_req, res) => {
|
||||
try {
|
||||
res.json({ success: true, data: maskConfigSecrets(loadConfig()) });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.post('/api/config', (req, res) => {
|
||||
try {
|
||||
saveConfig(mergeConfigWithExistingSecrets(req.body));
|
||||
reloadScheduler();
|
||||
res.json({ success: true, message: '配置已保存' });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// 发送报告邮件
|
||||
app.post('/api/send-email', async (req, res) => {
|
||||
try {
|
||||
const { emailConfig, report } = req.body;
|
||||
|
||||
// 验证必需的配置参数
|
||||
if (!emailConfig || !emailConfig.smtpHost || !emailConfig.smtpUser || !emailConfig.smtpPass) {
|
||||
return res.status(400).json({
|
||||
success: false,
|
||||
error: '邮件配置不完整,请填写SMTP服务器、用户名和密码',
|
||||
});
|
||||
if (!emailConfig?.smtpHost || !emailConfig?.smtpUser || !emailConfig?.smtpPass) {
|
||||
return res.status(400).json({ success: false, error: '邮件配置不完整' });
|
||||
}
|
||||
|
||||
if (!emailConfig.recipients || emailConfig.recipients.trim() === '') {
|
||||
return res.status(400).json({
|
||||
success: false,
|
||||
error: '请至少指定一个收件人',
|
||||
});
|
||||
if (!emailConfig.recipients?.trim()) {
|
||||
return res.status(400).json({ success: false, error: '请指定收件人' });
|
||||
}
|
||||
|
||||
if (!report) {
|
||||
return res.status(400).json({
|
||||
success: false,
|
||||
error: '没有可发送的报告数据',
|
||||
});
|
||||
return res.status(400).json({ success: false, error: '没有报告数据' });
|
||||
}
|
||||
|
||||
// 发送邮件
|
||||
const { sendReportEmail } = await import('./emailService.js');
|
||||
const result = await sendReportEmail(emailConfig, report);
|
||||
|
||||
res.json({
|
||||
success: true,
|
||||
message: '邮件发送成功',
|
||||
messageId: result.messageId,
|
||||
});
|
||||
} catch (error) {
|
||||
console.error('发送邮件API错误:', error);
|
||||
res.status(500).json({
|
||||
success: false,
|
||||
error: error.message,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// 获取配置
|
||||
app.get('/api/config', async (req, res) => {
|
||||
try {
|
||||
const { readFileSync } = await import('fs');
|
||||
const { join } = await import('path');
|
||||
const { fileURLToPath } = await import('url');
|
||||
const { dirname } = await import('path');
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = dirname(__filename);
|
||||
const configPath = join(__dirname, '..', 'config.json');
|
||||
|
||||
const configContent = readFileSync(configPath, 'utf-8');
|
||||
const config = JSON.parse(configContent);
|
||||
|
||||
// 不返回敏感信息(密码)
|
||||
if (config.email && config.email.smtpPass) {
|
||||
config.email.smtpPass = '***已配置***';
|
||||
}
|
||||
|
||||
res.json({ success: true, data: config });
|
||||
res.json({ success: true, message: '邮件发送成功', messageId: result.messageId });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// 更新配置
|
||||
app.post('/api/config', async (req, res) => {
|
||||
app.get('/api/scheduler/status', (_req, res) => {
|
||||
try {
|
||||
const { writeFileSync, readFileSync } = await import('fs');
|
||||
const { join } = await import('path');
|
||||
const { fileURLToPath } = await import('url');
|
||||
const { dirname } = await import('path');
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = dirname(__filename);
|
||||
const configPath = join(__dirname, '..', 'config.json');
|
||||
|
||||
const newConfig = req.body;
|
||||
|
||||
// 读取旧配置以保留敏感信息
|
||||
const oldConfigContent = readFileSync(configPath, 'utf-8');
|
||||
const oldConfig = JSON.parse(oldConfigContent);
|
||||
|
||||
// 如果密码字段是占位符,保留原密码
|
||||
if (newConfig.email && newConfig.email.smtpPass === '***已配置***') {
|
||||
newConfig.email.smtpPass = oldConfig.email?.smtpPass || '';
|
||||
}
|
||||
|
||||
// 保存配置
|
||||
writeFileSync(configPath, JSON.stringify(newConfig, null, 2), 'utf-8');
|
||||
|
||||
// 重新加载定时任务(如果定时任务配置有变化)
|
||||
reloadScheduler();
|
||||
|
||||
res.json({ success: true, message: '配置已保存并重新加载定时任务' });
|
||||
res.json({ success: true, data: getSchedulerStatus() });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// 获取定时任务状态
|
||||
app.get('/api/scheduler/status', async (req, res) => {
|
||||
app.post('/api/run-scheduled-task', (_req, res) => {
|
||||
try {
|
||||
const status = getSchedulerStatus();
|
||||
res.json({ success: true, data: status });
|
||||
runTaskNow().catch((error) => console.error('Scheduled task failed:', error));
|
||||
res.json({ success: true, message: '定时任务已在后台触发' });
|
||||
} catch (error) {
|
||||
res.status(500).json({ success: false, error: error.message });
|
||||
}
|
||||
});
|
||||
|
||||
// 手动触发定时任务的API(用于测试)
|
||||
app.post('/api/run-scheduled-task', async (req, res) => {
|
||||
try {
|
||||
console.log('手动触发定时任务...');
|
||||
// 在后台执行任务,不阻塞响应
|
||||
runTaskNow().catch(err => {
|
||||
console.error('定时任务执行失败:', err);
|
||||
});
|
||||
res.json({
|
||||
success: true,
|
||||
message: '定时任务已触发,正在后台执行...'
|
||||
});
|
||||
} catch (error) {
|
||||
res.status(500).json({
|
||||
success: false,
|
||||
error: error.message
|
||||
});
|
||||
}
|
||||
app.get('/results.html', (_req, res) => {
|
||||
res.redirect('/results');
|
||||
});
|
||||
|
||||
app.use((req, res, next) => {
|
||||
if (!HAS_DIST) return next();
|
||||
if (req.method !== 'GET') return next();
|
||||
if (req.path.startsWith('/api')) return next();
|
||||
if (req.path.includes('.')) return next();
|
||||
|
||||
res.sendFile(DIST_INDEX);
|
||||
});
|
||||
|
||||
initResultsStore();
|
||||
|
||||
app.listen(PORT, () => {
|
||||
console.log(`Server running at http://localhost:${PORT}`);
|
||||
|
||||
// 启动定时任务
|
||||
console.log('正在初始化定时任务...');
|
||||
initScheduler();
|
||||
});
|
||||
|
||||
49
vite.config.js
Normal file
49
vite.config.js
Normal file
@@ -0,0 +1,49 @@
|
||||
import { defineConfig } from 'vite';
|
||||
import vue from '@vitejs/plugin-vue';
|
||||
import { fileURLToPath, URL } from 'node:url';
|
||||
import AutoImport from 'unplugin-auto-import/vite';
|
||||
import Components from 'unplugin-vue-components/vite';
|
||||
import { ElementPlusResolver } from 'unplugin-vue-components/resolvers';
|
||||
|
||||
export default defineConfig({
|
||||
root: 'client',
|
||||
plugins: [
|
||||
vue(),
|
||||
AutoImport({
|
||||
imports: ['vue', 'vue-router'],
|
||||
resolvers: [ElementPlusResolver()],
|
||||
}),
|
||||
Components({
|
||||
resolvers: [ElementPlusResolver()],
|
||||
}),
|
||||
],
|
||||
resolve: {
|
||||
alias: {
|
||||
'@': fileURLToPath(new URL('./client/src', import.meta.url)),
|
||||
},
|
||||
},
|
||||
server: {
|
||||
host: '0.0.0.0',
|
||||
port: 5173,
|
||||
proxy: {
|
||||
'/api': {
|
||||
target: 'http://localhost:5000',
|
||||
changeOrigin: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
build: {
|
||||
outDir: '../dist',
|
||||
emptyOutDir: true,
|
||||
rollupOptions: {
|
||||
output: {
|
||||
manualChunks(id) {
|
||||
if (!id.includes('node_modules')) return;
|
||||
if (id.includes('element-plus')) return 'element-plus';
|
||||
if (id.includes('/vue/') || id.includes('@vue')) return 'vue-vendor';
|
||||
if (id.includes('axios')) return 'http-vendor';
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user