e想行空
0%

发表于 更新于 分类于 Waline：

常见的AI深度学习框架

PaddleSeg

飞桨（PaddlePaddle）以百度多年的深度学习技术研究和业务应用为基础，集深度学习核心训练和推理框架、基础模型库、端到端开发套件、丰富的工具组件于一体，是中国首个自主研发、功能完备、开源开放的产业级深度学习平台。国内的可以优先选择paddlepaddle，这是因为百度在大力推广该框架，能提供丰富的算力支持和技术支持，且百度开源了众多模型和应用，是人工智能入门的一个好选择，且能在模型上面进行一系列魔改，强烈推荐！

pytorch

是由Facebook人工智能学院提供支持服务的，目前主要在学术研究方向领域处于领先地位，许多学术论文都是用pytorch编写的，因此使用范围更广

常见的标注工具

labelImg

目前还没用过，官方已经不维护，建议用label-studio

label-studio

还没用过，docker安装

labelme

启动有点慢

发表于 更新于 分类于 Waline：

PyCharm安装PaddleSeg

  1. 在PyCharm欢迎页通过Get from VCS拉取github上PaddlePaddle/PaddleSeg项目

  2. 拉取成功后，PyCharm会提示创建venv虚拟环境，点击确定即可，在PyCharm打开终端命令行前面有venv代表创建成功，没有关闭终端点击Add new Interpreter，添加后，重新在PyCharm打开终端

    1
    2
    例如:
    (venv) ➜  PaddleSeg git:(release/2.9)

  3. 安装paddle，在PyCharm中Terminal命令行中执行pip install paddlepaddle

  4. 安装setuptools，在PyCharm中Terminal命令行中执行pip install -U pip setuptools

  5. [可选]，检查paddle是否安装成功，在PyCharm中Python console执行import paddle和paddle.utils.run_check()，检查版本执行print(paddle.__version__)

  6. 安装paddleseg，在PyCharm中Terminal命令行中执行pip install paddleseg，(这里采用的直接安装发布的版本，本地编译未实验通过)

  7. 验证安装是否成功，在在PyCharm中Terminal命令行中执行sh tests/install/check_predict.sh

PaddleSeg使用

准备自定义数据集

标注工具(标注数据)

PddleSeg已支持2种标注工具：LabelMe、精灵数据标注工具

  1. 下载安装labelme

  2. 根据文档操作，进行标注数据，简单流程打开图片目录->创建多边形->框选要标注的数据->保存会在图片目录生成一个json文件，标注目录下的所有图片

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    #标注前目录结构
    paddle
        |--image1.jpg
        |--image2.jpg
        |--...
    #标注后目录结构
    paddle
        |--image1.jpg
        |--image2.jpg    
        |--...  
        |--image1.json
        |--image2.json    
        |--...

  3. 将标注的数据转换为模型训练时所需的数据格式

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    #python tools/data/labelme2seg.py [-h] 图片目录 输出目录
    python tools/data/labelme2seg.py  /Users/x/Downloads/paddle /Users/xuanleung/Downloads/paddle_pr 
    #转换后目录结构
    paddle
    paddle_pr
        |--annotations       #红色背景，绿色标注的图片
        |  |--image1.png
        |  |--image2.png     
        |  |--...    
        |--images
        |  |--image1.jpg 
        |  |--image2.jpg     
        |  |--...  
        |--class_names.txt   #是数据集中所有标注类别的名称

切分数据

  1. 切分数据，对于所有原始图像和标注图像，需要按照比例划分为训练集、验证集、测试集。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    #python tools/data/split_dataset_list.py <dataset_root:原始图像目录名> <images_dir_name:原始图像目录名> <labels_dir_name:标注图像目录名> ${FLAGS}
    python tools/data/split_dataset_list.py /Users/x/Downloads/paddle_pr images annotations --split 0.6 0.2 0.2 --format jpg png
    #执行后目录结构
    paddle_pr
        |--test.txt   #测试集
        |--val.txt   #验证集
        |--train.txt   #训练集
        |--annotations       #红色背景，绿色标注的图片
        |  |--image1.png
        |  |--image2.png     
        |  |--...    
        |--images
        |  |--image1.jpg 
        |  |--image2.jpg     
        |  |--...  
        |--class_names.txt   #是数据集中所有标注类别的名称
    #txt文件内容格式
    images/image1.jpg annotations/image1.png
    images/image2.jpg annotations/image2.png
    ....

    FLAGS说明：

    FLAG 含义 默认值 参数数目
    –split 训练集、验证集和测试集的切分比例 0.7 0.3 0 3
    –separator txt文件列表分隔符 “ “ 1
    –format 原始图像和标注图像的图片后缀 “jpg” “png” 2
    –postfix 按文件主名（无扩展名）是否包含指定后缀对图片和标签集进行筛选 “” “”（2个空字符） 2

准备配置文件

拷贝paddle_pr数据集目录到项目根目录，在项目下面的目录configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml，修改这两个路径配置：

1
2
3
train_dataset:
  dataset_root: paddle_pr #数据集路径
  train_path: paddle_pr/train.txt #数据集中用于训练的标识文件

配置文件说明：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
batch_size: 4  #设定batch_size的值即为迭代一次送入网络的图片数量，一般显卡显存越大，batch_size的值可以越大。如果使用多卡训练，总得batch size等于该batch size乘以卡数。
iters: 1000    #模型训练迭代的轮数

train_dataset:  #训练数据设置
  type: Dataset #指定加载数据集的类。数据集类的代码在`PaddleSeg/paddleseg/datasets`目录下。
  dataset_root: data/optic_disc_seg #数据集路径
  train_path: data/optic_disc_seg/train_list.txt  #数据集中用于训练的标识文件
  num_classes: 2  #指定类别个数（背景也算为一类）
  mode: train #表示用于训练
  transforms: #模型训练的数据预处理方式。
    - type: ResizeStepScaling #将原始图像和标注图像随机缩放为0.5~2.0倍
      min_scale_factor: 0.5
      max_scale_factor: 2.0
      scale_step_size: 0.25
    - type: RandomPaddingCrop #从原始图像和标注图像中随机裁剪512x512大小
      crop_size: [512, 512]
    - type: RandomHorizontalFlip  #对原始图像和标注图像随机进行水平反转
    - type: RandomDistort #对原始图像进行亮度、对比度、饱和度随机变动，标注图像不变
      brightness_range: 0.5
      contrast_range: 0.5
      saturation_range: 0.5
    - type: Normalize #对原始图像进行归一化，标注图像保持不变

val_dataset:  #验证数据设置
  type: Dataset #指定加载数据集的类。数据集类的代码在`PaddleSeg/paddleseg/datasets`目录下。
  dataset_root: data/optic_disc_seg #数据集路径
  val_path: data/optic_disc_seg/val_list.txt  #数据集中用于验证的标识文件
  num_classes: 2  #指定类别个数（背景也算为一类）
  mode: val #表示用于验证
  transforms: #模型验证的数据预处理的方式
    - type: Normalize #对原始图像进行归一化，标注图像保持不变

optimizer: #设定优化器的类型
  type: SGD #采用SGD（Stochastic Gradient Descent）随机梯度下降方法为优化器
  momentum: 0.9 #设置SGD的动量
  weight_decay: 4.0e-5 #权值衰减，使用的目的是防止过拟合

lr_scheduler: # 学习率的相关设置
  type: PolynomialDecay # 一种学习率类型。共支持12种策略
  learning_rate: 0.01 # 初始学习率
  power: 0.9
  end_lr: 0

loss: #设定损失函数的类型
  types:
    - type: CrossEntropyLoss  #CE损失
  coef: [1, 1, 1] # PP-LiteSeg有一个主loss和两个辅助loss，coef表示权重，所以 total_loss = coef_1 * loss_1 + .... + coef_n * loss_n

model:  #模型说明
  type: PPLiteSeg  #设定模型类别
  backbone:  # 设定模型的backbone，包括名字和预训练权重
    type: STDC2
    pretrained: https://bj.bcebos.com/paddleseg/dygraph/PP_STDCNet2.tar.gz

训练模型

使用pyCharm添加一个python run config的启动配置，script设置为tools/train.py，script parameters设置为--config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --save_interval 500 --do_eval --use_vdl --save_dir output，然后点击右上角的run。
运行相关的日志如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
/Users/x/workspace/PaddleSeg/venv/bin/python -X pycache_prefix=/Users/x/Library/Caches/JetBrains/PyCharm2023.3/cpython-cache /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client 127.0.0.1 --port 64923 --file /Users/x/workspace/PaddleSeg/tools/train.py --config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --save_interval 500 --do_eval --use_vdl --save_dir output 
Connected to pydev debugger (build 233.15026.15)
2024-04-02 18:25:03 [WARNING]	Add the `num_classes` in train_dataset and val_dataset config to model config. We suggest you manually set `num_classes` in model config.
2024-04-02 18:25:03 [INFO]	
------------Environment Information-------------
platform: macOS-13.6.6-x86_64-i386-64bit
Python: 3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)]
Paddle compiled with cuda: False
GCC: Apple clang version 14.0.3 (clang-1403.0.22.14.1)
PaddleSeg: 2.8.1
PaddlePaddle: 2.6.1
OpenCV: 4.5.5
------------------------------------------------
2024-04-02 18:25:03 [INFO]	
---------------Config Information---------------
batch_size: 4
iters: 1000
train_dataset:
  dataset_root: paddle_pr
  mode: train
  num_classes: 2
  train_path: paddle_pr/train.txt
  transforms:
  - max_scale_factor: 2.0
    min_scale_factor: 0.5
    scale_step_size: 0.25
    type: ResizeStepScaling
  - crop_size:
    - 512
    - 512
    type: RandomPaddingCrop
  - type: RandomHorizontalFlip
  - brightness_range: 0.5
    contrast_range: 0.5
    saturation_range: 0.5
    type: RandomDistort
  - type: Normalize
  type: Dataset
val_dataset:
  dataset_root: data/optic_disc_seg
  mode: val
  num_classes: 2
  transforms:
  - type: Normalize
  type: Dataset
  val_path: data/optic_disc_seg/val_list.txt
optimizer:
  momentum: 0.9
  type: SGD
  weight_decay: 4.0e-05
lr_scheduler:
  end_lr: 0
  learning_rate: 0.01
  power: 0.9
  type: PolynomialDecay
loss:
  coef:
  - 1
  - 1
  - 1
  types:
  - type: CrossEntropyLoss
  - type: CrossEntropyLoss
  - type: CrossEntropyLoss
model:
  backbone:
    pretrained: https://bj.bcebos.com/paddleseg/dygraph/PP_STDCNet2.tar.gz
    type: STDC2
  num_classes: 2
  type: PPLiteSeg
------------------------------------------------

2024-04-02 18:25:03 [INFO]	Set device: cpu
2024-04-02 18:25:03 [INFO]	Use the following config to build model
model:
  backbone:
    pretrained: https://bj.bcebos.com/paddleseg/dygraph/PP_STDCNet2.tar.gz
    type: STDC2
  num_classes: 2
  type: PPLiteSeg
2024-04-02 18:25:03 [INFO]	Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/PP_STDCNet2.tar.gz
2024-04-02 18:25:04 [INFO]	There are 265/265 variables loaded into STDCNet.
2024-04-02 18:25:04 [INFO]	Use the following config to build train_dataset
train_dataset:
  dataset_root: paddle_pr
  mode: train
  num_classes: 2
  train_path: paddle_pr/train.txt
  transforms:
  - max_scale_factor: 2.0
    min_scale_factor: 0.5
    scale_step_size: 0.25
    type: ResizeStepScaling
  - crop_size:
    - 512
    - 512
    type: RandomPaddingCrop
  - type: RandomHorizontalFlip
  - brightness_range: 0.5
    contrast_range: 0.5
    saturation_range: 0.5
    type: RandomDistort
  - type: Normalize
  type: Dataset
2024-04-02 18:25:04 [INFO]	Use the following config to build val_dataset
val_dataset:
  dataset_root: data/optic_disc_seg
  mode: val
  num_classes: 2
  transforms:
  - type: Normalize
  type: Dataset
  val_path: data/optic_disc_seg/val_list.txt
2024-04-02 18:25:04 [INFO]	If the type is SGD and momentum in optimizer config, the type is changed to Momentum.
2024-04-02 18:25:04 [INFO]	Use the following config to build optimizer
optimizer:
  momentum: 0.9
  type: Momentum
  weight_decay: 4.0e-05
2024-04-02 18:25:04 [INFO]	Use the following config to build loss
loss:
  coef:
  - 1
  - 1
  - 1
  types:
  - type: CrossEntropyLoss
  - type: CrossEntropyLoss
  - type: CrossEntropyLoss
/Users/xuanleung/workspace/PaddleSeg/venv/lib/python3.12/site-packages/paddle/nn/layer/norm.py:824: UserWarning: When training, we now always track global mean and variance.
  warnings.warn(
2024-04-02 18:26:25 [INFO]	[TRAIN] epoch: 10, iter: 10/1000, loss: 0.8485, lr: 0.009919, batch_cost: 8.0518, reader_cost: 1.45770, ips: 0.4968 samples/sec | ETA 02:12:51
.......
2024-04-02 19:26:58 [INFO]	[TRAIN] epoch: 500, iter: 500/1000, loss: 0.1028, lr: 0.005369, batch_cost: 7.3678, reader_cost: 1.30898, ips: 0.5429 samples/sec | ETA 01:01:23
2024-04-02 19:26:58 [INFO]	Start evaluating (total_samples: 76, total_iters: 76)...
76/76 [==============================] - 39s 510ms/step - batch_cost: 0.5100 - reader cost: 8.5753e-04
2024-04-02 19:27:37 [INFO]	[EVAL] #Images: 76 mIoU: 0.4908 Acc: 0.9816 Kappa: 0.0000 Dice: 0.4954
2024-04-02 19:27:37 [INFO]	[EVAL] Class IoU: 
[0.9816 0.    ]
2024-04-02 19:27:37 [INFO]	[EVAL] Class Precision: 
[0.9816 0.    ]
2024-04-02 19:27:37 [INFO]	[EVAL] Class Recall: 
[1. 0.]
2024-04-02 19:27:38 [INFO]	[EVAL] The model with the best validation mIoU (0.4908) was saved at iter 500.
2024-04-02 19:28:51 [INFO]	[TRAIN] epoch: 510, iter: 510/1000, loss: 0.2417, lr: 0.005272, batch_cost: 7.3453, reader_cost: 1.31461, ips: 0.5446 samples/sec | ETA 00:59:59
.....
2024-04-02 20:29:09 [INFO]	[TRAIN] epoch: 1000, iter: 1000/1000, loss: 0.1945, lr: 0.000020, batch_cost: 7.4149, reader_cost: 1.36141, ips: 0.5395 samples/sec | ETA 00:00:00
2024-04-02 20:29:09 [INFO]	Start evaluating (total_samples: 76, total_iters: 76)...
76/76 [==============================] - 38s 503ms/step - batch_cost: 0.5028 - reader cost: 7.8113e-04
2024-04-02 20:29:47 [INFO]	[EVAL] #Images: 76 mIoU: 0.4908 Acc: 0.9816 Kappa: 0.0000 Dice: 0.4954
2024-04-02 20:29:47 [INFO]	[EVAL] Class IoU: 
[0.9816 0.    ]
2024-04-02 20:29:47 [INFO]	[EVAL] Class Precision: 
[0.9816 0.    ]
2024-04-02 20:29:47 [INFO]	[EVAL] Class Recall: 
[1. 0.]
2024-04-02 20:29:48 [INFO]	[EVAL] The model with the best validation mIoU (0.4908) was saved at iter 500.
<class 'paddle.nn.layer.conv.Conv2D'>'s flops has been counted
<class 'paddle.nn.layer.norm.BatchNorm2D'>'s flops has been counted
<class 'paddle.nn.layer.activation.ReLU'>'s flops has been counted
<class 'paddle.nn.layer.pooling.AvgPool2D'>'s flops has been counted
<class 'paddle.nn.layer.pooling.AdaptiveAvgPool2D'>'s flops has been counted
Total Flops: 9643807616     Total Params: 12251410

Process finished with exit code 0

生成的文件：

1
2
3
4
5
6
7
8
9
output
  ├── iter_500          #表示在500步保存一次模型
    ├── model.pdparams  #模型参数
    └── model.pdopt     #训练阶段的优化器参数
  ├── iter_1000         #表示在1000步保存一次模型
    ├── model.pdparams  #模型参数
    └── model.pdopt     #训练阶段的优化器参数
  └── best_model        #精度最高的模型权重
    └── model.pdparams

模型评估

拷贝paddle_pr数据集目录到项目根目录，在项目下面的目录configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml，修改这两个路径配置：

1
2
3
val_dataset:
  dataset_root: paddle_pr #数据集路径
  train_path: paddle_pr/val.txt #数据集中用于训练的标识文件

使用pyCharm添加一个python run config的启动配置，script设置为tools/val.py，script parameters设置为--config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --model_path output/iter_1000/model.pdparams，然后点击右上角的run。

输出结果：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/Users/x/workspace/PaddleSeg/venv/bin/python tools/val.py --config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --model_path output/iter_1000/model.pdparams 
2024-04-03 11:18:24 [WARNING]	Add the `num_classes` in train_dataset and val_dataset config to model config. We suggest you manually set `num_classes` in model config.
....
2024-04-03 11:18:25 [INFO]	Start evaluating (total_samples: 2, total_iters: 2)...
2/2 [==============================] - 40s 20s/step - batch_cost: 20.0147 - reader cost: 0.3020
#数据解释
 #Acc(准确率)：指类别预测正确的像素占总像素的比例，准确率越高模型质量越好。
 #mIoU(平均交并比)：对每个类别数据集单独进行推理计算，计算出的预测区域和实际区域交集除以预测区域和实际区域的并集，然后将所有类别得到的结果取平均。
 #Kappa系数：一个用于一致性检验的指标，可以用于衡量分类的效果。kappa系数的计算是基于混淆矩阵的，取值为-1到1之间，通常大于0。其公式如下所示，P0P_0P0为分类器的准确率，PeP_eP**e为随机分类器的准确率。Kappa系数越高模型质量越好。
2024-04-03 11:19:05 [INFO]	[EVAL] #Images: 2 mIoU: 0.9224 Acc: 0.9958 Kappa: 0.9162 Dice: 0.9581
2024-04-03 11:19:05 [INFO]	[EVAL] Class IoU: 
[0.9957 0.8491]
2024-04-03 11:19:05 [INFO]	[EVAL] Class Precision: 
[0.9972 0.94  ]
2024-04-03 11:19:05 [INFO]	[EVAL] Class Recall: 
[0.9984 0.8977]
Process finished with exit code 0

模型预测

使用pyCharm添加一个python run config的启动配置，script设置为tools/predict.py，script parameters设置为--config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --model_path output/iter_1000/model.pdparams --image_path paddle_pr/images/image1.jpg --save_dir output/result，然后点击右上角的run。

输出结果：

1
2
3
4
5
6
7
8
9
10
11
12
/Users/x/workspace/PaddleSeg/venv/bin/python tools/predict.py --config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --model_path output/iter_1000/model.pdparams --image_path paddle_pr/images/image1.jpg --save_dir output/result 
2024-04-03 11:45:21 [WARNING]	Add the `num_classes` in train_dataset and val_dataset config to model config. We suggest you manually set `num_classes` in model config.
2024-04-03 11:45:23 [INFO]	Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/PP_STDCNet2.tar.gz
2024-04-03 11:45:23 [INFO]	There are 265/265 variables loaded into STDCNet.
2024-04-03 11:45:23 [INFO]	The number of images: 1
2024-04-03 11:45:23 [INFO]	Loading pretrained model from output/iter_1000/model.pdparams
2024-04-03 11:45:23 [INFO]	There are 370/370 variables loaded into PPLiteSeg.
2024-04-03 11:45:23 [INFO]	Start to predict...
1/1 [==============================] - 21s 21s/step
2024-04-03 11:45:44 [INFO]	Predicted images are saved in output/result/added_prediction and output/result/pseudo_color_prediction .

Process finished with exit code 0

生成的文件：

1
2
3
4
5
6
7
8
9
10
11
output/result
|
|--added_prediction #叠加效果图
|  |--image1.jpg
|  |--image2.jpg
|  |--...
|
|--pseudo_color_prediction #预测mask
|  |--image1.jpg
|  |--image2.jpg
|  |--...

导出模型

使用pyCharm添加一个python run config的启动配置，script设置为tools/export.py，script parameters设置为--config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --model_path output/best_model/model.pdparams --save_dir output/inference_model，然后点击右上角的run。

输出结果：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
/Users/x/workspace/PaddleSeg/venv/bin/python tools/export.py --config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --model_path output/best_model/model.pdparams --save_dir output/inference_model 
2024-04-03 12:07:23 [WARNING]	Add the `num_classes` in train_dataset and val_dataset config to model config. We suggest you manually set `num_classes` in model config.
2024-04-03 12:07:23 [INFO]	
2024-04-03 12:07:24 [INFO]	Loaded trained params successfully.
/Users/xuanleung/workspace/PaddleSeg/venv/lib/python3.12/site-packages/paddle/jit/api.py:310: UserWarning: full_graph=False is not supported in Python 3.12+. Set full_graph=True automatically
  warnings.warn(
I0403 12:07:27.211655 1341450176 program_interpreter.cc:212] New Executor is Running.
2024-04-03 12:07:27 [INFO]	
---------------Deploy Information---------------
Deploy:
  input_shape:
  - -1
  - 3
  - -1
  - -1
  model: model.pdmodel
  output_dtype: int32
  output_op: argmax
  params: model.pdiparams
  transforms:
  - type: Normalize

2024-04-03 12:07:27 [INFO]	The inference model is saved in output/inference_model
Process finished with exit code 0

生成的文件：

1
2
3
4
5
output/inference_model
  ├── deploy.yaml            # 部署相关的配置文件，主要说明数据预处理方式等信息
  ├── model.pdmodel          # 预测模型的拓扑结构文件
  ├── model.pdiparams        # 预测模型的权重文件
  └── model.pdiparams.info   # 参数额外信息，一般无需关注

部署模型(Paddle Inference部署python)

使用pyCharm添加一个python run config的启动配置，script设置为deploy/python/infer.py，script parameters设置为-config output/inference_model/deploy.yaml --image_path /Users/xuanleung/Downloads/test.jpg --device cpu，然后点击右上角的run。

生产的文件：output/test.png

MacOS安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
#检查环境
#要求 pip 版本为 20.2.2 或更高版本
➜  ~ python3 -m pip --version
pip 24.0 from /usr/local/lib/python3.12/site-packages/pip (python 3.12)
#需要确认 python 的版本是否满足要求3.8/3.9/3.10/3.11/3.12
➜  ~ python3 --version
Python 3.12.2
#需要确认 Python 和 pip 是 64bit，并且处理器架构是 x86_64
➜  ~ python3 -c "import platform;print(platform.architecture()[0]);print(platform.machine())"
64bit
x86_64
#检查是否支持avx，结果输出avx即表示支持
➜  ~ sysctl machdep.cpu.features | grep -i avx
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX SMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C
➜  ~ sysctl machdep.cpu.leaf7_features | grep -i avx
machdep.cpu.leaf7_features: RDWRFSGS TSC_THREAD_OFFSET SGX BMI1 HLE AVX2 SMEP BMI2 ERMS INVPCID RTM FPU_CSDS MPX RDSEED ADX SMAP CLFSOPT IPT MDCLEAR TSXFA IBRS STIBP L1DF ACAPMSR SSBD

#安装unrar,官方命令brew install unrar，unrar找不到，新安装方法
➜  ~ brew install rar
==> Downloading https://www.rarlab.com/rar/rarmacos-x64-700.tar.gz
######################################################################### 100.0%
==> Installing Cask rar
==> Moving Generic Artifact 'default.sfx' to '/usr/local/lib/default.sfx'
==> Moving Generic Artifact 'rarfiles.lst' to '/usr/local/etc/rarfiles.lst'
==> Linking Binary 'rar' to '/usr/local/bin/rar'
==> Linking Binary 'unrar' to '/usr/local/bin/unrar'
🍺  rar was successfully installed!


#创建个虚拟环境目录
mkdir pytho work_python3
#Python 3创建虚拟环境
➜  paddleseg_python3 python3 -m venv .
#查看当前目录，可以发现有几个文件夹和一个pyvenv.cfg文件：
➜  paddleseg_python3 ls
bin        include    lib        pyvenv.cfg
#继续进入bin目录
➜  paddleseg_python3 cd bin
#激活该venv环境
➜  bin source activate
#查看当前目录，里面有python3、pip3等可执行文件，实际上是链接到Python系统目录的软链接。
(paddleseg_python3) ➜  bin ls
Activate.ps1  activate.csh  pip           pip3.12       python3
activate      activate.fish pip3          python        python3.12
#下面正常安装paddlepaddle
(paddleseg_python3) ➜  bin python3 -m pip install paddlepaddle==2.6.0 -i https://mirror.baidu.com/pypi/simple
#安装依赖
(paddleseg_python3) ➜  bin pip install -U pip setuptools
#验证安装
(paddleseg_python3) ➜  bin python3
Python 3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import paddle
>>> paddle.utils.run_check()
Running verify PaddlePaddle program ...
I0326 15:14:46.340384 1206859520 program_interpreter.cc:212] New Executor is Running.
I0326 15:14:46.356616 1206859520 interpreter_util.cc:624] Standalone Executor is Used.
PaddlePaddle works well on 1 CPU.
PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now.
>>> print(paddle.__version__)
2.6.0

#安装PaddleSeg https://github.com/PaddlePaddle/PaddleSeg/blob/release/2.9/docs/install_cn.md#22-%E5%AE%89%E8%A3%85paddleseg  失败

## 回退所有安装
(paddleseg_python3) ➜  PaddleSeg git:(release/2.6) deactivate
➜  PaddleSeg git:(release/2.6)

macOS docker 安装

1
2
3
4
5
6
7
mkdir paddle
cd paddle
#-v $PWD:/paddle：指定将当前路径（PWD 变量会展开为当前路径的绝对路径）挂载到容器内部的 /home/paddle 目录；
docker run -d -p 80:80 --env USER_PASSWD="123456" --name paddle -it -v $PWD:/home/paddle paddlepaddle/paddle:2.6.0-jupyter
#进入容器
docker exec -it paddle /bin/bash
#访问127.0.0.1 输入用户名/密码 jovyan/123456 进入jovyan

常见问题

  1. 安装python3 -m pip install paddlepaddle==2.6.0 -i https://mirror.baidu.com/pypi/simple提示如下错误

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    ➜  ~ python3 -m pip install paddlepaddle==2.6.0 -i https://mirror.baidu.com/pypi/simple
    error: externally-managed-environment

    × This environment is externally managed
    ╰─> To install Python packages system-wide, try brew install
        xyz, where xyz is the package you are trying to
        install.

        If you wish to install a non-brew-packaged Python package,
        create a virtual environment using python3 -m venv path/to/venv.
        Then use path/to/venv/bin/python and path/to/venv/bin/pip.

        If you wish to install a non-brew packaged Python application,
        it may be easiest to use pipx install xyz, which will manage a
        virtual environment for you. Make sure you have pipx installed.

    note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
    hint: See PEP 668 for the detailed specification.

    解决：参考pip(3) install，完美解决 externally-managed-environment

    方案一：添加参数–break-system-packages，这种直接安装到系统，可能会影响系统环境。

    1
    python3 -m pip install paddlepaddle==2.6.0 -i https://mirror.baidu.com/pypi/simple --break-system-packages

    方案二：pipx，安装完成后，无法导入paddle，初步判断无法进入虚拟环境

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    # pipx会为每个安装的应用创建一个独立的虚拟环境，避免不同应用之间的依赖冲突。
    ➜  ~ brew install pipx
    #当你首次安装pipx时，运行pipx ensurepath会自动检查并修改你的环境变量设置（如需要的话），以确保你可以轻松运行pipx安装的任何程序。这个步骤通常只需要执行一次。
    ➜  ~ pipx ensurepath
    #替换官方的安装命令，使用pipx安装
    # 2.9.0找不到
    ➜  ~ pipx install paddlepaddle==2.9.0 -i https://mirror.baidu.com/pypi/simple
    Fatal error from pip prevented installation. Full pip output in file:
        /Users/xuanleung/Library/Logs/pipx/cmd_2024-03-25_15.41.49_pip_errors.log

    Some possibly relevant errors from pip install:
        ERROR: Could not find a version that satisfies the requirement paddlepaddle==2.9.0 (from versions: 2.6.0)
        ERROR: No matching distribution found for paddlepaddle==2.9.0

    Error installing paddlepaddle from spec 'paddlepaddle==2.9.0'.
    # 换成安装2.6.0
    ➜  ~ pipx install paddlepaddle==2.6.0 -i https://mirror.baidu.com/pypi/simple
      installed package paddlepaddle 2.6.0, installed using Python 3.12.2
      These apps are now globally available
        - fleetrun
        - paddle
    ⚠️  Note: '/Users/xuanleung/.local/bin' is not on your PATH environment
        variable. These apps will not be globally accessible until your PATH is
        updated. Run `pipx ensurepath` to automatically add it, or manually modify
        your PATH in your shell's config file (i.e. ~/.bashrc).
    done! ✨ 🌟 ✨
    # 再次执行更新环境变量
    ➜  ~ pipx ensurepath

    【采用】方案三：使用venv

  2. 执行import paddle提示Python 3: ImportError “No Module named Setuptools”

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    >>> import paddle
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Users/xx/workspace/paddleseg_python3/lib/python3.12/site-packages/paddle/__init__.py", line 28, in <module>
        from .base import core  # noqa: F401
        ^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/xx/workspace/paddleseg_python3/lib/python3.12/site-packages/paddle/base/__init__.py", line 77, in <module>
        from . import dataset
      File "/Users/xx/workspace/paddleseg_python3/lib/python3.12/site-packages/paddle/base/dataset.py", line 20, in <module>
        from ..utils import deprecated
      File "/Users/xx/workspace/paddleseg_python3/lib/python3.12/site-packages/paddle/utils/__init__.py", line 16, in <module>
        from . import (  # noqa: F401
      File "/Users/xx/workspace/paddleseg_python3/lib/python3.12/site-packages/paddle/utils/cpp_extension/__init__.py", line 15, in <module>
        from .cpp_extension import (
      File "/Users/xx/workspace/paddleseg_python3/lib/python3.12/site-packages/paddle/utils/cpp_extension/cpp_extension.py", line 21, in <module>
        import setuptools
    ModuleNotFoundError: No module named 'setuptools'
    >>> paddle.utils.run_check()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'paddle' is not defined

    解决：执行pip install -U pip setuptools

  3. 执行

    1
    2
    3
    4
    5
    6
    python tools/train.py \
           --config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml \
           --do_eval \
           --use_vdl \
           --save_interval 500 \
           --save_dir output

    提示如下错误:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    (venv) ➜  PaddleSeg git:(release/2.9) ✗ python tools/train.py \
           --config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml \
           --save_interval 500 \
           --do_eval \
           --use_vdl \
           --save_dir output
    Traceback (most recent call last):
      File "/Users/x/workspace/PaddleSeg/tools/train.py", line 213, in <module>
        main(args)
      File "/Users/x/workspace/PaddleSeg/tools/train.py", line 145, in main
        cfg = Config(
              ^^^^^^^
    TypeError: Config.__init__() got an unexpected keyword argument 'to_static_training'

    原因：tools/train.py文件里面缺少参数to_static_training
    解决：切换到release/2.8.1分支，再次执行命令

  4. 执行

    1
    2
    3
    4
    5
    6
    python tools/train.py \
           --config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml \
           --do_eval \
           --use_vdl \
           --save_interval 500 \
           --save_dir output

    提示如下错误：

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
      - type: CrossEntropyLoss
      - type: CrossEntropyLoss
      - type: CrossEntropyLoss
    /Users/xuanleung/workspace/PaddleSeg/venv/lib/python3.12/site-packages/paddle/nn/layer/norm.py:824: UserWarning: When training, we now always track global mean and variance.
      warnings.warn(
    Traceback (most recent call last):
      File "/Users/x/workspace/PaddleSeg/tools/train.py", line 195, in <module>
        main(args)
      File "/Users/x/workspace/PaddleSeg/tools/train.py", line 170, in main
        train(
      File "/Users/x/workspace/PaddleSeg/venv/lib/python3.12/site-packages/paddleseg/core/train.py", line 273, in train
        avg_loss_list = [l[0] / log_iters for l in avg_loss_list]
                         ~^^^
    IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed

    解决：使用pyCharm添加一个python run config的启动配置，script设置为tools/train.py，script parameters设置为--config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --save_interval 500 --do_eval --use_vdl --save_dir output，然后点击右上角的run。

发表于 更新于 分类于 Waline：

tesseract

识别率较低(已卸载)

安装

mac安装执行brew install tesseract即可，安装过程可能有点慢，要下载很久

安装完成后，执行brew info tesseract，可以查看tesseract相关信息

使用

1
2
3
4
5
6
tesseract <图片名字> <输出的文本文件名(自带.txt)> [-l 语言类型] [--oem ocrenginemode] [--psm pagesegmode] [configfiles...]
#举例
tesseract test.jpg stout -l chi_sim+equ --psm 6
ls
|-test.jpg
|-stout.txt

PaddleOCR

识别率高

安装

  1. pyCharm拉取github PaddlePaddle/PaddleOCR仓库的代码，并切换到分支release/2.7

  2. 在pyCharm终端执行下面命令,注意需要venv加载成功

    1
    2
    3
    4
    5
    6
    7
    #环境所需依赖
    (venv) ➜  PaddleOCR git:(release/2.7) pip install paddlepaddle 
    (venv) ➜  PaddleOCR git:(release/2.7) pip install -U pip setuptools
    #安装
    (venv) ➜  PaddleOCR git:(release/2.7) pip install "paddleocr>=2.0.1"
    #使用
    (venv) ➜  PaddleOCR git:(release/2.7) paddleocr --image_dir /Users/xuanleung/Downloads/ppocr_img/imgs/11.jpg --use_gpu false

实战场景

有一大批照片，要从照片中获取指定信息作为图片的文件名，并根据文件名生产csv表格

  1. 提前训练好模型参考：PaddleSeg基于机器学习的图像分割

  2. 采用paddleSeg（模型预测）进行关键位置的识别

    1
    2
    #pycharm 运行 
    python tools/predict.py --config configs/quick_start/pp_liteseg_optic_disc_512x512_1k.yml --model_path output/iter_1000/model.pdparams --image_path /Users/x/Downloads/桩顶标高 --save_dir /Users/x/Downloads/桩顶标高out --custom_color 0 0 0 255 255 255

  3. 裁剪图片

    1
    2
    3
    #pycharm 运行 
    python tools/crop.py --input_mask_dir /Users/x/Downloads/桩顶标高out/pseudo_color_prediction --input_orig_dir /Users/x/Downloads/桩顶标高 
    Creating crop directory: /Users/x/Downloads/桩顶标高/crop_dir

  4. 执行ocr识别，会产生一个output目录，里面会产生和图片对应文件名的txt文本，内容为ocr识别的内容

    1
    2
    3
    #建议优化脚本里面的识别方式，替换成PaddleOCR识别率提高非常的多
    #在图片所在目录
    bash ocr.sh

  5. 执行关键文字提取脚本，将关键字在文本文件里面保留

    1
    2
    #在图片所在的output目录里面
    bash format.sh

  6. 替换文件名，根据文本里面的内容作为图片的文件名

    1
    2
    #在图片所在目录
    bash rename.sh

  7. 根据文件名生产csv表格

    1
    2
    #在图片所在目录
    bash csv.sh

文件附录

  • ocr.sh：批量将当前脚本所在目录的图片转换成文本

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    #建议优化脚本里面的识别方式，替换成PaddleOCR识别率提高非常的多
    #!/bin/bash
    # 当前脚本所在目录
    script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
    # 输入文件夹路径为当前目录
    input_folder="$script_dir"
    # 输出文件夹路径为当前目录
    output_folder="$script_dir/output"
    # 创建输出文件夹
    mkdir -p "$output_folder"
    # 遍历输入文件夹中的所有图片文件
    for image_file in "$input_folder"/*.jpg "$input_folder"/*.png "$input_folder"/*.jpeg; do
        # 提取文件名（不含路径和扩展名）
        filename=$(basename "$image_file" | cut -f 1 -d '.')
        # 构建输出文本文件的路径
        output_text_path="$output_folder/$filename"
        # 执行 Tesseract 命令
        tesseract "$image_file" "$output_text_path" -l chi_sim+equ --psm 6
        echo "Processed $image_file"
    done

  • format.sh：批量将当前脚本所在目录的文本只保留关键信息

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    #!/bin/bash
    # 获取当前脚本所在目录
    script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
    # 遍历当前目录下的所有文本文件
    for file in "$script_dir"/*.txt; do
        # 检查文件是否存在
        if [ -f "$file" ]; then
            # 使用grep命令查找匹配行并提取信息
            transaction=$(grep -oE '体验[0-9]+' "$file" | cut -d'验' -f2)
            elevation=$(grep -oE '[0-9]{3}\.[0-9]{3}' "$file" | cut -d':' -f2)
            # 输出提取的信息
            echo "交易编号: $transaction"
            echo "高程: $elevation"
            # 将提取的信息写入原文件
            echo "$transaction-$elevation" > "$file"
        fi
    done

  • rename.sh：批量重命名脚本所在目录的图片，文件名根据output目录对应文件名的文本内容进行命名

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    #!/bin/bash
    # 获取当前脚本所在目录
    script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
    # 图片文件夹路径
    image_folder="$script_dir"
    # 遍历图片文件夹中的所有文本文件
    for text_file in "$image_folder"/output/*.txt; do
        # 检查文件是否存在
        if [ -f "$text_file" ]; then
            # 获取文本文件名（不含路径和扩展名）
            base_name=$(basename "$text_file" .txt)        
            # 获取图片文件名
            image_file="$image_folder/$base_name.jpg"       
            # 检查图片文件是否存在
            if [ -f "$image_file" ]; then
                # 获取文本文件的内容作为新的图片文件名
                new_name=$(cat "$text_file")            
                # 检查是否有同名文件，如果有就给新名称添加序号
                new_image_file="$image_folder/$new_name.jpg"
                counter=1
                while [ -f "$new_image_file" ]; do
                    new_image_file="$image_folder/${new_name}_${counter}.jpg"
                    ((counter++))
                done            
                # 重命名图片文件
                mv "$image_file" "$new_image_file"            
                echo "Renamed '$image_file' to '$new_image_file'"
            fi
        fi
    done

  • csv.sh：根据文件名生产表格，表格数据有两列，文件名以-进行区分列

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    #!/bin/bash
    # 获取当前脚本所在目录
    script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
    # 图片文件夹路径
    folder="$script_dir"
    # CSV 文件路径
    csv_file="$script_dir/output.csv"
    # 创建 CSV 文件，并写入标题行
    echo "Column1,Column2" > "$csv_file"
    # 遍历文件夹中的所有图片文件
    for file in "$folder"/*.jpg; do
        # 提取文件名中的两个部分，使用 '-' 作为分隔符
        part1=$(basename "$file" | cut -d'-' -f1)
        part2=$(basename "$file" | cut -d'-' -f2 | sed 's/\.jpg//')
        # 将两个部分写入 CSV 文件
        echo "$part1,$part2" >> "$csv_file"
    done

  • crop.py：根据模型预测的数据，进行图片裁剪

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    # 参数：input_mask_dir 掩码图片目录，黑白图片
    # 参数：input_orig_dir 原始图片目录，文件名要和掩码图片文件名一致，后缀可以不一样
    # 输出：根据掩码图片裁剪原始图片，得到原图的裁剪图片
    import argparse
    import glob
    import os
    import os.path as osp
    import cv2

    from pathlib import Path
    from paddleseg.utils import logger

    def parse_args():
        parser = argparse.ArgumentParser(description='image crop.')
        parser.add_argument('--input_mask_dir', help='input annotated directory',type=str)
        parser.add_argument('--input_orig_dir', help='input annotated directory',type=str)
        return parser.parse_args()

    def main(args):
        output_dir = osp.join(args.input_orig_dir, 'crop_dir')
        if not osp.exists(output_dir):
            os.makedirs(output_dir)
            print('Creating crop directory:', output_dir) # 在原图目录创建裁剪后输出目录
        for mask_img in glob.glob(osp.join(args.input_mask_dir, '*.png')):  # 遍历input_mask_dir目录所有png的图片
            print('read mask img from:', mask_img)
            mask_img_name = Path(mask_img).stem  # 获取文件名
            print('read orig img name:', args.input_orig_dir+mask_img_name+".jpg")
            # 读取原始图像和关键部位掩码图像
            original_image = cv2.imread(args.input_orig_dir+"/"+mask_img_name+".jpg")
            mask_image = cv2.imread(mask_img, cv2.IMREAD_GRAYSCALE)  # 假设是单通道的灰度图像
            # 查找掩码图像中的边界
            contours, _ = cv2.findContours(mask_image, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
            # 找到最大的边界
            max_contour = max(contours, key=cv2.contourArea)
            # 获取最大边界的边界框
            x, y, w, h = cv2.boundingRect(max_contour)
            # 从原始图像中裁剪关键部位
            key_part = original_image[y:y + h, x:x + w]
            # 保存裁剪出的关键部位
            cv2.imwrite(output_dir+"/"+mask_img_name+".jpg", key_part)
        logger.info(f'crop img is saved in {output_dir}')

    if __name__ == '__main__':
        args = parse_args()
        main(args)

发表于 更新于 分类于 Waline：

升讯威在线客服与营销系统官方在线文档

安装

  1. 下载私有化部署包

  2. 解压下载的压缩包

  3. 在解压的目录新建一个Dockerfile

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    FROM centos:centos7.9.2009

    WORKDIR /wwwroot

    COPY init.sh /tmp/sh/
    COPY appsettings.json /tmp/config/
    COPY Management /wwwroot/Management
    COPY Resource /wwwroot/Resource
    COPY Server /wwwroot/Server

    RUN rpm -Uvh https://packages.microsoft.com/config/rhel/7/packages-microsoft-prod.rpm && yum -y install dnf && dnf install dotnet-sdk-3.1 -y && rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm && yum install libgdiplus-devel nginx -y && ln -s /usr/lib64/libgdiplus.so /usr/lib/gdiplus.dll && ln -s /usr/lib64/libgdiplus.so /usr/lib64/gdiplus.dll && chmod +x /tmp/sh/init.sh  && systemctl enable nginx

    COPY default.conf /etc/nginx/conf.d/

    EXPOSE 80

    ENTRYPOINT ["/tmp/sh/init.sh"]

  4. 创建初始化启动脚本,在解压的目录新建一个init.sh

    1
    2
    3
    4
    5
    6
    #!/bin/bash
    cp /tmp/config/appsettings.json /wwwroot/Server/appsettings.json
    nginx
    nginx -s reload
    cd /wwwroot/Server
    dotnet Sheng.Linkup.Server.dll

  5. nginx代理配置,在解压的目录新建一个default.conf

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    # https://kf.shengxunwei.com
       # Start
       
       map $http_upgrade $connection_upgrade {
       default upgrade;
       ''   close;
       }
       
       upstream dotnet_server_proxy {                                                         
           server localhost:5000;                                                
           keepalive 2000;
       }
       
       # kf-api (Server)
       server{
           listen 80;
           listen [::]:80;
       
           server_name kf-api.domain.io;
       
           location / {
                       proxy_pass http://dotnet_server_proxy;
                       proxy_http_version 1.1;
                       proxy_set_header Upgrade $http_upgrade;
                       proxy_set_header Connection keep-alive; 
                       proxy_set_header Host $host;
                       proxy_cache_bypass $http_upgrade;
                       proxy_set_header X-Forwarded-For $remote_addr;
       
                       proxy_set_header Upgrade $http_upgrade;
                       proxy_set_header Connection $connection_upgrade;
       
           }
       }
       
       
       # kf-resource (Resource)
       server {
           listen       80;
           server_name  kf-resource.domain.io;
       
           location / {
               root   /wwwroot/Resource;
               index  v.html;
           }
       
           error_page   500 502 503 504  /50x.html;
           location = /50x.html {
               root   html;
           }
       
       }
       
       # kf-m (Management)
       server {
           listen       80;
           server_name   kf-m.domain.io;
       
           location / {
               root   /wwwroot/Management;
               index  index.html;
           }
       
           error_page   500 502 503 504  /50x.html;
           location = /50x.html {
               root   html;
           }
       
       }
       
       # End
       # https://kf.shengxunwei.com

  6. 在解压的目录新建一个appsettings.json配置，里面进行mysql数据库配置，初始化脚本在：数据库建表脚本/CreateDatabase_MySql.sql

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    {
      "Version": 3,
      "Logging": {
        "LogLevel": {
          "Default": "Information",
          "Microsoft": "Warning",
          "Microsoft.Hosting.Lifetime": "Information"
        }
      },
      "AllowedHosts": "*",
      "DatabaseEngine": "mysql",
      "ConnectionStrings": {
        "DefaultConnection": "Server=172.1.1.2;database=kf_shengxunwei;user=root;password=Tech@2023"
      },
      "AppSettings": {
        "FileStore": {
          "Store": "static",
          "AliOSS": {
            "AccessKeyId": null,
            "AccessKeySecret": null,
            "BucketName": null,
            "PublicAddress": null,
            "EndPoint": null
          }
        },
        "Email": {
          "Account": "",
          "Password": ""
        },
        "Translation": {
          "Engine": "",
          "Youdao": {
            "AppKey": null,
            "AppSecret": null
          },
          "Baidu": {
            "AppId": null,
            "AppSecret": null
          }
        },
        "Environment": {
          "DefaultLanguage": "zh-CHS",
          "CustomerContext": {
            "HeartbeatTimeoutSecond": 70,
            "CellphoneHeartbeatTimeoutSecond": 70,
            "OfflineOverSecond": 70
          },
          "TcpIpAddress": "172.1.1.44",
          "TcpPort": "9527",
          "ResourceAddress": "http://kf-resource.domain.com.cn",
          "HostAddress": "http://kf-api.domian.com.cn"
        }
      }
    }

  7. 执行docker build -t harbor.domain.dev/base/shengxunwei:1.3 .

  8. 运行docker run harbor.domain.dev/base/shengxunwei:1.3

  9. 访问https://kf-api.domain.io/Status

  10. 初始化访问https://kf-api.domain.io/Status/Setup

配置

Resource对应客户访问的界面

Management对应客服管理后台界面

ApiUrl：http://kf-api.domain.io

ResourceUrl：http://kf-resource.domain.io

管理后台地址：http://kf-m.domain.io

需要修改下面几个文件里面的ApiUrl和ResourceUrl

1
2
3
/wwwroot/Resource/embedded.js
/wwwroot/Resource/WebChat/Config.js
/wwwroot/Management/config.js

使用

客户访问http://kf-resource.domain.io/WebChat/MobileWebChat.html?siteCode=freesite

客服下载客户端，两个就能进行聊天了

发表于 更新于 分类于 Waline：

window系统免费安装软件制作系统，用于打包封装nginx、emqx、mysql、redis、springboot等合成一个windows的exe安装包。

安装脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
[Setup]
  
#define MyAppName "myapp"
#define MyAppVersion "1.0"
#define MyAppPublisher "LP"
#define MyAppURL "http://www.lp.top:8090/"
#define MyAppExeName "myapp.exe"
 
[Setup]
AppId= {{ED9D5968-F178-48C5-AC61-2AED59BBCEF6}} //每次编译需要修改该uuid
AppName={#MyAppName}
AppVersion={#MyAppVersion}
;AppVerName={#MyAppName} {#MyAppVersion}
AppPublisher={#MyAppPublisher}
AppPublisherURL={#MyAppURL}
AppSupportURL={#MyAppURL}
AppUpdatesURL={#MyAppURL}
DefaultDirName={pf}\{#MyAppName}
DefaultGroupName={#MyAppName}
DisableDirPage=yes
OutputDir=D:\innosetup\output
OutputBaseFilename=gateId
Compression=lzma
SolidCompression=yes
LicenseFile=D:\workspace\license.rtf #软件安装申明
;AlwaysRestart=yes
;PrivilegesRequired=admin

[Code]
//#############################自定义函数###################################
// 判断单个端口占用
function IsNotPortOccupation(strPortNum: String): Boolean;
// 变量定义
var ErrorCode: Integer;
var bRes: Boolean;
var strFileContent: AnsiString;
var strTmpPath: String;  // 临时目录
var strTmpFile: String;  // 临时文件，保存查找软件数据结果
var strCmdFind: String;  // 查找端口命令， netstat  -natp tcp |findstr "LISTENING" |findstr ":80 "| find /C  ":80 "  找到的话返回个数，找不到为0
begin
  strTmpPath := GetTempDir();
  strTmpFile := Format('%sfindProtRes.txt', [strTmpPath]);
                                                                                             
  strCmdFind := Format('/c netstat -natp tcp |findstr "LISTENING" |findstr  ":%s "|find /C ":%s " > "%s "', [strPortNum, strPortNum ,strTmpFile]);
  log(strCmdFind);
 
  bRes := ShellExec('open', ExpandConstant('{cmd}'), strCmdFind, '', SW_HIDE, ewWaitUntilTerminated, ErrorCode);  
  bRes := LoadStringFromFile(strTmpFile, strFileContent);
  strFileContent := Trim(strFileContent);
  if StrToInt(strFileContent)   > 0 then begin
    result:=true;
    end else result:=false;
  end;
// 判断多个端口占用
function IsNotPortsOccupation(const portList: array of String): array of String;
var
  i: Integer;
begin
  for i := 0 to GetArrayLength(portList)-1 do
  begin
    if IsNotPortOccupation(portList[i]) then 
      begin 
        SetLength(Result,Length(Result)+1);
        Result[High(Result)] := portList[i];
      end;
  end;
end;
// 数组转换为字符串
function JoinArrayToString(arr: array of String; delimiter: String): String;
var
  i: Integer;
begin
  Result := '';
  for i := 0 to GetArrayLength(arr) - 1 do
  begin
    if Result <> '' then
      Result := Result + delimiter;
    Result := Result + arr[i];
  end;
end;
// 检查是否需要初始化数据
function IsInitMySqlData: Boolean;
begin
  if MsgBox('存在历史数据，是否初始化历史数据？',mbInformation,MB_YESNO) = IDYES then
  begin
    Result := true;
  end else
    Result := false;
end;
//#############################自定义界面###########################################
//----------------安装类型界面-------------------------------------
var
Page: TWizardPage;

RadioButton1, RadioButton2, RadioButton3, RadioButton4: TRadioButton;
Lbl1, Lbl2: TNewStaticText;


procedure CreateTypeSelectPage;
begin
Page := CreateCustomPage(wpInfoBefore, '选择安装类型', '请根据您的需要选择安装的类型');

RadioButton1 := TRadioButton.Create(Page);
RadioButton1.Left := ScaleX(80);
RadioButton1.Top := ScaleY(40);
RadioButton1.Width := Page.SurfaceWidth;
RadioButton1.Height := ScaleY(17);
RadioButton1.Caption := '标准安装';
RadioButton1.Checked := True;
RadioButton1.Parent := Page.Surface;

Lbl1 := TNewStaticText.Create(Page);
Lbl1.Left := ScaleX(95);
Lbl1.Top := ScaleY(60);
Lbl1.Width := ScaleX(250);
Lbl1.Height := ScaleY(50);
Lbl1.Caption := '按照标准模式安装软件到您的电脑';
Lbl1.Parent := Page.Surface;

RadioButton2 := TRadioButton.Create(Page);
RadioButton2.Left := ScaleX(80);
RadioButton2.Top := RadioButton1.Top + ScaleY(60);
RadioButton2.Width := Page.SurfaceWidth;
RadioButton2.Height := ScaleY(17);
RadioButton2.Caption := '自定义安装';
RadioButton2.Checked := false;
RadioButton2.Parent := Page.Surface;

Lbl2 := TNewStaticText.Create(Page);
Lbl2.Left := ScaleX(95);
Lbl2.Top := Lbl1.Top + ScaleY(60);
Lbl2.Width := ScaleX(250);
Lbl2.Height := ScaleY(50);
Lbl2.Caption := '您可以选择单个安装项，建议经验丰富的用户使用';
Lbl2.Parent := Page.Surface;
end;
//------------------------安装状态界面--------------------------------------------
procedure CreateShowStatusPage;
begin
Page := CreateCustomPage(wpInfoAfter, '等待服务状态', '请耐心等待服务启动完成');
RadioButton3 := TRadioButton.Create(Page);
RadioButton3.Left := ScaleX(80);
RadioButton3.Top := ScaleY(40);
RadioButton3.Width := Page.SurfaceWidth;
RadioButton3.Height := ScaleY(17);
RadioButton3.Caption := 'redis';
RadioButton3.Checked := false;
RadioButton3.Parent := Page.Surface;

RadioButton4 := TRadioButton.Create(Page);
RadioButton4.Left := ScaleX(80);
RadioButton4.Top := RadioButton1.Top + ScaleY(60);
RadioButton4.Width := Page.SurfaceWidth;
RadioButton4.Height := ScaleY(17);
RadioButton4.Caption := 'emqx';
RadioButton4.Checked := false;
RadioButton4.Parent := Page.Surface;
end;
//-------------------------参数配置界面--------------------------------------------


//#############################安装开始时掉用###################################
procedure InitializeWizard();
var 
  UsePorts: array of String;
  UsePortsMessage: String;
begin
    UsePorts := IsNotPortsOccupation(['80','1883','8083','8084','8883','18083','3306','6379','8080','8081','8082','8085']);
    if Length(UsePorts)>0 then //检测这些端口是否占用，包含nginx、emqx、mysql、redis、springboot
    begin
     UsePortsMessage := '占用的端口：'+ JoinArrayToString(UsePorts,'，'); 
     MsgBox(UsePortsMessage, mbInformation, MB_OK);
     Abort;
    end;
    CreateTypeSelectPage;
    CreateShowStatusPage;
end;

function ShouldSkipPage(PageID: Integer): Boolean; //是否跳过组件选择界面
begin
  if (PageID = wpSelectComponents) and (RadioButton1.Checked) then
  Result := True
  else if (PageID = wpSelectProgramGroup) and (RadioButton1.Checked) then
  Result := True
end;
//#############################安装结束时掉用#########################################
procedure CurStepChanged(CurStep: TSetupStep);
var 
  UsePorts: array of String;
  UsePortsMessage: string;      
begin
  if CurStep = ssPostInstall then
  begin
    UsePorts := IsNotPortsOccupation(['80','1883','8083','8084','8883','18083','3306','6379','8080','8081','8082','8085']);
    if Length(UsePorts)<12 then  //检测服务是否正常运行，有些服务还没启动成功，这里就已经运行了，会导致一些服务检测不到，而且检测到没运行，也不支持安装回退
    begin
     UsePortsMessage := '启动成功的服务端口：'+ JoinArrayToString(UsePorts,'，');  
     MsgBox(UsePortsMessage, mbInformation, MB_OK);
     Abort;
    end;                                                       
  end;
end;

[Types]
Name: "Custom"; Description: "Custom Installation"; Flags: iscustom

[Components] 
Name: MySql; Description: MySql ; Types: Custom //设置组件树(安装选择组件界面显示的内容)
Name: Redis; Description: Redis ; Types: Custom
Name: Emqx; Description: Emqx ; Types: Custom
Name: Nginx; Description: Nginx ; Types: Custom
Name: Java; Description: Java ; Types: Custom 
//通过 exclusive 完成 DMR、XPT 的互斥选择            
Name: Java\Gateway; Description: Gateway; Types: Custom;
Name: Java\Gateid; Description: Gateid; Types: Custom;
Name: Java\Base; Description: Base; Types: Custom;
Name: Java\Auth; Description: Auth; Types: Custom;
 
[Languages]
Name: "chinesesimp"; MessagesFile: "compiler:Default.isl"
 
[Tasks]
Name: "desktopicon"; Description: "{cm:CreateDesktopIcon}"; GroupDescription: "{cm:AdditionalIcons}"; Flags: unchecked; OnlyBelowVersion: 0,6.1
 
[Files]
Source: "D:\workspace\mysql\*"; DestDir: "{app}\mysql"; Flags: ignoreversion recursesubdirs createallsubdirs; Components: MySql
Source: "D:\workspace\redis\*"; DestDir: "{app}\redis"; Flags: ignoreversion recursesubdirs createallsubdirs; Components: Redis
Source: "D:\workspace\emqx\*"; DestDir: "{app}\emqx"; Flags: ignoreversion recursesubdirs createallsubdirs; Components: Emqx
Source: "D:\workspace\nginx\*"; DestDir: "{app}\nginx"; Flags: ignoreversion recursesubdirs createallsubdirs; Components: Nginx
Source: "D:\workspace\project\*"; DestDir: "{app}\project"; Flags: ignoreversion recursesubdirs createallsubdirs;Components: Java
; 注意: 不要在任何共享系统文件上使用“Flags: ignoreversion ?
 
[Icons]
; Name: "{commondesktop}\project";Filename: "{app}\project\start.exe"; WorkingDir: "{app}\HSDServer"
 
[INI]
;修改数据库配置文
Filename:"{app}\mysql\my.ini";Section:"mysqld";Key:"basedir"; String:"{app}\mysql";Components:MySql;
Filename:"{app}\mysql\my.ini";Section:"mysqld";Key:"datadir"; String:"{app}\mysqlData";Components:MySql;
Filename:"{app}\mysql\my.ini";Section:"mysqld";Key:"port"; String:"3306";Components:MySql; 
Filename:"{app}\mysql\my.ini";Section:"client";Key:"port"; String:"3306";Components:MySql;
 
[Run] //安装脚本，包含数据初始化脚本
Filename: "{app}\mysql\bin\mysql_install.bat";Components:MySql;
Filename: "{app}\mysql\bin\mysql_init.bat";Components:MySql;Check:IsInitMySqlData;
Filename: "{app}\redis\init-redis.bat";Components:Redis;
Filename: "{app}\redis\start-redis.bat";Components:Redis;
Filename: "{app}\emqx\bin\mqttSatrt.bat";Components:Emqx;
Filename: "{app}\nginx\startNginx.bat";Components:Nginx;
Filename: "{app}\project\start.bat";Components:Java;
 
[UninstallRun] //卸载执行的脚本
Filename: "{app}\mysql\bin\mysql_stop.bat";Components:MySql;
Filename: "{app}\redis\stop-redis.bat";Components:Redis;
Filename: "{app}\emqx\bin\mqttStop.bat";Components:Emqx;
Filename: "{app}\nginx\stopNginx.bat";Components:Nginx;
Filename: "{app}\project\stop.bat";Components:Java;
 
[UninstallDelete] //删除目录
Type:filesandordirs;Name:"{app}\mysql";Components:MySql;
Type:filesandordirs;Name:"{app}\redis";Components:Redis;
Type:filesandordirs;Name:"{app}\emqx";Components:Emqx;
Type:filesandordirs;Name:"{app}\nginx";Components:Nginx;
Type:filesandordirs;Name:"{app}\project";Components:Java;

[Code]
//卸载时，弹窗提示删除数据
procedure CurUninstallStepChanged(CurUninstallStep: TUninstallStep);
var
  DeleteProfile: string;
  DeleteConfirm: Boolean;
  ErrorCode: Integer;
begin
  case CurUninstallStep of
  //卸载后的收尾工作
  usPostUninstall:begin
  // 确认是否删除整个目录
    DeleteProfile := ExpandConstant('{app}');
    DeleteConfirm :=MsgBox('是否保留数据库数据?',mbConfirmation,MB_YESNO) = idYes;
    if DeleteConfirm=False then
    DelTree(ExpandConstant('{app}'), True, True, True);
  end;
end;
end;

参考：

Inno Setup添加自定义页面

使用innoSetup将mysql+nginx+redis+jar包打包成windows安装包

发表于 更新于 分类于 Waline：

go入门

docker打包构建部署

  1. 创建一个项目目录和两个文件，结构如下

    1
    2
    3
    go-hello
    |-hello.go
    |-Dockerfile

  2. 文件内容分别如下

    hello.go文件内容：

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    package main

    import (
        "fmt"
        "log"
        "net/http"
    )

    func hello(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "Hello World")
    }

    func main() {
        http.HandleFunc("/", hello)
        if err := http.ListenAndServe(":8080", nil); err != nil {
            log.Fatal(err)
        }
    }

    Dockerfile文件内容：

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    FROM golang:1.21.3-alpine AS builder
    WORKDIR /build
    ADD . /build
    RUN go build -o hello ./hello.go

    FROM alpine
    WORKDIR /build
    COPY --from=builder /build/hello /build/hello
    EXPOSE 8080
    CMD ["./hello"]

  3. 在项目跟目录(go-hello)，执行docker build -t hello:latest .进行镜像打包

  4. 运行镜像docker run -p 8080:8080 hello:latest然后访问http://127.0.0.1:8080/

手动测试

  1. 在项目跟目录执行go build -o hello ./hello.go生成hello可执行文件
  2. 执行./hello，然后访问http://127.0.0.1:8080/

导入github包

  1. 在项目跟目录执行go mod init go-hello，会生成一个go.mod和go.sum的文件，文件内容如下:

    go.mod

    1
    2
    3
    4
    module qiniu_go
    go 1.21.3
    require github.com/qiniu/go-sdk/v7 v7.19.0
    require golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 // indirect

    go.sum

    1
    2
    3
    github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
    github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
    .....

  2. 在项目跟目录执行go mod tidy,删除错误或者不使用的modules

实战

写一个接口将接口的json数据保存到七牛云，支持修改

main.go，dockerfile根据hello的例子进行修改即可

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"

	"github.com/qiniu/go-sdk/v7/auth"
	"github.com/qiniu/go-sdk/v7/storage"
)

func upload(writer http.ResponseWriter, request *http.Request) {
	// 检查请求方法是否为POST
	if request.Method != http.MethodPost {
		http.Error(writer, "Invalid request method", http.StatusMethodNotAllowed)
		return
	}

	// 读取请求体
	body, err := io.ReadAll(request.Body)
	if err != nil {
		http.Error(writer, "Error reading request body", http.StatusInternalServerError)
		return
	}

	// 解析JSON数据
	var jsonData map[string]interface{}
	err = json.Unmarshal(body, &jsonData)
	if err != nil {
		http.Error(writer, "Error decoding JSON", http.StatusBadRequest)
		return
	}

	// 打印接收到的JSON数据
	fmt.Printf("Received JSON: %+v\n", jsonData)

	// 可选：将JSON数据转为字节
	data, err := json.Marshal(jsonData)
	if err != nil {
		http.Error(writer, "Error encoding JSON", http.StatusInternalServerError)
		return
	}

	accessKey := ""
	secretKey := ""
	bucket := "test"
	keyToOverwrite := "3D/3d.json"

	putPolicy := storage.PutPolicy{
		Scope: fmt.Sprintf("%s:%s", bucket, keyToOverwrite),
	}
	mac := auth.New(accessKey, secretKey)
	upToken := putPolicy.UploadToken(mac)

	cfg := storage.Config{}
	// 空间对应的机房
	cfg.Region = &storage.ZoneHuanan
	// 是否使用https域名
	cfg.UseHTTPS = true
	// 上传是否使用CDN上传加速
	cfg.UseCdnDomains = false

	formUploader := storage.NewFormUploader(&cfg)
	ret := storage.PutRet{}
	putExtra := storage.PutExtra{
		Params: map[string]string{
			"x:name": "github logo",
		},
	}
	//data := []byte("hello, this is qiniu cloud")
	dataLen := int64(len(data))
	qErr := formUploader.Put(context.Background(), &ret, upToken, keyToOverwrite, bytes.NewReader(data), dataLen, &putExtra)
	if err != nil {
		fmt.Println("err:", qErr)
		return
	}
	fmt.Println(ret.Key, ret.Hash)
	fmt.Fprintf(writer, "https://qiniu.xx.com/"+keyToOverwrite)
}

func main() {
	fmt.Println("service start")
	http.HandleFunc("/upload", upload)
	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatal(err)
	}

}

发表于 更新于 分类于 Waline：

时间同步的两种方式chrony vs ntp

ntp：传统的时间同步配置，既可以当服务端，也可以当客户端

chrony：新式时间配置，采用微调修改同步时间

常见命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#查看时间
[root@master ~]# date
#查看时间详情
[root@master ~]# timedatectl
      Local time: 三 2024-01-10 00:27:35 CST #本地时间，初始值来自于RTC，通常等于RTC+时区值（如本地时间= RTC + 8小时）
  Universal time: 二 2024-01-09 16:27:35 UTC #世界时间，中国时区=UTC+8
        RTC time: 二 2024-01-09 17:50:38 #BIOS时间,硬件时间
       Time zone: Asia/Shanghai (CST, +0800) #时区
     NTP enabled: yes #表示开启NTP同步
NTP synchronized: no #NTP是否同步完成
 RTC in local TZ: no  #no=RTC时间用UTC,yes=RTC时间用LOCAL time
      DST active: n/a
# 默认情况下，系统配置为使用UTC，也可使用本地时间
[root@master ~]# timedatectl set-ntp true／false #开启关闭ntp
[root@master ~]# timedatectl set-local-rtc true   # 将RTC设置为本地时间
[root@master ~]# timedatectl set-local-rtc false  # 将RTC设置为UTC   
[root@master ~]# chronyc tracking |grep System    # 查看服务器时间和NTP server偏差
System time     : 23735.990234375 seconds fast of NTP time
[root@master ~]# chronyc activity #显示有多少NTP源在线/离线
200 OK
4 sources online
0 sources offline
[root@master ~]# chronyc makestep #立即手动同步时间（需要chronyd服务运行）

[root@master ~]# hwclock -w #将当前时间和日期写入BIOS，避免重启后失效（改了之后恢复）
[root@master ~]# hwclock --systohc #将系统时间写入RTC（待观察）
[root@master ~]# hwclock --localtime #显示 BIOS 中实际的时间
2024年01月09日 星期二 18时10分55秒  -0.062371 秒
[root@master ~]# systemctl status chronyd.service #查看chronyd服务是否启动

[root@master ~]# ntpdate cn.pool.ntp.org #ntp同步时间
[root@master ~]# timedatectl set-time 2023-12-28 #手动设置时间，需要关闭自动同步时间
[root@master ~]# timedatectl set-ntp no
[root@master ~]# timedatectl set-time 2023-12-28
[root@master ~]# timedatectl set-time 16:06:17
[root@master ~]# timedatectl set-timezone Asia/Shanghai #设置时区

问题

  1. 修改时间过后，时间过一段时间又不对了

    分析，怀疑是chronyc慢慢修正成了错误时间

    解决过程：

    1. 通过把chronyc停用，换成ntpd进行同步时间，过了一天，时间还是改变了(RTC+8)，发现ntpd服务在3点的时候停止了

    2. 添加时间ntpd服务是否运行的检测的脚本check_ntp.sh

      1
      2
      3
      4
      5
      6
      7
      8
      #!/bin/bash

      # 检查 NTP 服务是否正在运行
      if systemctl is-active --quiet ntpd; then
          echo "NTP service is running."
      else
          echo "NTP service is not running."
      fi

    3. 添加时间是否同步time_sync_status.sh

      1
      2
      3
      4
      5
      6
      7
      8
      #!/bin/bash

      # 检查时间是否同步
      if timedatectl | grep "NTP synchronized: yes" > /dev/null; then
          echo "System time is synchronized."
      else
          echo "System time is not synchronized."
      fi

    4. 整合服务检测，与时间同步

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      #!/bin/bash

      LOG_FILE="/var/log/sync_time.log"

      # 检查系统时间是否正确
      if timedatectl | grep "NTP synchronized: yes" > /dev/null; then
          echo "$(date): System time is synchronized." >> "$LOG_FILE"
          echo "$(timedatectl)" >> "$LOG_FILE"
      else
          echo "$(date): System time is not synchronized." >> "$LOG_FILE"
      		echo "$(timedatectl)" >> "$LOG_FILE"
          # 检查 NTP 服务状态
          if systemctl is-active --quiet ntpd; then
              echo "$(date): NTP service is running. Synchronizing time..." >> "$LOG_FILE"
              echo "$(timedatectl)" >> "$LOG_FILE"
              systemctl stop ntpd  # 停止 NTP 服务，以便手动同步时间
              ntpd -gq  # 强制同步时间
              hwclock --systohc  # 同步时间到 RTC
              systemctl start ntpd  # 重新启动 NTP 服务
              echo "$(date): Time synchronized successfully." >> "$LOG_FILE"
              echo "$(timedatectl)" >> "$LOG_FILE"
          else
              echo "$(date): NTP service is not running. " >> "$LOG_FILE"
              echo "$(timedatectl)" >> "$LOG_FILE"
              ntpd -gq  # 强制同步时间
              hwclock --systohc  # 同步时间到 RTC
              systemctl start ntpd  # 重新启动 NTP 服务
              echo "$(date): Time synchronized successfully." >> "$LOG_FILE"
              echo "$(timedatectl)" >> "$LOG_FILE"
          fi
      fi

    5. 创建一个定时任务crontab -e

      1
      */1 * * * * /root/sync_time.sh

    6. 后面观察日志tail -f /var/log/sync_time.log即可，执行more /var/log/sync_time.log | grep " is not " -A 28 -B 10进行查看什么时候通不过时间

参考：

chrony时间同步服务

发表于 更新于 分类于 Waline： 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#添加用户
sudo useradd -d /home/exxk/shared_bikes_html -m -s /bin/bash bikesfe
#创建修改密码
sudo passwd bikesfe
#添加bikesfe用户只能使用sftp
sudo vim /etc/ssh/sshd_config
Match User bikesfe
     ChrootDirectory /home/exxk/shared_bikes_html
     ForceCommand internal-sftp
     AllowTcpForwarding no
     X11Forwarding no
#修改/home/exxk/shared_bikes_html目录及上层目录权限
sudo chmod 755 /home/exxk
sudo chmod 755 /home/exxk/shared_bikes_html
sudo chown root:root /home/exxk/shared_bikes_html
sudo chown root:root /home/exxk
#测试
#sftp登录成功
sftp bikesfe@172.16.10.2
#ssh登录失败
ssh bikesfe@172.16.10.2

常见问题：

  1. 不修改目录及上级目录权限会提示如下错误：
1
2
3
4
5
6
Jan 08 14:24:26 ubuntu sshd[2677787]: Accepted password for bikesfe from 172.16.30.210 port 52532 ssh2
Jan 08 14:24:26 ubuntu sshd[2677787]: pam_unix(sshd:session): session opened for user bikesfe(uid=1001) by (uid=0)
Jan 08 14:24:26 ubuntu systemd[2677793]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Jan 08 14:24:26 ubuntu sshd[2677897]: fatal: bad ownership or modes for chroot directory component "/home/hcytech/"
Jan 08 14:24:26 ubuntu sshd[2677787]: pam_unix(sshd:session): session closed for user bikesfe
Jan 08 14:24:37 ubuntu systemd[2677793]: Closed GnuPG cryptographic agent (ssh-agent emulation).

发表于 更新于 分类于 Waline：

Firebase是个啥

服务端

  1. 添加依赖

    1
    2
    3
    4
    5
    <dependency>
        <groupId>com.google.firebase</groupId>
        <artifactId>firebase-admin</artifactId>
        <version>9.2.0</version>
    </dependency>

  2. 在Firebase控制台，点击项目设置->服务账号->Firebase Admin SDK->选择java->生成新的私钥等待下载成功，复制下载成功的文件到项目里面

  3. 创建FireBaseConfig.java，添加初始化代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    import com.google.auth.oauth2.GoogleCredentials;
    import com.google.firebase.FirebaseApp;
    import com.google.firebase.FirebaseOptions;
    import org.springframework.boot.CommandLineRunner;
    import org.springframework.stereotype.Component;
    import java.io.FileInputStream;
    import java.io.IOException;

    @Component
    public class FireBaseConfig implements CommandLineRunner {
        @Override
        public void run(String... args) throws Exception {
            initFireBase();
        }

        private void initFireBase() throws IOException {
            //协议开放 克服握手错误
            System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2");
            System.out.println("协议开放成功: "+System.getProperty("https.protocols"));
            //firebase初始化
            FileInputStream refreshToken = new FileInputStream("path/to/serviceAccountKey.json"); //上一步下载的文件(私钥json)
            FirebaseOptions options = FirebaseOptions.builder()
                    .setCredentials(GoogleCredentials.fromStream(refreshToken))
                    .setDatabaseUrl("https://<DATABASE_NAME>.firebaseio.com/")
                    .build();
            FirebaseApp.initializeApp(options);
            System.out.println("firebase初始化成功!!!");
        }
    }

  4. 创建工具类FireBaseUtils

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    import com.google.firebase.messaging.*;
    import java.util.Map;

    public class FireBaseUtils {
        public static String sendOne(String token, String title, String body, Map<String,String> data) throws FirebaseMessagingException {
            Message message = Message.builder()
                    .setToken(token)
    //                .setAndroidConfig(AndroidConfig.builder()
    //                        .setNotification(AndroidNotification.builder().setTitle(title).setBody(body).build()).build())
                    .setNotification(Notification.builder().setTitle(title).setBody(body).build())
                    .putAllData(data)
    //                .setNotification(Notification.builder().build())
                    .build();
            String result= FirebaseMessaging.getInstance().send(message);
            System.out.println("消息发送成功: "+result);
            return result;
    }

  5. 测试发送消息

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    @GetMapping("/send")
    public String send(){
        try {
            Map<String,String> data=new HashMap<>();
            data.put("allDataKey","allDataValue");
          	//设备的token由客户端传过来
            return FireBaseUtils.sendOne("dtgeiUDK<设备的token>RVhF9:APAi1_xR4JQt5dELt",
                    "我是标题","我是body",data);
        } catch (FirebaseMessagingException e) {
            throw new RuntimeException(e);
        }
    }

Firebase如何进行消息推送

  1. 在firebase控制台创建项目

  2. 在产品类别->吸引->Messaging界面，点击添加应用

  3. 在项目设置->常规->您的应用查看应用接入配置

    • web版本(未成功，应该是没有token的原因)

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      import { initializeApp } from "firebase/app";
      import { getAnalytics } from "firebase/analytics";
      import { getMessaging, onMessage } from "firebase/messaging"; // 接入messaging模块

      const firebaseConfig = {
        apiKey: "AIxxx9HY",
        authDomain: "fir-502c5.firebaseapp.com",
        projectId: "fir-502c5",
        storageBucket: "fir-502c5.appspot.com",
        messagingSenderId: "47xxxx14",
        appId: "1:471967859514:web:b1d4axx6cac0d748603",
        measurementId: "G-FxxK"
      };

      // Initialize Firebase
      const app = initializeApp(firebaseConfig);
      const analytics = getAnalytics(app);

      const messaging=getMessaging(app); // 接入messaging模块
      onMessage(messaging, (payload) => {
        console.log('Message received. ', payload);
      });

    • Android

      可以使用 android studio再带的firebase进行集成，点击tools->firebase后续跟着提示操作即可。

      下面介绍手动版本：

      1. 引入依赖

        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        //在项目(project)build.gradle添加com.google.gms.google-services依赖
        plugins {
            id 'com.android.application' version '8.2.0' apply false
          //添加依赖
            id 'com.google.gms.google-services' version '4.3.15' apply false
        }

        //在模块(module)build.gradle添加com.google.gms.google-services依赖
        plugins {
            id 'com.android.application'
            id 'com.google.gms.google-services'
        }
        ....
        dependencies {
        		....
            implementation(platform("com.google.firebase:firebase-bom:32.3.1"))
            implementation("com.google.firebase:firebase-analytics")
            implementation("com.google.firebase:firebase-inappmessaging")
            implementation("com.google.firebase:firebase-inappmessaging-display")
        }

      2. 添加获取token的方法，服务端发送消息的时候需要该token，一般在设备登录的时候把token发给服务端

        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        14
        15
        16
        FirebaseMessaging.getInstance().getToken()
                .addOnCompleteListener(new OnCompleteListener<String>() {
                    @Override
                    public void onComplete(@NonNull Task<String> task) {
                        if (!task.isSuccessful()) {
                            Log.w(TAG, "Fetching FCM registration token failed", task.getException());
                            return;
                        }
                        // Get new FCM registration token
                        String token = task.getResult();
                        // Log and toast
                        String msg = "toke is: "+token;
                        Log.i(TAG, msg);
                        Toast.makeText(MainActivity.this, msg, Toast.LENGTH_SHORT).show();
                    }
                });

      3. 添加接受消息的代码

        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        import android.util.Log;
        import androidx.annotation.NonNull;
        import com.google.firebase.messaging.FirebaseMessagingService;
        import com.google.firebase.messaging.RemoteMessage;

        public class MyFirebaseMessagingService extends FirebaseMessagingService {
            private static final String TAG="MyFirebaseMessagingService";
            @Override
            public void onNewToken(@NonNull String token) {
                Log.i(TAG,"Refreshed token: " + token);
            }
            @Override
            public void onMessageReceived(@NonNull RemoteMessage message) {
                Log.d(TAG, "From: " + message.getFrom());
             		//From: 471967859514
                Log.d(TAG, "AllData: "+message.getData());
              	//AllData: {allDataKey=allDataValue}
                Log.d(TAG, "Notification title: "+message.getNotification().getTitle());
                //Notification title: 我是标题
                Log.d(TAG, "Notification body: "+message.getNotification().getBody());
                //Notification body: 我是body
            }
        }

  4. 制作首个宣传活动(或者新建宣传活动)

发表于 更新于 分类于 Waline：

腾讯IM vs 环信IM

文档对比，腾讯IM容易理解，环信IM文档复杂

腾讯IM使用示例

数据流入流出逻辑

1
2
3
4
客户端->IM服务端:创建用户获取userid
客户端->后端:传入userid获取userSig
客户端->IM服务端:通过usersSig进行登录
客户端->IM服务端:使用聊天等其他IM api功能

  1. 登录腾讯云控制台

  2. 在即时通信IM创建新应用(已有忽略)

  3. 查看SDKAppID和密钥

  4. 在IM腾讯云控制台创建用户(或者通过客户端创建)

  5. 在服务端计算UserSig（建议，不在服务端生成，容易泄露SDKAppID和密钥）

    • 添加依赖(不想添加依赖，可以直接复制里面的两个文件到自己的项目)

      1
      2
      3
      4
      5
      <dependency>
          <groupId>com.github.tencentyun</groupId>
          <artifactId>tls-sig-api-v2</artifactId>
          <version>2.0</version>
      </dependency>

    • 引用工具类，生成userSig

      1
      2
      3
      4
      public static void main(String[] args) {
          TLSSigAPIv2 tlsSigAPIv2=new TLSSigAPIv2(sdkappid:1600,key:"35680781295a9");
          System.out.println(tlsSigAPIv2.genUserSig(userid:"y",expire:86400));
      }

  6. 编写客户端聊天代码，这里体验web版（快速入门（Web & H5 Vue2/Vue3））

    • 填入返回的userSig

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      // 客户端生成userSig
      TUIKit.login({
        userID: userID,
        userSig: genTestUserSig({
          SDKAppID,
          secretKey,
          userID,
        }).userSig, 
      });
      // 替换为服务端生成的userSig
      TUIKit.login({
        userID: userID,
        userSig: userSig, 
      });

环信IM使用示例