可以试试这个
https://github.com/gxywy/rl-plotter

Installation from PIP

pip install rl_plotter

from source

python setup.py install

Usage
Add our logger (compatible with OpenAI-baseline) in your code or just use OpenAI-baseline bench.Monitor (recommended):

from baselines import bench
env = bench.Monitor(env, log_dir)

After the training or when you are training your agent, you can plot the learning curves in this way:

switch to log directory (default: ./)

run command to plot:

rl_plotter --save --show
more general commands in practice:

rl_plotter --save --show --avg_group --shaded_std
rl_plotter --save --show --avg_group --shaded_std --time
rl_plotter --save --show --avg_group --shaded_std --shaded_err

for help:

rl_plotter --help
and you can find parameters to custom the style of your curves.

optional arguments:
-h, --help show this help message and exit
--fig_length matplotlib figure length (default: 6)
--fig_width matplotlib figure width (default: 6)
--style matplotlib figure style (default: seaborn)
--title matplotlib figure title (default: None)
--xlabel matplotlib figure xlabel
--xkey x-axis key in csv file (default: l)
--ykey y-axis key in csv file (default: r)
--ylabel matplotlib figure ylabel
--smooth smooth radius of y axis (default: 10)
--resample if not zero, size of the uniform grid in x direction
to resample onto. Resampling is performed via
symmetric EMA smoothing (see the docstring for
symmetric_ema). Default is zero (no resampling). Note
that if average_group is True, resampling is
necessary; in that case, default value is 512.
(default: 512)
--smooth_step when resampling (i.e. when resample > 0 or
average_group is True), use this EMA decay parameter
(in units of the new grid step). See docstrings for
decay_steps in symmetric_ema or one_sided_ema functions.
(default: 1.0)
--avg_group average the curves in the same group and plot the mean
--shaded_std shaded region corresponding to standard deviation of the group
--shaded_err shaded region corresponding to error in mean estimate of the group
--legend_loc location of legend
--legend_outside place the legend outside of the figure
--no_legend_group_num don't show num of group in legend
--time enable this will set x_key to t, and activate parameters about time
--time_unit parameters about time, x axis time unit (default: h)
--time_interval parameters about time, x axis time interval (default: 1)
--xformat x-axis format
--xlim x-axis limitation (default: None)
--log_dir log dir (default: ./)
--filename csv filename
--show show figure
--save save figure
--dpi figure dpi (default: 400)
finally, the learning curves looks like this:

2 个月 后
3 个月 后
4 个月 后

panchao12345 代表置信区间或者叫误差区间,因为在x轴上每个小小的区间段,有很多个episode rewards,所以有一个上下波动的范围,画图软件会自动帮你估计出中间值的位置,也就是那条线的位置。

  • Yhp 回复了此帖
    8 天 后

    bug404 您好,这个seaborn如何根据tensorboard文件绘制出类似上面的曲线图呢


    [未知] 还是不太会画,能否指导一下呢

    [未知] 我画不出这种效果,能否指导一下呢

      6 个月 后
      9 天 后

      请问我用了seaborn画图,数据是由三组随机种子生成的,为什么画出来的图没有阴影啊?好像都叠在了一起

        5 个月 后

        想问怎么用rl_plotter绘制已经存在的csv文件,绘制成文章中图的样子?

        • aoao 回复了此帖
          3 个月 后
          1 年 后

          wagh311

          import os
          import numpy as np
          import matplotlib.pyplot as plt
          import pandas as pd
          from statsmodels.nonparametric.smoothers_lowess import lowess
          
          def loess_smooth(data, frac=0.035):
              x = np.arange(len(data))
              smooth_result = lowess(data, x, frac=frac)
              return smooth_result[:, 1]
          
          # 生成示例数据
          environments = ["HalfCheetah-v2", "Hopper-v2", "Swimmer-v2", "Walker2d-v2"]
          algorithms = ["PPO", "SAC", "DPPO"]
          
          # 设置文件路径列表
          file_paths = {
              "HalfCheetah-v2": ['Half1.csv', 'Half2.csv', 'Half3.csv'],     # 例如:Half.csv中包含seed1,seed2,seed3三列数据(奖励)
              "Hopper-v2": ['Hopper1.csv', 'Hopper2.csv', 'Hopper3.csv'],
              "Swimmer-v2": ['Swimmer1.csv', 'Swimme2.csv', 'Swimmer3.csv'],
              "Walker2d-v2": ['Walker1.csv', 'Walker2.csv', 'Walker3.csv']
          }
          
          # 创建一个2x2的子图布局
          fig, axs = plt.subplots(2, 2, figsize=(12, 8))
          
          # 创建颜色映射
          color_map = {alg: plt.cm.tab10(i) for i, alg in enumerate(algorithms)}
          
          # 模拟一些数据,以便演示
          for i, env in enumerate(environments):
              ax = axs[i // 2, i % 2]
          
              # 存储当前环境的算法标签
              env_handles = []
              env_labels = []
          
              for file_path, algorithm in zip(file_paths[env], algorithms):
                  df = pd.read_csv(file_path)
                  reward_data = [df[col].tolist() for col in df.columns]
                  mean_rewards = np.mean(reward_data, axis=0)
                  std_rewards = np.std(reward_data, axis=0)
          
                  # 对均值进行Loess平滑处理
                  smoothed_mean_rewards = loess_smooth(mean_rewards, frac=0.035)
          
                  # 对标准差进行Loess平滑处理
                  smoothed_std_rewards = loess_smooth(std_rewards, frac=0.035)
          
                  # 特殊处理 PPO 的颜色
                  if algorithm == "PPO":
                      color = 'orange'
                  else:
                      color = color_map[algorithm]
          
                  # 绘制平滑处理后的奖励函数图
                  line, = ax.plot(smoothed_mean_rewards, label=f'{algorithm}', alpha=0.8, color=color)
                  ax.fill_between(range(len(smoothed_mean_rewards)), smoothed_mean_rewards - smoothed_std_rewards,
                                  smoothed_mean_rewards + smoothed_std_rewards, alpha=0.2, color=color)
          
                  # 存储当前算法的标签和线条
                  env_handles.append(line)
                  env_labels.append(f'{algorithm}')
          
          
              ax.set_title(f"{env}")
              ax.set_xlabel('Time Steps(1e6)')
              ax.set_ylabel('Average Reward')
          
              # 在当前子图中手动创建图例
              ax.legend(handles=env_handles, labels=env_labels, loc='lower right')
          
              # 添加浅灰色虚线网格线
              ax.grid(True, linestyle='--', color='lightgrey')
          
              # # 修改横坐标
              labels=["0",'0','0.2','0.4','0.6','0.8','1M']
              # total_steps = len(smoothed_mean_rewards)  # 获取总步数
              # steps_in_millions = np.linspace(0, total_steps - 1, 6) / 1e6  # 将总步数分割为6个点
              ax.set_xticklabels(labels)
          
          
          # 调整布局以防止标题重叠
          plt.tight_layout()
          # 保存图为PDF文件(dpi=600)
          plt.savefig('multi-result.pdf', format='pdf', dpi=600)
          
          # 显示图形
          plt.show()
            18 天 后
            说点什么吧...
            Document