
Show HN: Visualizing disk IO activity using log-scale banded graphs - apankrat
https://bvckup2.com/wip/10042018
======
apankrat
Author here.

We had a need for a tool to graph IO activity at arbitrary sampling rates, so
there was finally a solid excuse to do some datavis work :)

The thing I wanted to show is not the program itself, but the idea of
factoring values and displaying factorization to cover several magnitude
levels in one go. I suspect that this has been done before, but I couldn't
find anything similar when I was researching this beforehand.

~~~
arnioxux
This visualization is really cool and innovative! It's useful for seeing the
changes in the stream of values even if they are small relative to absolute
magnitude (e.g., 1000001 -> 1000005 -> 1000003 etc).

The tradition way to solve this is to rescale the y axis (for example google
stock price chart ranges from 1000-1200). But this doesn't work for graphing
IO because it spikes up/down too often in a huge range.

Your visualization plots the base-1024 digits so whenever the IO is sustained
at a certain range, the top few digits will be held constant and the next
digit will show the variation within that scale.

Unfortunately I don't think this solves the spiking problem completely. It's
mentally hard to parse since two adjacent values are only comparable whenever
their top digits match.

For example it's only meaningful to compare (1gb+2mb+3kb) with (1gb+2mb+5kb)
on the kb graph. If it spikes down to 0gb+0mb+0kb, you need to switch to
reading the gb graph (since comparing the kb value to one before is pretty
meaningless unless for some crazy reason you actually care about the mod 1024
value of two very different numbers).

To fix this, you should color code the "runs" of values that are meaningful to
compare! In other words, kb values have the same color only if their gb/mb
values match. And mb values have the same color if their gb value matches.

~~~
jdmichal
I like the idea of using a color signal to show a changed value, with the
change cascading to lower "digits". So, for instance, if the GB change, then
so do MB, KB, and B. If GB and MB remains the same, then only KB and B will
change.

Hard part would be how to do this over a long span of columns without the
whole thing looking like a unicorn puked on the chart...

------
edmundhuber
This is a great idea, and I have one suggestion:

It can be kinda hard to see if two neighboring entities are the same in a
given band (kbs, mbs, gbs), because you're looking for the absence of a
difference of a pixel or two. Maybe if they are the same and neighboring, join
the two columns rather than have them spaced? That would visually instantly
tell you that those two samples are the same.

~~~
apankrat
Interesting idea, thanks. I'll give it a try.

------
hs86
This just reminds me that I miss something like iStat Menus [0] on Windows.

Process Explorer [1] can show some small graphs as tray icons but there is not
much detail visible without opening the main application's window.

XMeters [2] does a little more but its configuration is very inflexible and
clicking on any item just opens the plain Windows Taskmanager instead of
showing additional details and graphs like iStat Menus.

[0] [https://bjango.com/mac/istatmenus/](https://bjango.com/mac/istatmenus/)

[1] [https://docs.microsoft.com/en-
us/sysinternals/downloads/proc...](https://docs.microsoft.com/en-
us/sysinternals/downloads/process-explorer)

[2] [https://entropy6.com/xmeters/](https://entropy6.com/xmeters/)

------
yoodenvranx
To the people who use log-scales for visualization of discrete events (e.g. 0,
1, 2, 3, 4, ...):

How do you deal with a value of 0?

Do you use the same distance between "0" and 1 as between 1 and 10?

On a different note:

In a lot of cases I prefer using something like sqrt(data) or data^(0.x)
because a) it smoothly supports the case of 0 and b) it squashes the data less
than log.

Regarding b): In a lot of cases I think that log() squashes the information
too much and you easily lose a sense of scale. A scaling funtion like
sqrt(data) still squashes data with high values but it maintains the feeling
of "Oh, this value is much higher than that value" better than log(data).

~~~
aidenn0
The intercept of a log-scaled axis should always be positive, and a value of 0
would not appear on the chart. Many times it's non-sensical for a value of 0
(e.g. pressure, frequency), but wikipedia[1] has an example of tracking
pandemic cases (which tends to be logistic, so includes an exponential growth
phase) and you see that values below 0 do not appear at all on the graph.

In this case, the slope of each dataset is the important thing, and very low
values are probably too noisy to give you information anyways

1: [https://en.wikipedia.org/wiki/File:Influenza-2009-cases-
loga...](https://en.wikipedia.org/wiki/File:Influenza-2009-cases-
logarithmic.png)

------
ryandrake
Recommend testing the site on mobile. I stared confused at a black screen for
a while before realizing that it had a massive fixed width and I had to pan
right to see anything.

~~~
adrianN
For me it shows the number 2 and nothing else, even if I view source.

~~~
dsl
Same here. Looks like the server broke.

~~~
apankrat
Things are OK on the server side.

Ah, scratch that. I see the issue!... Check now?

------
JepZ
While I find the graphs not very intuitive to read, they hold great
information once you understand how to read them :-)

------
TwistedWave
I don't understand. In most of your samples, there is no bar in the first two
bands. Does that mean that your samples are a multiple of 1 MB?

If your read/write rate is not constant, such as in the non-optimized file
copy and you display the rate mod 1024 in the first band, I expect to see
seemingly random bars that would not show any meaningful information. What am
I missing?

~~~
apankrat
> Does that mean that your samples are a multiple of 1 MB?

No my samples, but the buffer sizes in read/write requests are multiples of 1
MB.

Consider _when_ a disk driver would update its read/write counters - it
happens when a request is completed, so even if a request takes longer than a
sampling interval to complete, the byte counter will still go up in buffer
size, atomically.

------
jamesbrink
Here is a quick, far less useful visualization of file activity using Gource
:)

This was run on OSx, linux users can use strace to watch for open files, much
better performance.

# Using lsof in a loop (no sudo required) (while :; do lsof |gsed -r
"s#(\w+)\s+. _\s+(.+)#$(gdate + "%s")\|\1\|M\|\2#;t;d";done) | \ gource
--realtime --filename-time 2 --highlight-users --1280x800 --log-format custom
-

# Using Dtrace (better option, could replace with opensnoop possibly but still
requires dtrace) sudo dtrace -n 'syscall::open_:entry {
printf("%u|%s|%s",walltimestamp,execname,copyinstr(arg0)); }' | \ gsed -r
's#.+ ([0-9]{10}).+\|(.+)\|(.+).+#\1\|\2\|M\|\3#;t;d' |\ gource --realtime
--filename-time 2 --highlight-users --1280x800 --log-format custom -

------
nemo1618
Really neat! Any plans for Linux support? I would love to do some guided
optimizations using this tool.

~~~
apankrat
Zero plans, sorry.

This is an internal tool that just happens to produce good looking graphs, so
we released a build to let people play with it if they want to. It's not a
beta of a product, more of a tech demo... if even that.

~~~
TheWiseOne
Is this going to be part of Bvckup 2 by any chance?

~~~
locusm
Think its going in this [https://ccsiobench.com/](https://ccsiobench.com/)

------
sgc
Unreadable for me on a 15 inch 1920x1080 monitor. I can't see the MB/KB/b
labels. It's tiny and should be resizable. I don't know what the lines
indicate, so I can only see real time data, I have to guess what past values
represent.

------
cbcoutinho
This is a really cool visualization, and a nice implementation of an idea.
Thanks for sharing it

------
gbrown_
The images really need to be larger or at the least be available in a higher
resolution.

------
Groxx
Page isn't loading for me (I just get a "2", no other content), but
archive.org appears to have it cached:
[https://web.archive.org/web/20180411180009/https://bvckup2.c...](https://web.archive.org/web/20180411180009/https://bvckup2.com/wip/10042018)

~~~
apankrat
It was a hiccup on the server side. Sorry about that. Should be back to normal
now.

------
Gys
(For Windows)

~~~
apankrat
As I said in another comment, the interesting part that I wanted to share is
how the data is graphed, and not that there's an executable.

------
imhoguy
Clever idea!

I would see this nicely applicable to system tray icons for monitoring tools.
16 pixels - 4 log bands with 0.25 precision.

------
java-man
Your work is very pleasant to look at, Alex!

~~~
apankrat
Thanks, glad you like it!

------
boromi
great tool however the font size is really small and hard too read

------
productionx
I like it, the size is way to small though. Resource Monitor gives this
information as well btw:edit, reread your post for the arbitrary file size
thing-makes more sense now.

Couple requests: option to scroll the timeline, option to export the timeline,
ability to view more than one drive at a time.

~~~
apankrat
You can't get "Read X bytes since last reboot" from the standard performance
counters, so, no, Resource Monitor doesn't have this data. The best it can do
is the IO rate at one-second resolution.

