
Staffpad: a new class of notation app - kayfloriven
http://www.staffpad.net/
======
owenversteeg
Holy shit, this is amazing. For those that haven't written music before,
previous programs are some of the clunkiest, slowest, most infuriating pieces
of crap that I've ever had to deal with. This makes the process literally ten
times easier, and so much more fun to use - and the MP3 export? Amazing.

I'm actually considering using Windows because of this and the new hardware.
If you had told me that two days ago I and my beautiful Arch setup would have
laughed in your face. Kudos to Microsoft/Staffpad, this is really great.

~~~
Omnus
It is a very cool piece of tech to be sure, but as a composer who has used
notation programs almost daily for over a decade, this really doesn't solve
any new problems, and it isn't anywhere near powerful enough for professionals
(yet), so most will be importing files from StaffPad into Finale/Sibelius/etc.

With a MIDI keyboard and some keyboard shortcuts I can enter music extremely
quickly. To write a string of 8 16th notes, for example, I can press 1 button
to select the note length, then press the 8 notes on the keyboard to write the
notes (this is with any modern notation program). With this app, I have to
draw each individual notehead, beam across the top, and then an additional
beam for the 16th notes, then I have to hope I got everything in the right
place so that it correctly recognizes my intentions. On that last point,
anything beyond very simple scores would probably end up being a nightmare in
terms of productivity. A video from last April shows a person making a very
common action - beaming more than 2 eighth notes together in a 4/4 measure,
but the program thinks he wants triplets, so he has to go back and erase the
marking. I can only imagine how many of these "hitches" would pop up in a more
complex score.

The price is fairly low, and they seem to promising more features and
refinement, so it will be interesting to see where it goes. But the input
method, while neat to look at, is fundamentally a regression for
professionals. A Vim-style system that allows for extremely rapid entry of
music would be an interesting and useful approach, but something that might
not have a wide appeal. This app will be useful for those that want to jot
down ideas while not at their desktop, and many people will buy it for the
novelty, but I would imagine most professionals will wait.

~~~
codeulike
_To write a string of 8 16th notes, for example ..._

You haven't actually tried out the app have you? To do 8 16th notes, you just
need to draw 8 small slanted lines.

~~~
Omnus
If you want to explicitly enter 16th notes, you have to make it obvious to
StaffPad that that's what you want, otherwise it can interpret the lines as
8th or 32nd notes, or a combination of those depending on where the notes fall
in the measure. Again, for simple scores where the handwritten music is fairly
obvious, it works fine. The key point is that the entry method slows down
professional users, because you have to make sure you're doing things that the
program can understand, which means taking time to place your marks
methodically and reviewing every entry you make to make sure the program was
correct to your intentions.

~~~
codeulike
OK, but ... have you tried it out? You seem to be assuming quite a lot about
how well it can interpret input.

~~~
shermanyo
it looks like you'd add dashes for the tails, just like you would for a
semiquaver on paper. one of the great things about this is that there's zero
'design' to sheet music, its essentially a spec. There's a lot of implied
information that can be fed into the handwriting recognition, and it looks
like it doesn't try to parse until you've moved onto the next bar, so can
analyze the block against a very well defined domain. Vertical lines all
represent a note (besides small ticks for sharps maybe), and the time
signature narrows this down to only so many possible permutations of
possibilities.

------
codeulike
Interestingly, it interprets each pen stroke in the context of the wider piece
of music, and makes use of temporal information such as the order the strokes
are made in. Whereas classic OCR would just try and interpret the static
'picture':

 _Key to the way StaffPad works is its method of recognizing your scribbles.
It looks at every individual stroke you make and then interprets what you
wrote based on the relationship of each stroke to all of the others. David
says that “it’s more efficient and accurate to take the position and temporal
information from the pen, and then use musical context to decide what the
music is trying to be. That way, you can do things that would totally confuse
OCR. Because we know the order of the strokes and where they are in relation
to the notes, we can say, OK, that’s a natural, that’s a sharp.”_

[http://www.sibeliusblog.com/news/staffpad-is-a-music-
handwri...](http://www.sibeliusblog.com/news/staffpad-is-a-music-handwriting-
app/?singlepage=1)

~~~
mbubb
Fascinating - wonder if this is similar to how a Korean phoneme or an ideogram
is assembled while typing.

~~~
nanofortnight
Similar to how Chinese characters are written: with a prescribed stroke order
learnt at school.

~~~
mbubb
No - I understand that part - I meant the transformation of the syllable as it
is assembled. It changes shape in a similar way.

------
restlake
Staffpad is incredible but it's not brand new or developed by Microsoft.
Hanselminutes (Scott Hanselman) did an excellent interview with the engineer
who created it a few months back, definitely worth listening to if this piqued
your interest. Thoroughly covers how it was conceived and coded in C#/C++,
with comments about the relationship to Microsoft:
[http://www.hanselminutes.com/473/developing-staffpad-a-
new-c...](http://www.hanselminutes.com/473/developing-staffpad-a-new-class-of-
music-notation-application-with-dr-matthew-tesch)

~~~
mike_hearn
It's actually a project started and run by my brother! I'm tremendously proud
of him and what he's achieved with StaffPad, whilst simultaneously running a
business as a contract music producer as well no less.

Seeing him on stage with Nadella was a particular highlight.

------
danielhunt
As far as I'm concerned, _this_ is world-changing technology - a (seemingly?)
easy to use, natural-looking, efficient data-entry mechanism that looks like
something from the future.

That is an absolutely fantastic demo - I was hooked on the background-video
within 3 seconds, and found myself wishing the MP3 would play when he was
exporting it at the end.

Bravo.

~~~
calinet6
Really, I was hooked on the video as soon as he did the pen flourish 4 seconds
in. Nailed it.

~~~
pierrec
In light of this example, it's interesting to consider that world-class pen
spinners [1] might have opportunities as advertising consultants. "Let us
perform or teach you the exact pen spin that's required for your advertisement
- should it communicate confidence, sophistication, should it be quick n'
simple or slightly over-the-top? We've got you covered."

[1]:
[https://en.wikipedia.org/wiki/Pen_spinning](https://en.wikipedia.org/wiki/Pen_spinning)

------
archagon
I've been working on an iPad app in a similar direction, though with a
different end goal. As I'm sure the founders of this company can relate, I've
found Sibelius and Finale incredibly frustrating to use, especially for
anything resembling experimentation. However, I also feel that sheet music is
generally incompatible with modern popular music. (See the Aphex Twin joke
below!) Much of my favorite rock and electronic music has syncopation,
changing meters, pitch bending, and a general fluidity about it that's
incompatible with the rigidity of staff notation. Plus, staff notation is very
difficult to read.

With my project, I'm hoping to bring some of that flexibility back into the
compositional process. In short, it's an infinite canvas with time on the
x-axis and pitch on the y-axis that you can simply paint with your fingers.
You're not constrained by key signature, time signature, or note length,
though there are snapping tools to help with that if you need it. It's meant
more as a tool to take musical "notes" than as something that can produce a
clean final product. As someone who doesn't play the guitar very well (yet),
I'm looking forward to using it to experiment with guitar solos.

Here's an early demo video:
[https://www.youtube.com/watch?v=Ra8OvnoxKQw](https://www.youtube.com/watch?v=Ra8OvnoxKQw)

I hope to release it in the next few months, though we all know how that goes!
iPad Pencil support is definitely something I want to do as well.

~~~
6stringmerc
I'd like to wish you the best in your project, and mention that as a huge fan
of modern drum & bass and other EDM forms, plus as a longtime guitar player, I
believe you're actually working in a space that lacks such a tool. The video
makes me smile. That's a nifty pocket resource to me!

Just going to spit-ball a couple thoughts based on my experiences and your
write-up!

I've been using Blend.io to share my Ableton Live projects with others online
(via Dropbox). This is helpful because settings are exported - do you think
that in time you might have a mechanism in the program to list & chart out the
settings for a synth, say like Massive? Could be a whole can of worms, but
important stuff to recall going forward when taking notes.

As for guitar solos, I think you could use one of the blues Kings for material
in testing, either B.B. King or Freddie King, depending on how much pitch
bending or note rapidity you'd be interested in. If you want to get extreme,
Dimebag Darrell of Pantera had some crazy harmonic & Floyd Rose "dive-bomb"
tricks he'd use that might test capability!

~~~
archagon
Thanks for the thoughts!

The initial offering will be pretty sandboxed from the rest of the production
toolchain; I'm aiming for an MVP with the simple goal of "sketch music
quickly". So there won't be much to offer in terms of interacting with other
software.

After that, however, anything is up for grabs. Eventually, I'd love to add
features that would make it an invaluable music production tool, not just a
note-taking app. Under the hood, each instrumental layer in my app has a set
of "traits" (volume, vibrato, expression, etc.) that get matched to the
appropriate settings in the output instrument if they exist. Only the settings
that exist in the output instrument are used; the rest are ignored. For the
time being, these traits are only used internally for the built-in MIDI synth.
But if I'm understanding you correctly, this means that it should be pretty
simple to add arbitrary traits for outside instruments. And eventually, when I
add Audiobus and/or MIDI-out support, it should be possible to play the track
directly through the synth! (But that's a bit into the future... need to
focus! Need to focus!)

Thanks for the guitar solo tips, I will definitely be using some of those!

------
ThePhysicist
When I saw this my first thought was: We need something like that for Math /
technical drawing.

I can understand that music is a good first application of this technology,
but entering mathematical and engineering-related content through such an
interface could be a huge deal as well.

~~~
hliyan
Flow charts and geometric shapes too.

~~~
wlesieutre
Microsoft's iPad Pro demo showed drawing geometric shapes in PowerPoint.
Reminded me of drawing in Flash but not quite as free form. I'd be very
surprised if PowerPoint on Surfaces never got similar features. Maybe it does
already? I don't have Office on mine.

------
KrisJordan
FWIW, Brown University was doing research in this *Pad field (stylus-based
input apps for domain specific notations) in the late 90s/2000s. I was a
member of this group for a brief period of time. It's really exciting to see a
resurgence in high quality pen computing with affordable, entry-level consumer
hardware like the Surface 3.

Music Notepad (original):
[http://graphics.cs.brown.edu/research/music/home.html](http://graphics.cs.brown.edu/research/music/home.html)

Music Notepad (tabletpc):
[http://graphics.cs.brown.edu/research/music/tpc.html](http://graphics.cs.brown.edu/research/music/tpc.html)

Mathpad: (Commercialized)
[http://www.fluiditysoftware.com/](http://www.fluiditysoftware.com/)
[https://www.youtube.com/watch?v=BAFGONn4KoQ](https://www.youtube.com/watch?v=BAFGONn4KoQ)

Chempad:
[http://pen.cs.brown.edu/research.html#chempad](http://pen.cs.brown.edu/research.html#chempad)

~~~
greenpizza13
I've been working on a similar idea for the iPad pro, and you're work has been
instrumental in doing the handwriting recognition ML part. Thanks!

~~~
jason_slack
I agree, my idea for an iPad Pro app also needs to pay homage to these
projects.

------
jerf
Curiosity: For all those ethusing about this, do you have keyboard skills?
Piano was my primary instrument and my take on this may be skewed by the fact
my primary instrument also turned out to be pretty good for composing music
with, when you can literally bash out full chords in real time if the mood
strikes you. If your instruments were all single-note instruments that don't
hook up to computers worth a darn I could see this being much more exciting.

~~~
calinet6
Composition by writing out rhythms and chords, and writing out multiple parts
for multiple instruments, is just a different experience than playing it out
on a keyboard. It lets you put more thought into how things line up and are
organized, versus how they sound. That sounds downright unmusical but it's a
big difference, I think.

This is just from my limited experience as a hobby composer sitting in front
of a keyboard with pencil and staff paper, but I've heard it from other
composers I know as well.

~~~
jerf
I did it with synthesizer and computer. What you say is still true, though,
since that probably qualifies as yet a third way of composing.

For that matter I _instantly_ noticed a difference going from ScreamTracker to
a MIDI synth, too. Didn't even have an acclimation period, it was immediately
of a different nature.

------
gjm11
Context: This was a prominent part of Microsoft's "Windows 10 devices" event
yesterday. MS used it to show off the capabilities of the new Surfaces.

~~~
apetrovic
The software is actually a bit older, built for Windows 8 and previous Surface
Pro.

Here's a blog post about the design of the app:

[https://www.microsoft.com/en-
gb/developers/articles/week04ma...](https://www.microsoft.com/en-
gb/developers/articles/week04may15/designing-staffpad-5-key-design-
principles/)

------
tatx
Just wondering - wouldn't it be easier, not only for music notation but also
for math and other scientific notation, if there was an on-screen keyboard
custom-built for this very purpose with the required symbols and movement
keys? After all, why require a stylus and why sketch the characters when you
can just as easily type (actually, typing may turn out to be a much faster
input method). Is there a real benefit to sketching or is it something else.

Why wasn't handwriting recognition a big success?

~~~
pervycreeper
Being able to select a note value from a palette, and then just drawing the
head without having to bother with stems and beams would be a time saver in
this case.

~~~
codeulike
The app seems to allow a sort of shorthand

'A quick slanted line is what StaffPad wants for quarter notes and smaller;
simple circles or ovals for half notes and whole notes.'

[http://www.sibeliusblog.com/news/staffpad-is-a-music-
handwri...](http://www.sibeliusblog.com/news/staffpad-is-a-music-handwriting-
app/?singlepage=1)

------
Jedd
So, the MyScript team (behind the Stylus beta handwriting tool on Android)
released the MyScript calculator[1] a while back that lets you start doing
(albeit relatively basic) math with the stylus on a touchpad. (I use it on my
Samsung Note Pro.) It has a similar feel / intent to this thing, but somewhat
less sophisticated (mind, it's evidently a proof of concept demo more than
anything else).

The StaffPad app was reviewed _at least_ 5 months ago [2] and truly does look
very impressive, even if it's of particular interest for a relatively small
demographic. I noted there was no mention in the SP4 announcement of the USB /
sound device latency for the SP4 with W10 - so it's either so good we don't
need to mention it, or a bit uncomfortable so best not to mention it.

[1]
[https://play.google.com/store/apps/details?id=com.visionobje...](https://play.google.com/store/apps/details?id=com.visionobjects.calculator&hl=en)
[2] [http://www.sibeliusblog.com/news/staffpad-is-a-music-
handwri...](http://www.sibeliusblog.com/news/staffpad-is-a-music-handwriting-
app/)

~~~
MichaelGG
Using generic ASIO4ALL drivers I'm able to get latency around 30ms using
"generic" sound cards like the stuff onboard a ThinkPad. ASIO4ALL takes over
the sound device so only one app can use it. Without it, latency can be rather
much higher.

~~~
6stringmerc
Interesting - to me 30 ms is pretty bad. I've got a ThinkPad i5 with 8GB
running Win10 and can get in the 10-15ms range in Ableton Live Suite 9, and I
have access to a SP3 that I might be able to test out. Granted, I don't think
the design of the software is for "live" type use but I have noticed there are
some 'sweet spots' for certain programs in the buffer size, e.g. Kontact
doesn't like lower than 128 for some reason, whereas Traktor Pro doesn't care
in the least.

------
atwrkrmrm
HN front-end devs: If you were to make a web version of this, would it be
better to use canvas or svg? And why?

~~~
tantalor
Either. [http://www.vexflow.com/](http://www.vexflow.com/) and
[http://www.musicxml-viewer.com/](http://www.musicxml-viewer.com/) use canvas.
[http://www.alphatab.net/](http://www.alphatab.net/) and
[http://abcjs.net/](http://abcjs.net/) use svg.

~~~
iraphael
What would be the advantages of using one over the other. I'm in the process
of refactoring/improving a web game (connwars.com) I made a while ago and I
was planning on using canvas, but I don't know if svg would work better.

~~~
adrianh
Use canvas if performance is important. Use SVG if it's important to assign
functionality to specific graphical elements (as opposed to rolling your own
hit detection). See my comment above in this thread for more.

[Usual caveats apply...this is general advice, and I don't know your
situation, etc.]

------
haberman
Now that Apple Pencil exists, I hope they consider a port. This looks amazing
but I just don't think any app could convince me to buy a Windows machine.

~~~
luuio
The iPad Pro doesn't not have palm detection, as far as I know. This is kind
of a deal breaker to port the app.

~~~
anonred
It has hardware palm rejection.

~~~
luuio
Can you give me a link? I tried searching around after seeing the demos, and
it seems like they always have to hover their hands while writing.

~~~
haberman
I saw this today: [http://www.macrumors.com/2015/10/06/disney-artists-ipad-
pro-...](http://www.macrumors.com/2015/10/06/disney-artists-ipad-pro-hands-
on/)

"In the Periscope, Ranjo was impressed with the grip of the iPad Pro's screen
when used with the Pencil, and he also was a fan of the palm rejection
features. When drawing on an iPad Pro, the palm of a hand can be rested on the
screen, but it's still responsive to zoom gestures."

------
6stringmerc
Relevant capability testing joke:

Aphex Twin should give it a whirl!

[http://i.imgur.com/OOspNNV.jpg](http://i.imgur.com/OOspNNV.jpg)

~~~
cmiller1
That sheet music isn't Apex Twin, it's a "joke" unplayable piece of written
music called Faerie’s Aire and Death Waltz made by John Stump

~~~
6stringmerc
I figured as much; I've never thought Aphex Twin's lines were particularly
challenging, he's just got a very unique approach to sound design and
composition.

Would you prefer a technically playable piece by way of Michael Angelo Batio?

[http://www.fundamental-changes.com/wp-
content/uploads/2012/1...](http://www.fundamental-changes.com/wp-
content/uploads/2012/12/Ex-2_0001.png)

------
hellofunk
This is great. A good week to see new MSFT products and demos. Great for
competition. Apple has some catchup to do on a few things.

------
bkolobara
I would love to see a programming/math prototyping environment based on this
idea. Transforming data by sketching. Quickly writing down a formula and
pulling data through it with an arrow. Putting filters on the arrow. Like
Wolfram Mathematica on steroids.

------
6stringmerc
This looks great!

...but will it do Guitar Tab?

Thousands of passionate staff-illiterate-to-barely-competent songwriting
guitarists would probably like to know.

Edit: Went ahead and emailed through their contact form, I'm eager to find out
what they say.

~~~
6stringmerc
Got the email back: Nope, no Guitar Tab support in the application, nor is it
ever planned.

"Other programs already do that well," according to the response.

------
truebosko
The idea of writing via a stylus and having it transformed not into a massive
PNG, but perhaps a simple Unicode file is really growing on me.

I love pen and paper for jotting down ideas or using it to solve problems, but
it seems this medium is slowly going to take over due to the benefits of
having a software tool analyzing your writing, and thus providing
tools/suggestions/etc. along the way.

------
dedene
Wow! Amazing. Does something like this exist for iPad or Android tablets?

~~~
rqin
NotationMe

~~~
jason_slack
It is actually - NotateMe

Located here:
[http://www.neuratron.com/notateme.html](http://www.neuratron.com/notateme.html)

------
Bjartr
I really want to try it out of curiosity, but as someone who doesn't actually
know how to read, write, or play music I can't justify actually paying for it.

~~~
ctdonath
All the more reason to get it! Unlike other tools requiring more intermediary
technical details, this lets you get straight from note-on-staff (in the most
natural way possible short of actual instruments) to proper notation to
hearing it instantly.

------
tvon
FYI, the preview that shows up when shared to Facebook is broken (just shows
some CSS).

