
Mate: Netflix browser extension for learning words via subtitles - tosh
https://www.matetranslate.com/netflix
======
cormacrelf
It would be cool (but super niche) to have a "wait, what was that" button in
Netflix. Skip back 10s, enable subs. Would be great when you're trying not to
use the subtitles, but sometimes get a bit outpaced.

Edit: maybe not so niche! Someone build a WebExtension!

~~~
99052882514569
I just always have subs on. There are shows which are downright unwatchable
without subs (e.g. True Detective). I'm not a native speaker, but I'm fluent.

~~~
justaman
Subs also add a lot of the plot sometimes. Certain quiet sounds you wouldn't
otherwise hear are mentioned in the subs that can set a scene, foreshadow, or
explain motives.

~~~
mamon
So, are those sounds really that quiet? I noticed this before, that sometimes
subtitles mention a sound I haven't heard, but I always assumed it's due to my
slightly impaired hearing. I wonder if this happens to people with good
hearing as well.

~~~
Zenbit_UX
Never came across this myself and have above average hearing...

One explanation might be if you're watching a 5.1 surround sound movie and
playing it through your monitor's built-in speakers which can't handle the
extra channels.

------
cschneid
[https://languagelearningwithnetflix.com](https://languagelearningwithnetflix.com)
is a similar tool. Allows side-by-side subtitles in two languages. It also
lets you skip back one whole sentence at a time to re-hear a line.

I used it to start trying to hear and parse Japanese (only about a month in of
learning, so brand new, and just hearing the words and sounds of the language
is still tricky).

~~~
chernikovalexey
It's also amazing. However, Mate is less intrusive in your Netflix experience.
It's still the same-looking subtitle which you can just translate and save for
learning now. Also, an option to export saved vocab as a CSV or to a
vocabulary learning app Reji may come in handy.

------
malodyets
Watching TV and movies with subtitles is a great way for intermediate language
learners (B1 and above) to progress. Kudos to Mate for thinking of capturing
this in Netflix with a browser add-on.

However, creating a flash card with a translation to your native language is
_not_ the best way to master vocabulary. A translation (or, "gloss") can be
helpful initially, but it's more important to see the word in context.

It would be great if a browser extension would include spaced repetition /
flash card creation of both production cards (sentence with a blank usually)
and comprehension cards (the word alone) and pull up the original video frame
for context whenever checking one's recall.

FD: I hold an MA in TESOL / applied linguistics / language acquisition, and I
have several years' experience teaching ESL students. However, I am not yet
fluent in my own second language (Russian). Working on that with the Fluent
Forever methodology (though not their new app).

~~~
AntonOfTheWoods
I think you are underestimating how effective flashcards can be, particularly
as part of a larger system. I'm starting a 4-year PhD in September to develop
a system that will gloss words (in text/subtitles in a variety of clients,
like FF, ebook readers, media players, etc.) that aren't in your (Anki-
compatible) flashcards. That means you get to read "normal" text, enriched
with glosses only for words you don't know. The idea is basically to provide
automated scaffolding at exactly the level the learner is currently at vocab-
wise.

There is an extension to Anki that does what you suggest with the video frames
(at least for Japanese?), and I want to integrate that (or similar) at some
stage. But there are _heaps_ of cool ideas to implement if you take the
flashcard database as a first approximation of a representation of the
learner's (lexical) knowledge of a language and then enrich real-world texts
that the learner is interested in with enough help to appreciate the real-
world content. The beauty with having a representation of the learner's
knowledge is that you can take a number of different approaches (say depending
on the % of unknown words) or even just let the learner decide how much help
they want/need on a given content consumption session. With even a small
number of users the data generated will also be pretty awesome for research
purposes.

It's pretty clunky still (I haven't started the thesis yet) but it is all
going to be completely open-source and I definitely hope I'm not the only one
to contribute to it, so hopefully it will get some momentum at some stage, and
there will be heaps of clients that can talk to the server part.
[https://transcrob.es](https://transcrob.es)

~~~
garfieldnate
Can I sign up to receive updates about it? I'm very curious. This could be
pretty awesome!

------
the_duke
For me, watching TV shows with subtitles is an excellent way to learn a
language, once you are beyond the basics.

I would watch the show with subtitles; when I didn't understand something, I
would pause, look up the phrase/words, write them down for future repetition
and continue.

A technology solution for this is awesome.

~~~
spurgu
I'm doing the opposite. I'm trying to learn Spanish so I'm currently
rewatching Game of Thrones in Spanish, with English subtitles (delayed by 2
seconds). I first try to understand what's being said and if I don't then I
glance down for assistance.

~~~
simias
The subtitle delay is clever, I'll have to remember that. Although personally
I would probably use subtitles in the target language (and use a dictionary if
there's something I don't understand).

~~~
spurgu
Sure, I do that as well. I guess it depends on your method of learning and/or
current level. The downside to staring at subtitles in my mind is that you: a)
miss the acting (just keeping your left hemisphere glued to the text and
trying to decipher it) b) miss actually _hearing_ the language, which even if
you don't even understand a word is useful since it will introduce you to the
overall "sound" of it as well as a lot of repeating phrases

If you're learning how to read/write the language subtitles in that language
seems like a great option. If you're starting out, it helps a lot to hear the
language.

------
pgz
I did something similar for myself using mpv's scripting interface (for
Japanese immersion).

I can press a button and have the current subtitle analyzed for words with
mecab, or press another button and have audio + screenshot + text sent to Anki
for review.

mpv allows to seek previous/next subtitle, so even relistening a particular
phrase is really convenient.

It makes language learning really fun.

~~~
Southland
The audio + screenshot + text sent to Anki for review script sounds awesome as
well as the current subtitle analysis. Can you share these?

~~~
pgz
Indeed, you get high quality in context cards, something that is still missing
from most Anki decks.

The code is here [https://github.com/pigoz/mpv-
nihongo](https://github.com/pigoz/mpv-nihongo), keep in mind that a lot of
stuff is hardcoded, poorly documented, and it requires an mpv patch to work.

If there's enough interest, I can try and make it simple for someone that's
not me to use.

~~~
Southland
Awesome. I'd be interested in adopting it to use for French!

------
wst_
There is a streaming platform Rakuten Viki (quite popular among Korean drama
lovers) which already has a language learning tools as a part of the product.
I haven't used it personally because they are focusing on Korean, which is not
my language choice, but it looked quite nice.

Also they have a subtitling community - any user can contribute to shows
translation.

I am struggling to understand why Netflix is not thinking about that. It would
greatly improve the experience and, probably, user base too.

------
dergachev
Many years ago, I created BabelFrog, a chrome extension to to instantly
displaying the translation for any highlighted text. Unlike other similar
extensions, I don't have access to your pages until it's activated, slowing
down your browsing or risk your privacy.

It's activated with a keyboard shortcut: Cmd-E or Ctrl-E.

[https://babelfrog.com](https://babelfrog.com)

------
AriaMinaei
Subtitles are an extremely effective way to learn a language. And this
extension makes it that much better.

But these edge use cases will actually become less and less common as big
copyright tightens its grip on how content should be consumed.

Like, how can you get this functionality on an iPad?

------
henrywarren
Theres a ton of research on the impact of subtitles on literacy acquisition
for kids. We are currently running a project aiming to convince the major
broadcasters to turn these on by default for kids TV. You can find out more on
www.turnonthesubtitles.org

~~~
TheSpiceIsLife
I’ve read that reading is involuntary to literate adults. Do you know anything
about this?

This is such a great idea. In my experience, most people complain the
subtitles are distracting because they can’t help but focus on them and read
them.

This is _exactly_ what learner readers should be doing in front of a screen.

------
_underfl0w_
Thanks for sharing!

This is generally a good way to immerse yourself in a language with content
that actually interests you - now you don't have to maintain a separate
window/notes/etc.

I really like the idea, and it looks well executed.

------
garfieldnate
This is really fantastic! I have been looking for a good app for this for a
while. It seems like there are several solutions for it, all emerging around
the same time, which is great news for language learners!

Request: please add an option to not continue playing the video after I click
"save". It is aggravating when there are several words that need to be saved
at once and I have to pause the video quickly to get all of them.

You could also make the UI much smoother by copying the way the FluentU player
works: pause the video and show a translation as soon as the user mouses over
any subtitles. Unpause the video when they mouse away from the subtitles.

It would also be really helpful to have some contextual information saved with
each word. The surrounding phrase would be helpful, for instance. It would be
really fantastic to be able to save the screenshot (in low resolution to save
space), but I understand if copyright prevents that somehow.

I really, really wish there were a more open solution for this. Has anyone
else used LWT (Learning with Texts)? You could specify the dictionary API/URLs
to use. (There were also really primitive word splitting regex settings, but I
didn't find them very useful.) I want to be able to get more information on
complicated words, from either compounding (German) or extensive morphology
(Turkish). I want to be able to look up Chinese, Japanese, Vietnamese or Thai
words; these languages require extra processing to extract words because they
don't separate words explicitly. When studying Japanese or Chinese, I want to
be able to save characters for later study, not just words.

In general I really wish I could dig in and add functionality that I know I
would love but I'm not confident that a large enough audience to warrant core
app changes would need.

~~~
garfieldnate
Just noticed, this also doesn't play well with NetflixMultiSubs
([https://github.com/dannvix/NflxMultiSubs](https://github.com/dannvix/NflxMultiSubs)).
I would like to be able to have multiple subtitles shown on the screen while
studying.

------
austinhutch
One benefit of using a VPN was that Netflix provided shows with far more
subtitle options in regions outside the USA. I never understood why if the
subtitles existed for a particular language that they wouldn't be present in
the USA version of Netflix. For example, now that you can't use a VPN with
Netflix I haven't been able to access German subtitles from the USA. Is there
a way around this?

~~~
misiti3780
I have been saying this forever, i have no clue why all subtitles are not
available. When I was in italy all of my favorite shows had italian subtitles,
now im back in the US, they disappeared.

~~~
macd
Change your netflix app language to your target language and you magically get
more subtitle options, if available. In US Netflix, when I set my language to
english, I see english, spanish, and maybe french or italian for some shows.
When I set my language to Korean, I see English, Korean, Japanese, and Chinese
(trad and simplified) when they are available for that show.

Why they base it on your display language and don't just have a setting that
is "show me X, Y, and Z language subtitles if available", I don't know.

------
imranq
What if Mate kept track of the vocab you learned and as you browsed the
internet, replaced English words in articles with that vocab

------
abductee_hg
Had the same idea a while ago - only with "watching a series that you know
already by heart". Unfortunately I soon found out that there is no spanish
version of Start Trek TNG on Netflix/Amzn Prime. ( but I _so_ wanted to see
Patrick Steward yelling "Señor Worf!" )

~~~
psalminen
I think Patrick Stewart saying 'hazlo así' or 'make it so' would've been
brilliant.

------
mhd
While my primary wish would still be something that targets novels ("Learn
enough Italian to read Eco"), something similar for TV shows wouldn't be that
bad. Preferably not a dubbed version, so that it's clearer that you've already
got a benefit from learning what little you did. Problem is that a lot of TV
shows are very much in the vernacular. I wouldn't recommend e.g. the Wire for
ESL studies.

I learned a lot of my early English from reading tabletop RPGs, so I've got no
problem learning about twelve kinds of polearms before I learn the words you
need to get by in a hotel…

~~~
rsl7
Maybe dual-language novels are what you're thinking of?

------
mrits
A list of vocabulary words from the movie to memorize ahead of time would be
nice.

------
nategri
Anyone know of a "reader" app for books that does something similar?

(Totally getting this though---I might not be sold on the Netflix stuff but in
general this plugin seems invaluable to me at this point in my Russian
skills).

~~~
pgz
You can look up words on the Kindle. And if you highlight them there are ways
to access the highlighted sentences in a machine-friendly way (to i.e. export
them to Anki).

------
bduerst
This is fantastic. It would be great to use this to also train text-to-speech
models replicating specific people by feeding unsubbed video clips of the same
person into it, a la something like wave net [1], or what Google is doing with
John Legend [2].

[1] [https://deepmind.com/blog/wavenet-generative-model-raw-
audio...](https://deepmind.com/blog/wavenet-generative-model-raw-audio/)

[2]
[https://www.youtube.com/watch?v=AgHghcYqeto](https://www.youtube.com/watch?v=AgHghcYqeto)

------
RandomInteger4
I could see there being use for something similar to this targeted towards
children for expanding their vocabulary.

Similarly, how neat would it be to have a program that does this for object in
video, and testing the viewer on the name of the object? Perhaps to avoid
error as a result of imperfections in the neural net, it would require 2
parties, the learners and the teachers. The teachers would go through a video
ahead of time with the video, making sure there are no errors, then that
session would be saved and the teacher could play it back for the learners.

~~~
tosh
@ language learning: I wish there was a way to watch movies in various
vocabulary difficulty levels. Would be awesome for learning. Makes me wonder
whether there are fan-dubs with different wordings.

~~~
asztal
Kindle has this in eBooks, for English and Chinese. There's 3 or 4 difficulty
levels. I'm just hoping they'll add more languages. (Or that there will be a
free software solution...)

~~~
tosh
wow, did not know, thanks for the pointer

------
eastendguy
Nice & useful. I just wish this worked for Youtube videos,too. Currently I am
using Copyfish to grab text from images/videos:
[https://chrome.google.com/webstore/detail/copyfish-%F0%9F%90...](https://chrome.google.com/webstore/detail/copyfish-%F0%9F%90%9F-free-
ocr-soft/eenjdnjldapjajjofmldgmkjaienebbj?hl=en)

Copyfish works well, but since it is a generic solution for copying text from
embedded subtitles/images, it is not optimized for language learning.

~~~
chernikovalexey
We're planning on supporting Youtube in the same way as Netflix now. Within a
month or so.

------
garfieldnate
Warning for potential users: in order to export the words that you save, you
must pay for the premium version of the app.

At $10 lifetime user price, it's not bad. But if you watch a couple shows to
try it out and then want to save your work, you will have to pay.

~~~
garfieldnate
I paid. It's worth the price :D

------
jlrubin
Would be cool to see more up front in the privacy policies/terms in conditions
(or, source even) to get assurances that an app that can access all my
browsing data is only doing so when I actively enable it (e.g., for the video
I'm watching).

------
aidenn0
A friend of mine learned english this way with TV and closed-captioning. He
said that without the CCs, he often couldn't figure out how the word was
spelled sufficiently to look it up, no matter how clearly he heard it.

------
jaflo
Interested in seeing the linked tool Reji but the link
([https://reji.me/en/](https://reji.me/en/)) has a redirect loop for me.

------
AH2mdte8kPnJS
The permissions for the Chrome extension require data from all websites?

~~~
chernikovalexey
The extension also translates words and texts on any web page by double
clicking or highlighting with the mouse. There‘s no way to do that without
that permission.

------
kaori
It seems like this doesn't work with Japanese subtitles but works for some
other languages. Am I doing something wrong? Is there some way to make it
work?

~~~
garfieldnate
Ah yes, I know of this problem. For whatever reason Netflix subtitles for a
few Asian languages are images instead of text. Maybe because of font support?
I'm not sure. But you basically have to run OCR on the downloaded subtitles
for these languages.

------
YeahSureWhyNot
I should try this on my mom. she is learning English

------
forkLding
Social media links are broken btw, wanted to give you guys a like.

~~~
chernikovalexey
Oops... Fixed now! :)

------
rootsudo
And here I Was doing it the old fashioned way, this is cool.

------
YeahSureWhyNot
btw it didnt work on firefox
[https://imgur.com/a/Lpf1dIF](https://imgur.com/a/Lpf1dIF)

------
ajaysolleti
interesting

