
How much did i screw up? - LELISOSKA
i know there are alot of programmers on this board, so tell me me how on a scale of 1 to 10 how much of a moron i am for using node.js and mongodb to do the following:
scrape 8 different social outlet &#x2F; portfolio apis
sync the the scraped data with 3 different post model schemas as well as a user model using a key store database mongodb
all of that wrapped in promise library to avoid callbacks.
scrape data live and based on certian queries, save, sync etc. 
everything wrapped in promises and i dont even know what will happen if two users run the same query async.
why would i ever use node.js instead of something like python or ruby or whatever and just split the stuff into threads instead of having to deal with retarded javascript async??<p>the only reason i did it is because javascript was the first thing i learned besides some c++ or whatever in a few semesters of college.<p>should i be using a normal synchronous language like c# or java or something?
======
ralusek
None of the problems you have seem to be related to language, let alone
whether it's asynchronous, synchronous, multithreaded or single threaded. What
you've described here sounds simple enough from an outside perspective, so
it's more than likely just time to reconsider your architecture.

So let's talk about solving this with NodeJS. First of all, you've allowed
this to get overwhelming to you because you consider this whole thing to be a
monolithic service, when it's really just time to break it into modules. It
sounds like you have a scraper, a parser, some application which needs the
scraped data, and you have a web API for that application. Cool, easy enough.

So the first thing I would do is pull out your scraper into a completely
different service which has nothing to do with your API. Just have it be a
javascript class which takes some configurations to manage the scraping
settings, and that's about it. All you really need it to do is have a method
which takes some input which tells it what site to scrape, and it will then
give you the output.

Cool, now you need a parser. You need to go through the html returned by your
scraper and format it as JSON. Fortunately, Javascript has a lot of tools very
well suited for dealing with HTML. A tool I like is called Cheerio, which
makes it very easy to parse HTML and get the data you need from it. So you
have a parser now.

All that's left is integrating with your application now. Based off of you
saying that you need to "scrape data live and based on certain queries", what
we want is to expose some routes in your application which make that possible.

If you tell me a little bit more about what your application does, I can give
you some idea of how I would structure this from here.

------
coreyp_1
You never "screw up" by learning. For that matter, you learned _a lot_ about
javascript and asynchronous programming. Furthermore, you would probably now
(more than the average programmer) appreciate the advances of ES6 and the
promises/generator pattern.

As for what you "should" be using, there is no right or wrong answer. The
biggest metric is whether or not the language is a good fit for the task. The
only thing that JS is bad at, IMO, is CPU-intensive tasks. Aside from that, it
may be perfectly adequate for the job.

The only other factors would be if there were a language that offered features
(or libraries) that you need.

------
niftich
Does your product work? If yes, you're not doing too bad.

Under what circumstances does your product break? Test this, don't just
speculate!

Are you finding that you're having to jump through too many hoops to get the
code to do what you want, and are wishful for a different
language/platform/API? If you're fighting against your current tools,
investigate refactoring, or replacing small components to see if they help.

Never throw anything that worked away! Version control, version control!

------
bigiain
Depending on what assurances and promises you've made to other people about
the robustness of your current solution, I reckon there's a high likelyhood
you haven't "screwed up" at all. You've investigated your problem space, found
some workable solutions, and got some proof-of-concept code running.

Writing something in a language you know is always better than not writing
anything in a "better" language that you don't know. For _you_, Javascript was
the right language, at least for your initial explorations. You're now a long
way ahead of where you started, with new-found knowledge about some of the
edge case problems and implementation details that you never had when you
started.

Javascript async is difficult (I'd hesitate to jump all the way to
"retarded"), mostly because async itself is difficult (though, to be honest, a
lot of people consider JS to have implemented it badly as well, but that's
kind of secondary).

Having said that, there are going to be similar downsides to doing it in
Python, Ruby, Haskel, Erlang, Go, Groovy, Befunge, Brainfuck, Assembler, Perl
- that's why there isn't just one language, they all have different
compromises, different learning curves, different support communities,
different library availability and robustness, different expertise
availability...

Try not to ask "how much have I screwed up?", but instead ask "what has this
version taught me?", and "which of the decisions I made would I have made
differently if I knew at the beginning what I knew now?"

Bottom line - your project _can_ be written in Javascrip. It's almost
certainly significantly less complex than Gmail or Googlemaps. It just might
take some very senior level Javascript developers to get there.

Don't think that jumping from college-project-level Javascript to I've-read-a-
bit-about-it-level Ruby or Python is going to magically make the hard things
easy though.

~~~
LELISOSKA
its a personal project and i'm the only one working on it so i have to figure
it out on my own but thanks for the advice

------
bwackwat
If you have a working implementation and have yielded moderate results, then
you have earned "success".

In this case I get the feeling you _want_ to use a different set of
technologies to either improve performance or ease extensibility.

How much did you screw up? 7/10\. You have found that node.js and mongodb are
not the most mature technologies and would likely find python + sql (mssql,
sqlite, postgresql, whatever) ... nicer.

------
asimuvPR
You only know how to build a good program after you've built it. Go back and
re do it. Repeat as much as possible.

