

Ask HN: Server daemon, web service or other? - polyfractal

I'm building an app that requires an adapter/bridge that resides on a server and interfaces with the database. The adapter allows limited database access to third-party servers. I'm trying to determine the best method to implement this adapter.<p>I could build a traditional daemon process using C. I'd prefer to avoid this, since working with databases in C isn't my favorite activity. I'm also decidedly mediocre with C and haven't done anything serious with it in ages.<p>I could build a daemon using a scripting language like Python. I'd feel more comfortable doing this, but I've never daemonized a script before. The code would also have to include network code to be accessible by third-party servers. Is this a good or bad idea? Or nothing unusual?<p>Lastly, I could build equivalent scripts, but serve them through a web-server. The advantage is that I can defer networking to Apache/nginx/whatever. The downside is that the implementation becomes reliant on an existing web-server, and can't be installed on a system that doesn't have one running.<p>Thoughts? Am I just over-thinking this?
======
eduardordm
I built software like this not once, but several times. Server for verifone
pos, RPC servers to allow any client to execute natural (it's a programming
language) subprograms.

First rule: Do never bother writing your own socket server, there are so many
reasons to not build your own server, but just consider that they already
exist and are better than what you could build.

That said, it's all about the type of service: For verifone machines server I
use a custom protocol which I regret I had to construct every single day. I
had to build it because it was a long time ago and connections speeds were too
slow for https.

SOAP is a reasonable option if you have text+binary data. Also if clients
already use SOAP. I dislike XML deeply, but if you have have complex objects
you can combine your web service with XSD schemas and make it great.

Now, the best option: just use dead simple HTTP transporting JSON, it may be
REST or not. But anyways, this is the best option you have. It's reliable,
tested and fast. I don't know python, but there must be something lighter than
django around. Use that with nginx or apache as http server.

Java is pretty cool to build this kind of software. Check out Apache Mina, you
can create a rock solid, production ready server in minutes if you decide to
roll your own protocol.

~~~
eduardordm
Oh, one more thing, if you are doing this to avoid client licensing on
databases, you shouldn't. Most of them have specific licenses for connection
pooling.

~~~
polyfractal
Great, thanks for the advice! HTTP + JSON is exactly what I would love to do.
I was just unsure if it was kosher to do (embedding a lightweight web-server
and running a daemon on top of it)

This isn't for sharing databases...I just need the info that is in a client's
database to perform the required action. Technically, it could be performed
all on their end, but I'd prefer this to be a SaaS for a variety of reasons.

~~~
donavanm
This discussion is all about the agent that sits on the clients premises,
correct? First define your server side RESTy JSON API. Then write poll based
client agent, extract data, transform, PUT to API. Don't bother with embedded
http server or rpc etc. Add a --daemonize option (it's _really_ not too
complicated) for customers who don't like crontab. And finally customers who
don't want to run your agent can write their own against your published API.
Maybe they don't want out bound http directly from their db. They could
implement different transport and have a shim to your API service.

PS: keep batching in mind when doing API. It'll behoove you and the customer
when you get to real RPS.

edit: wait, you want the agent to receive requests for data? From what other
clients?

~~~
polyfractal
> This discussion is all about the agent that sits on the clients premises,
> correct?

Correct.

> wait, you want the agent to receive requests for data? From what other
> clients?

I originally had this in mind, but while working through the problem it is
both unnecessarily and a potential friction point (firewall issues with
outside clients starting connections, etc). I'll just make the client poll the
server at intervals for whatever functions need doing.

> First define your server side RESTy JSON API. Then write poll based client
> agent, extract data, transform, PUT to API. Don't bother with embedded http
> server or rpc etc. Add a --daemonize option (it's _really_ not too
> complicated) for customers who don't like crontab. And finally customers who
> don't want to run your agent can write their own against your published API.
> Maybe they don't want out bound http directly from their db. They could
> implement different transport and have a shim to your API service.

Excellent, this sounds simple and robust. I realized last night that if the
client initiates all the connections, all it needs to do is create HTTP
connections to URLs. Nothing fancy. Should even be pretty easy to BasicAuth
over SSL too, which will be important for security.

I know Python has a few libraries for daemonizing scripts, I'll investigate
those after the basic crontab functionality is complete.

Thanks again everyone. Glad to see that I was over-thinking things. Simple is
good =)

------
staunch

                                           +------------------+
                                           |                  |
                                +---------+|    Job Queue     +--------------+
                                |          |                  |              |
      +---------------+    +----+-----+    +------------------+    +---------+--------+
      |               |    |          |                            |                  |
      |  Client App   +---+| Web App  |                            |  Worker Daemons  |
      |               |    |          |                            |                  |
      +---------------+    +-----+----+    +------------------+    +---------+--------+
                                 |         |                  |              |
                                 +---------+     Database     |+-------------+
                                           |                  |
                                           +------------------+
    

This is a really easy way to create really reliable and flexible systems.
Highly recommend you go this route, rather than trying to recreate a webserver
or database.

~~~
polyfractal
So I was probably too vague in my original description. This is supposed to be
a server-to-server application. Similar to how New Relic's integration
monitors your server and reports home to their servers. So something like
this:

    
    
      +--------------+    +--------------+                          +--------------+
      |              |    |              |                          |              |
      |   Database   +----+  Client App  +--------+           +-----+  Job Queue   +---+
      |              |    |              |        |           |     |              |   |
      +--------------+    +--------------+        |           |     +--------------+   |
                                                  |           |                        |
      +--------------+    +--------------+        |           |                        |
      |              |    |              |        |    +--------------+         +-------+------+
      |              |    |              |        |    |              |        |              |
      |   Database   +----+  Client App  +--------+----+   Web App    |        |Worker Daemons|
      |              |    |              |        |    |              |        |              |
      |              |    |              |        |    +--------------+        +-------+------+
      +--------------+    +--------------+        |                                    |
                                                  |           |                        |
                                                  |           |                        |
      +--------------+    +--------------+        |           |     +--------------+   |
      |              |    |              |        |           |     |              |   |
      |  Database    +----+  Client App  +--------+           +-----+--Database    +---+
      |              |    |              |                          |              |
      +--------------+    +--------------+                          +--------------+
      
    

The "adapter" will sit on "Client App" and interface with their database,
reporting back to "Web App". I need something that is compact and relatively
portable, so that it can be installed on client servers with minimal hassle.
It will be provided limited access to various attributes of the client's
database.

Edit: Damnit. How'd you get your ascii formatting to stick? Edit2:Got it!

Try this: <http://www.asciiflow.com/#3338844714040119796>

~~~
staunch
Well, I'd still treat it like any other "Client App". It does make more sense
for it to be a long-lived daemon though.

If it's the kind of thing where you can run it once in a while:

    
    
      $ bin/myagent --daemon
      .. while 1
      .... exec bin/myagent --report
      .... sleep 300
      
      $ bin/myagent --report
      .. hit the Database
      .. send to Web App
      .. exit
    

Or maybe you want to be able to send requests _to_ the daemon from outside?
The issue there would be firewall rules generally.

Personally, I highly recommend you avoid creating a smart daemon and recommend
the approach above if it will work at all. Embedding a web server (or similar)
tends to create all kinds of issues, including performance, firewll, memory
issues, logging, etc.

~~~
polyfractal
I originally wanted bi-directional communication, since it would simplify
command processing. But you're right, that's just asking for all kinds of
problems.

It'll be easier to have the client poll the server to find out updates or new
commands.

------
jtchang
What is the purpose of the agent that sits on the server with the database? Is
it to gather metrics?

One possible architecture:

\- Agent is a process that sits on the database server written in Python.

\- Code a VERY lightweight, highly stable plugin written in the native code of
the database. The sole purpose is to connect with the local agent and give it
data.

Agent connects to the central server to dump data and do reports. The agent
can poll every 60 seconds for new management commands or new settings.

~~~
polyfractal
The agent that sits on the database server is responsible for two things:

\- Query the database to identify when various trigger conditions are met

\- When a trigger condition is met, extract another piece of data from the
database and send to the "Web App" server for processing

Theoretically, the entire agent is unescessary...I could use remote DB access
credentials to perform the same task. I assumed most companies don't like
giving out remote-access credentials though, and decided a local agent that
reports home is a better option.

So a plugin to the database is unnecessary since a local DB connection would
be sufficient, but I think the rest of what you said is exactly what I'll end
up doing.

