
Amazon reveals new web service: Amazon Simple Database - BitGeek
http://seattle.craigslist.org/see/eng/299279879.html
======
mattjaynes
Wow, this is awesome. Unfortunately it will probably be a good wait before
this is built and released to the public.

In the meantime, I've been developing a framework using EC2, S3, S3DFS (DFS =
Distributed File System) along with SQLite to try and solve the database
problem on AWS.

S3DFS allows you to mount an S3 bucket to one or multiple EC2 instances as if
it were a local filesystem. It works at the block level and has read/write
caching - so is very fast. (Note: It's free to use for development, but
otherwise requires a paid commercial license.)

Since SQLite uses flat files and has no database server or client, it is ideal
for use on S3. SQLite only scales well to a point, so to get around its
limitations I am separating each user into their own database (each database
is a separate flat file). This can sometimes be a bad idea, but SQLite allows
you to attach multiple databases together to essentially create a temporary
master database where you can run queries across all the user databases at
once. This will be handy for search indexing, site-wide stats, etc. In order
to change the user databases I will have utilities to allow global scheme
alterations.

So, by adding EC2 instances that are all attached to the same S3 bucket, and
using round-robin dns to distribute traffic among them, you can theoretically
scale a web service infinitely. Just launch a new EC2 instance and add it to
the round-robin dns pool and you're done! Even that could be automated to
respond dynamically to traffic.

Imagine the long-term cost-savings from only having to build your app and
architecture once and not having to go through a more complex scaling process
like this: <http://www.scribd.com/doc/21793/Scaling-PHPMySQL-Presentation-
from-Flickr>

Certainly you should only worry about scaling after your app is up and
running, but what if you didn't have to ever worry about it? That's the goal
of this framework.

This is pre-alpha of course and I'm still in the development and
experimentation stage, but I wanted to share with the community and get some
feedback!

<http://www.openfount.com/blog/s3dfs-for-ec2>

<http://www.sqlite.org/>

<http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/>

<http://docs.amazonwebservices.com/AmazonS3/2006-03-01/gsg/>

Update: Since this was so lengthy, I also added it as a post here:
<http://blog.nanobeepers.com/2007/04/07/infinitely-scalable-framework-with-
aws/>

Update2: Please leave any feedback here:
<http://news.ycombinator.com/comments?id=10001>

------
BitGeek
Its a craigslist job posting, rather than an announcement... but just in case
it disappears:

Were looking for an experienced technical expert in highly scalable
distributed systems technologies to join our Software Platform group as a Sr.
Level Engineer. The engineering team is responsible for the overall design and
delivery of all the hard-core technical components that drive SimpleDBs huge
multi-tier service-oriented architecture. The product (SimpleDB) is a new
product designed to optimize real-time lookup of information for internet
applications. As a senior leader in the business, the successful candidate
will work closely with Management, Customers, QA, Design, Product Management,
and Customer Support to maintain and enhance existing product as well as to
develop new products. You will help architect, design, and implement highly
scalable distributed infrastructure that has availability, reliability and
performance guarantees. Drive server-wide and cross-group initiatives, and be
the domain expert. Ivory tower architects need not apply - this is hands on
pos!

ition where you will be asked to do everything from build rock-solid
components to formulate strategy and evangelize technology, lead company-wide
architectural changes, mentor other engineers and provide training and support
for our supported technologies. You need to not only be a top software
developer with an established track record of delivering, but also excel in
communication, leadership and customer focus. If you want to test your limits
and tackle technical challenges you won't see anywhere else then come join our
group

Key Responsibilities:

_Translation of complex functional and technical requirements into detailed
architecture and design_ Be very hands-on; work with the engineering team to
manage the day-to-day development activities by leading architecture
decisions, participating in designs, design review, code review, and
implementation. _Responsible for the over-all systems architecture,
scalability, reliability, and performance._ Help the development team delivery
quality applications on-time and on-budget and execution against project plans
and delivery commitments _Maintain current technical knowledge to support
rapidly changing technology, always on a look out for new technologies and
work with management and development team in bringing new technologies._
Candidate must be able to work with a minimum of technical supervision and
supplemental engineering support, while responding efficiently to multiple
program priorities _Work with engineering teams to design and optimize a wide
variety of development projects using a variety of development tools and
practices. In addition, the role involves developing shared components and
tools, analyzing the latest technologies and development trends, establishing
and implementing standard practices as well as training and mentoring other
engineers.

QUALIFICATIONS:

_ Bachelors degree or equivalent experience _Excellent verbal and written
communication skills_ Ability to work well with people and be both highly
motivated and motivating _Experience working in a small startup, or venture-
funded company highly desirable_ Proven results oriented person with a
delivery focus. _10+ year's overall development experience and 7+ year's
enterprise software experience._ Strong knowledge of data structures,
algorithm, enterprise systems, and asynchronous architectures. _Deep
understanding of web services software architectural and design issues._
Effective at architecting complex interactions across several subsystems to
resolve real-world issues, articulating the architecture and obtaining buy-in
from other teams. _Deep understanding of performance, memory, and I/O,
threading._ Solid highly scalable database application design skills _Deep
systems-level programming skills in Java/C++._ Experience developing
middleware systems and/or distributed system software is preferred
_Outstanding adherence to project deadlines and understands the balance
between doing it right and getting the job done._ Works well in a team
environment and be able to effectively drive cross-team solutions that have
complex dependencies and requirements. _Experience of leading the architecture
of an open source messaging product is preferred._ Strong technical vision,
presentation and technology leadership skills. _Ability to handle multiple
competing priorities in a fast-paced environment_ Previous demonstrated
project management success on similar projects emphasis placed on experience
with high-volume web-based applications

In joining our team, you'll enjoy a competitive salary, great benefits, a
creative and comfortable work environment, and the exciting opportunity to be
part of a fast-paced and growing technology company.

------
bootload
joe gregorio wrote more about this trend ~
<http://bitworking.org/news/158/ETech-07-Summary-Part-2-MegaData,> posted ~
bigdata ~ <http://news.ycombinator.com/comments?id=9483>

