Hi Everyone,
This is a wee bit off topic, but given the sorts of problems the Clojure
community likes to solve and the enterprise background of a lot of people I
thought this list might be a good place to start.
We are building a fairly large web-infrastructure running over a
combination of internal and cloud-hosted VMs. It's basically
service-oriented, but we have a number of cases where we need to bulk load
data from CSV file and largish JSON files. At the moment, we are using
cron to run the jobs, but it's not really providing the level of visibility
and control we need.
I've added my problem statement below and I am thinking of a solution that
is something along the lines of a cheaper and more agile-friendly version
the enterprise schedulers I used to use when I was building ETL/data
warehouse solutions a few years ago. Before I dive in and start building
something, does anyone know of any open-source projects and/or libraries I
might want to look at?
Also, if anyone else is interested in this space, I would love to hook up
and bounce some ideas around.
Many Thanks
Adrian
The Problem
We need a reusable solution that allows us to schedule, execute and monitor
batch processes accross all our applications.
It should...
- Be controlled by artifacts developers control
- Probably be github friendly
- Provide a direct relationship between an application and its tasks
- Support separate sandbox, staging and production environments
- Be scalable
- Be distributed - jobs for application X can run on the same host as
application X or on a different host or cluster as needed
- Be secure
- Be easy to administer
- Job progress and status is visible
- Alert when a job fails
- Easy to re-run a job
- Easy to spin up new hosts and/or move all processing to a different
host
- Provide a standard way of organising assets like files and configs
across all our applications
- Comply with our hybrid infrastructure (stuff runs internally and in
the cloud)
- Data can move internal -> cloud
- Data can move cloud -> internal
- Data can be processed entirely within a host
- Support different ways of triggering a job
- Scheduled tasks
- Run when file x arrives
- Run job y after job x completes
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
